System

Abstract

To provide a system for enabling a user to easily perform 3D printing through an instruction by voice, and to easily prepare a product close to an ideal product without making it necessary to provide any expertise or technique.SOLUTION: A voice input unit that receives a voice instruction, a voice recognition unit that converts the voice instruction into text data, a natural language processing unit that analyzes the text data to understand the user's intention, a response generation unit that generates an additional question for the user based on the analysis result, a response reception unit that presents the generated question to the user and receives a response thereto, a design generation unit that generates a 3D design drawing based on the response of the user, and a display control unit that displays the 3D design drawing on the display unit; A system includes a design confirming means for obtaining a confirmation, a date converting means for converting a confirmed design to a date for a 3D printer, and a print control means for transmitting the date for the 3D printer to the 3D printer and starting printing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional 3D printing technology requires knowledge of 3D design software, is highly specialized, and is difficult for non-experts to create highly accurate designs. Furthermore, repeated revisions are required to create the specific design desired by the user, which is time-consuming. The present invention aims to solve these problems and provide a system that allows users to easily create objects with the desired design using a 3D printer. [Means for solving the problem]

[0005] The system of the present invention includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's intentions, a natural language generation means for generating additional questions for the user based on the analysis results, a response acceptance means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a 3D design plan based on the user's responses, a design confirmation means for presenting the 3D design plan to the user for confirmation, a data conversion means for converting the confirmed design plan into 3D printer data, and a print control means for transmitting the 3D printer data to the 3D printer and starting printing. This system enables users to easily perform 3D printing through voice instructions, allowing them to easily create products that are close to their ideal without requiring specialized knowledge or skills.

[0006] The "voice input means" is a device or function that allows the user to input voice, and plays a role in passing voice instructions to the system.

[0007] "Speech recognition means" refers to software or algorithms for converting speech received by a speech input means into digital text data.

[0008] "Natural language processing means" refers to a technology or algorithm for analyzing text data converted by speech recognition means and understanding the user's intent.

[0009] A "natural language generation means" is software or an algorithm that generates output in natural language based on the analysis results of the natural language processing means to provide the user with additional questions or information.

[0010] The "response receiving means" is a device or function for receiving a user's response to the question or information presented by the natural language generating means and passing the content of that response to the system.

[0011] A "design generator" is software or algorithm for generating a 3D design blueprint based on user responses.

[0012] The "design confirmation means" is a device or function for presenting the generated 3D design drawing to the user and obtaining confirmation.

[0013] "Data conversion means" means software or algorithms for converting the confirmed 3D design blueprints into data for a 3D printer.

[0014] "Print control means" means software or algorithms for sending 3D printer data to a 3D printer and initiating printing.

[0015] A "progress notification means" is a device or function for notifying the user of the operating status of the 3D printer and the progress of printing in real time. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The specific configuration and functions of a system according to an embodiment of the present invention will be described below.

[0038] This system allows users to design 3D prints by giving voice instructions. The user gives voice instructions to a terminal, which is accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal.

[0039] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, it understands the user's intent and extracts specific design requirements. The server then uses natural language generation to generate any necessary follow-up questions and create text data to ask the user.

[0040] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[0041] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses the design generation means to generate a 3D design blueprint. The generated blueprint is sent to the terminal and requested for user confirmation.

[0042] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[0043] Finally, after the user approves the design drawing, the server uses the data conversion means to convert the design drawing into data for the 3D printer, which is then sent to the 3D printer through the print control means, and printing begins.

[0044] The terminal receives the progress status of the 3D printer in real time and notifies the user of the progress status via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[0045] As a concrete example, consider the case where a user requests, "Please create a new design blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design blueprint based on the response. Finally, once the user confirms and the blueprint is finalized, the server sends the data to a 3D printer, which prints the custom coffee cup.

[0046] In this way, the system of the present invention allows users to 3D print their ideal products simply through voice instructions, without requiring specialized knowledge.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The user speaks to the device, for example, "Create a new custom coffee cup design."

[0050] Step 2:

[0051] The terminal uses the voice input means to accept and record the user's voice.

[0052] Step 3:

[0053] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[0054] Step 4:

[0055] The terminal transmits the converted text data to the server.

[0056] Step 5:

[0057] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[0058] Step 6:

[0059] The server uses natural language generation to generate questions to ascertain any additional details needed from the user, such as "Do I need a handle?" or "Do I need a color preference?"

[0060] Step 7:

[0061] The server transmits the generated question to the terminal as text data.

[0062] Step 8:

[0063] The terminal presents the question to the user via a user interface or audio output.

[0064] Step 9:

[0065] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[0066] Step 10:

[0067] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[0068] Step 11:

[0069] The terminal transmits the text data to the server.

[0070] Step 12:

[0071] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[0072] Step 13:

[0073] The server sends the generated 3D design blueprint to the terminal.

[0074] Step 14:

[0075] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[0076] Step 15:

[0077] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[0078] Step 16:

[0079] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[0080] Step 17:

[0081] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[0082] Step 18:

[0083] The user makes a final check and approves, saying "This is fine."

[0084] Step 19:

[0085] The server converts the final design into data for the 3D printer.

[0086] Step 20:

[0087] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[0088] Step 21:

[0089] The 3D printer begins printing the object based on the received data.

[0090] Step 22:

[0091] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[0092] Step 23:

[0093] When the terminal completes printing, it notifies the user of the completion.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] The design process using conventional 3D printing devices requires specialized knowledge and skills, making it difficult for many general users. Furthermore, creating the desired design requires operating complex software and understanding technical terminology, often resulting in design errors and trial and error. The present invention aims to solve these problems by providing a system that allows users to intuitively create 3D designs using voice commands and then print them directly on a 3D printing device.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's instructions, a natural language generation means for automatically generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a three-dimensional design drawing based on the user's responses, a design confirmation means for presenting the generated three-dimensional design drawing to the user and obtaining confirmation, a data conversion means for converting the confirmed three-dimensional design drawing into three-dimensional printing data, and a print control means for transmitting the three-dimensional printing data to a three-dimensional printing device and starting printing. This allows users to easily create and print their ideal three-dimensional design through voice instructions without requiring specialized knowledge.

[0099] The "voice input means" is a device or module that has the function of accepting voice instructions from the user.

[0100] "Speech recognition means" refers to software or hardware that has the function of converting voice data into text data.

[0101] "Natural language processing means" refers to software or algorithms that have the ability to analyze text data and understand the user's intent.

[0102] A "natural language generation means" is software or an algorithm that has the function of automatically generating follow-up questions for the user based on the analysis results.

[0103] The "response receiving means" is a device or module that has the function of receiving a user's response to a generated question.

[0104] A "design generator" is software or algorithm capable of generating a three-dimensional design drawing based on user responses.

[0105] The "design confirmation means" is a device or module that has the function of presenting the generated three-dimensional design drawing to the user and obtaining confirmation.

[0106] "Data conversion means" refers to software or algorithms that have the function of converting the confirmed three-dimensional design drawings into data for three-dimensional printing.

[0107] "Printing control means" refers to software or hardware that has the function of sending three-dimensional printing data to a three-dimensional printing device and starting printing.

[0108] The "progress notification means" is a device or module that has the function of notifying the user of the progress of the three-dimensional printing device in real time.

[0109] An "artificial intelligence algorithm" is an algorithm that performs speech recognition and natural language processing using technologies such as machine learning and neural networks.

[0110] The specific configuration and functions of a system for implementing the present invention will be described below. This system allows a user to design a 3D print by issuing voice instructions. The user issues voice instructions to a terminal, which are accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal. Specifically, a microphone is used as the voice input means, and a voice recognition API (e.g., Google Speech-to-Text API) is used as the voice recognition means.

[0111] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, the server understands the user's intent and extracts specific design requirements. A natural language processing model (e.g., OpenAI ChatGPT) is used for the natural language processing.

[0112] The server then uses a natural language generation means to generate any necessary follow-up questions and create text data to ask the user. A natural language generation model (e.g., OpenAI GPT-3) is also used as the natural language generation means.

[0113] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[0114] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses a design generation tool to generate a 3D design drawing. The design generation tool uses a 3D modeling software API (e.g., Autodesk Fusion 360 API). The generated design drawing is sent to the terminal and the user is asked to confirm it.

[0115] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[0116] Finally, after the user approves the design drawing, the server uses a data conversion means to convert the design drawing into data for 3D printing. This data conversion includes converting the data into a standard 3D printer file format (e.g., STL format). This data is sent to the 3D printing device via a print control means, and printing begins. Specifically, a communication module is used to link with the 3D printing device via a network.

[0117] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user of this via the progress status notification means. When printing is completed, the terminal notifies the user of the completion.

[0118] As a concrete example, consider the case where a user requests, "Please create a new design for a blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design for a blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design based on the response. Finally, once the user confirms and the design is finalized, the server sends the data to a 3D printing device, which prints the custom coffee cup.

[0119] In this way, the system of the present invention allows users to three-dimensionally print their ideal products simply through voice instructions, without requiring specialized knowledge.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The user inputs a voice command.

[0123] The user speaks instructions into the device regarding the desired three-dimensional design, for example, "Create a new design for a blue coffee cup with a handle."

[0124] Input: Voice commands

[0125] Output: Audio data recorded on the device

[0126] Specific operation: The user speaks into the device's microphone.

[0127] Step 2:

[0128] The device converts the speech to text.

[0129] The device's built-in speech recognition means (e.g., Google Speech-to-Text API) converts voice input into text data.

[0130] Input: Audio data

[0131] Output: Text data

[0132] Specific operation: The device analyzes the voice data and outputs it as text data.

[0133] Step 3:

[0134] The terminal transmits the text data to the server.

[0135] The terminal transmits the converted text data to the server via the network.

[0136] Input: Text data

[0137] Output: Text data sent to the server

[0138] Specific operation: The device uploads text data to the server via the network.

[0139] Step 4:

[0140] The server analyzes the text data and understands the intent.

[0141] The server analyzes the text data using natural language processing tools (e.g., OpenAI ChatGPT) to understand the user's intent.

[0142] Input: Text data

[0143] Output: Analysis results that understand the user's intent

[0144] Specific operation: The server receives the text data and analyzes it using a natural language processing algorithm.

[0145] Step 5:

[0146] The server generates a follow-up question.

[0147] Based on the analysis, the server generates additional questions to obtain the required details.

[0148] Input: Analysis results

[0149] Output: Additional question text data

[0150] Specific operation: The server generates a question using a natural language generation means.

[0151] Step 6:

[0152] The terminal presents the user with a follow-up question.

[0153] The terminal displays the question received from the server to the user or presents it to the user by voice.

[0154] Input: Additional question text data

[0155] Output: The question presented to the user

[0156] What happens: Your device displays the question on the screen or uses text-to-speech to read the question aloud.

[0157] Step 7:

[0158] The user responds to additional questions.

[0159] The user responds verbally to the questions posed, for example, "The height is 10 cm and the diameter is 8 cm."

[0160] Input: Additional Question

[0161] Output: User's voice response

[0162] Specific operation: The user speaks the response into the device's microphone.

[0163] Step 8:

[0164] The device converts the voice response into text and sends it to the server.

[0165] The terminal converts the user's voice response into text data and sends it to the server.

[0166] Input: Voice response

[0167] Output: User response as text data

[0168] Specific operation: The device converts the voice into text and sends it to the server over the network.

[0169] Step 9:

[0170] The server determines the user's detailed requirements.

[0171] The server then re-analyzes the received text data to determine the user's specific requirements.

[0172] Input: User's detailed text response

[0173] Output: Detailed design requirements

[0174] Specific operation: The server re-analyzes the text data using natural language processing means.

[0175] Step 10:

[0176] The server generates the three-dimensional design drawings.

[0177] The server uses design generation tools (e.g. Autodesk Fusion 360 API) to generate 3D design drawings based on the user's requirements.

[0178] Input: Detailed design requirements

[0179] Output: 3D design data

[0180] Specific operation: The server calls the API to automatically generate a 3D design drawing.

[0181] Step 11:

[0182] The server sends the generated design drawing to the terminal and asks the user for confirmation.

[0183] The server transmits the three-dimensional design drawing data to the terminal and sends a message to the user requesting confirmation.

[0184] Input: 3D design data

[0185] Output: The design presented to the user, along with a message requesting confirmation

[0186] Specific behavior: The device displays a preview of the blueprint and presents a confirmation message to the user.

[0187] Step 12:

[0188] The user checks the design drawings and gives instructions for corrections if necessary.

[0189] The user checks the design drawings and, if any corrections are necessary, gives voice instructions.

[0190] Input: Design drawings and modification instructions

[0191] Output: Audio prompts for corrections

[0192] Specific operation: The user speaks the corrections into the terminal.

[0193] Step 13:

[0194] The terminal converts the correction instructions into text and sends it to the server.

[0195] The terminal converts the user's correction instructions into text data and transmits it to the server.

[0196] Input: Voice commands

[0197] Output: Correction instructions as text data

[0198] Specific operation: The device converts the voice into text and sends it to the server over the network.

[0199] Step 14:

[0200] The server updates the blueprint again.

[0201] The server updates the design drawings again based on the received correction instructions.

[0202] Input: Correction instructions as text data

[0203] Output: Corrected 3D design data

[0204] Specific operation: The server regenerates the design drawing using the design generation means.

[0205] Step 15:

[0206] The user provides final confirmation, and the server converts the data for 3D printing.

[0207] After the user finally approves the design drawing, the server uses a data conversion means to convert it into data for three-dimensional printing (e.g., STL format).

[0208] Input: Final confirmed 3D design data

[0209] Output: 3D printing data

[0210] Specific operation: The server converts the blueprint data into the appropriate file format.

[0211] Step 16:

[0212] The server sends the 3D printing data to the printing device and begins printing.

[0213] The server sends the converted data to the three-dimensional printing device and initiates the printing process using a printing control means.

[0214] Input: 3D printing data

[0215] Output: Print execution

[0216] Specific operations: The server sends data to the printing device and issues a command to start printing.

[0217] Step 17:

[0218] The terminal receives real-time progress and notifies the user.

[0219] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user using the progress status notification means.

[0220] Input: Progress data

[0221] Output: Progress notification

[0222] Specific operation: The device receives the progress data, displays it on the screen, and notifies the user.

[0223] Step 18:

[0224] The terminal notifies the user that printing is complete.

[0225] When printing is complete, the terminal notifies the user of the completion.

[0226] Input: Print completion data

[0227] Output: Completion notification

[0228] Specific operation: The device will display a completion message and notify you with an alarm sound or vibration.

[0229] (Application example 1)

[0230] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] Conventional 3D design systems required specialized knowledge and skills, making it difficult to quickly design and manufacture custom parts on the factory floor. This limited factory productivity and flexibility. Furthermore, voice-based systems were unable to accurately grasp the user's intent and were unable to meet the user's specific design requirements. Furthermore, there was no way to know the progress in real time, making it difficult to constantly monitor whether the print was running properly.

[0232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0233] In this invention, the server includes an input device that accepts voice instructions, a voice analysis device that converts the voice instructions into text data, a natural language processing device that analyzes the text data to understand the user's intent, a response acceptance device that presents generated questions to the user and accepts their responses, a design generation device that generates a 3D design blueprint based on the user's responses, a data conversion device that converts the confirmed blueprint into 3D printer data, a print control device that sends the 3D printer data to the 3D printer and starts printing, and a means for issuing design instructions for commercial parts in a factory via voice instructions, generating blueprints based on those instructions, and directly controlling printing on the 3D printer. This allows even users without specialized knowledge to quickly and easily design custom parts in a factory using voice instructions and manufacture them on a 3D printer. Furthermore, progress is monitored in real time and the user is notified, making it easy to confirm whether printing is proceeding normally.

[0234] An "input device" is a means for receiving voice instructions from a user.

[0235] A "voice analysis device" is a means for converting voice instructions into text data.

[0236] A "natural language processing device" is a means for analyzing text data and understanding the user's intent.

[0237] A "natural language generation device" is a means for generating follow-up questions for the user based on the analysis results.

[0238] The "response accepting device" is a means for presenting the generated question to the user and accepting the response thereto.

[0239] A "design drawing generator" is a means for generating a 3D design drawing based on a user's response.

[0240] The "verification device" is a means for presenting the generated 3D design blueprint to the user and obtaining their confirmation.

[0241] The "data conversion device" is a means for converting the confirmed design drawings into data for a 3D printer.

[0242] A "print control device" is a means for sending 3D printer data to a 3D printer and starting printing.

[0243] A "progress notification device" is a means of notifying the user of the progress of a 3D printer in real time.

[0244] MODE FOR CARRYING OUT THE INVENTION

[0245] In this embodiment, we will explain the specific configuration and operation of a system that uses voice instructions to create 3D designs in a factory and quickly create custom parts. The main components used here and their roles are as follows:

[0246] Key components of the system

[0247] 1. Input Devices

[0248] This is a device with a microphone that allows the user to input voice commands. The microphone built into a smartphone or tablet is used.

[0249] 2. Voice analysis device

[0250] This is a device that has the function of converting user voice instructions into text data. Specifically, it uses Google's voice recognition service to convert voice into text.

[0251] 3. Natural Language Processing Systems

[0252] This is a device that analyzes text data sent from a voice analysis device and understands what the user is looking for. It uses natural language processing algorithms, especially generative AI models.

[0253] 4. Natural language generator

[0254] This is a device that generates follow-up questions for the user based on the analysis results of the natural language processing device. For example, it creates questions to confirm details of the design.

[0255] 5. Response and Reception Device

[0256] This device presents the generated questions to the user and collects their responses. It reuses voice input and converts it into text data.

[0257] 6. Design Drawing Generator

[0258] This device generates 3D design blueprints based on user responses. Design software is used to create blueprints based on specific shapes and dimensions.

[0259] 7. Verification Device

[0260] This device presents the generated 3D design blueprint to the user for confirmation, and displays it on a smartphone or tablet screen.

[0261] 8. Data Conversion Device

[0262] This is a device that converts the confirmed design drawings into data for 3D printers. It converts the design drawings into a format that can be read by 3D printers, such as STL format.

[0263] 9. Printing Control Device

[0264] This device sends the converted 3D data to the 3D printer and starts printing.

[0265] 10. Progress notification device

[0266] This device notifies users of the progress of their 3D printer in real time, sending notifications to their smartphones and tablets.

[0267] Example of operation

[0268] If an employee voice-inputs, "I'd like to create a custom wrench with a new design," the input device collects the speech and the speech analysis device converts it into text data. The natural language processing device analyzes the data, and the natural language generation device generates follow-up questions about the specific size and shape of the wrench. For example, a question such as, "What is the length and width of the wrench?" is generated. If the user responds, "It's 20 cm long and 5 cm wide," the response acceptance device collects the data and converts it back into text.

[0269] The blueprint generation device then generates a 3D design blueprint based on the response, and the confirmation device presents the blueprint to the user. Once the user reviews and approves the blueprint, the data conversion device converts the blueprint into data for the 3D printer. This data is then sent to the 3D printer via the print control device, and the printing process begins. The user is notified of the printing progress in real time via the progress notification device.

[0270] Prompt Sentence Examples

[0271] "Imagine an application where a user gives voice instructions for designing custom tools to be used in a factory, converts those instructions into text, sends it to a server, and generates a 3D design. As a concrete example, explain the process by which an employee might say into their smartphone, 'I want to create a new custom wrench design,' and then specify the exact size and shape of the wrench by voice."

[0272] In a specific implementation of this invention, factory robots can implement voice-activated design instructions and rapidly 3D print custom-made parts, improving productivity and providing a flexible manufacturing process.

[0273] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0274] Step 1:

[0275] The user issues a voice command into the input device, which then collects the voice as a digital voice signal. This digital voice signal is the input for step 1, and its output is digital voice data. Specifically, the smartphone's microphone collects the user's voice and stores it as digital data.

[0276] Step 2:

[0277] The device uses a voice analyzer to convert the digital voice data from step 1 into text data. The input is digital voice data and the output is text data. Specifically, the Google speech recognition service API is used to convert the collected voice data into text format.

[0278] Step 3:

[0279] The server uses a natural language processing device to analyze the text data from step 2 and understand the user's intent. The input is text data, and the output is the analysis result that includes the user's intent. Specifically, the generative AI model analyzes the text data and understands the "custom wrench design."

[0280] Step 4:

[0281] The server uses a natural language generator to generate follow-up questions for the user based on the analysis results. The input is the analysis results, and the output is text data containing the follow-up questions. Specifically, the AI ​​model generates questions such as "Please specify the length and width of the wrench."

[0282] Step 5:

[0283] The terminal uses the response reception device to present the follow-up question of step 4 to the user and collects the user's voice response. The input is the text data of the follow-up question, and the output is the user's voice response. Specifically, the question is displayed on the smartphone screen, and the user's voice response is collected by the microphone.

[0284] Step 6:

[0285] The terminal uses a voice analysis device to convert the response voice from step 5 back into text data. The input is the response voice, and the output is the response text data. Specifically, the voice recognition service API is used again to convert the voice response into text.

[0286] Step 7:

[0287] The server uses a blueprint generator to generate a 3D design blueprint based on the response text data from step 6. The input is the response text data, and the output is the 3D design blueprint. Specifically, dedicated design software is used to generate a blueprint of the wrench based on the specified length and width.

[0288] Step 8:

[0289] The server uses a confirmation device to present the generated 3D design blueprint to the user and ask for confirmation. The input is the 3D design blueprint, and the output is the user's confirmation response. Specifically, the blueprint is displayed on the smartphone screen, and an interface is provided for the user to confirm.

[0290] Step 9:

[0291] The server converts the user-confirmed design drawings into data for a 3D printer using a data converter. The input is the confirmed 3D design drawing, and the output is data for a 3D printer. Specifically, the design drawing is converted into a data format for a 3D printer, such as STL format.

[0292] Step 10:

[0293] The server uses the print control device to send the 3D printer data from step 9 to the 3D printer and start printing. The input is the data for the 3D printer, and the output is an instruction to start printing. In concrete terms, the 3D printer receives the data and prints based on the specified design.

[0294] Step 11:

[0295] The server uses a progress notification device to notify the user of the 3D printer's progress in real time. The input is the 3D printer's progress data, and the output is a progress notification. Specifically, the progress is notified to the smartphone, allowing the user to check the print progress.

[0296] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0297] This invention relates to a system that creates and prints 3D designs based on user voice commands. In particular, it is characterized by its ability to optimize the design process according to the user's emotional state by incorporating an emotion engine that recognizes the user's emotions.

[0298] The system starts with the user inputting a voice command into the terminal, such as "Please create a new custom coffee cup design." The voice command is then accepted using a voice input means.

[0299] The terminal uses a speech recognition means to convert the user's speech into text data. This text data is sent to a server. The server uses a natural language processing means to analyze the text data and understand the user's intention. During this analysis process, the server runs an emotion engine in parallel to recognize the user's emotions.

[0300] The emotion engine analyzes the user's tone, speech rate, context, etc. to estimate the user's emotional state. For example, it can identify whether the user is excited or relaxed. The estimated emotional information is passed to the natural language generator, which generates appropriate follow-up questions.

[0301] The server sends the generated follow-up questions to the terminal as text data and presents them to the user. For example, questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are presented to the user, and the user's answers are converted back into text data via speech recognition. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[0302] The server generates a 3D design blueprint based on the user's responses. The generated blueprint is sent to the device and displayed to the user. The user can check the blueprint and make any necessary corrections by voice. The correction instructions are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[0303] Finally, when the user approves the design drawing, the server converts it into data for the 3D printer using the data conversion means and sends it to the 3D printer via the print control means. The 3D printer prints an object based on the received data.

[0304] The terminal also receives the progress status of the 3D printer in real time and notifies the user via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[0305] For example, if a user says, "Please create a new design of blue coffee cup with a handle," the device converts the voice to text data and sends it to the server, which uses its emotion engine to analyze the instruction and determine whether the user is excited or relaxed.

[0306] If the emotion engine determines that the user is relaxed, it generates specific questions such as, "What shade of blue cup would you like?" and presents them to the user via the device. As the user provides more detailed answers, the blueprint is generated in a form that is closer to the user's ideal.

[0307] In this way, the system of the present invention allows users to 3D print their ideal product based on voice instructions and emotional information, without the need for specialized knowledge or skills and in a short amount of time.

[0308] The processing flow will be explained below.

[0309] Step 1:

[0310] The user speaks to the device, for example, "Create a new custom coffee cup design."

[0311] Step 2:

[0312] The terminal uses the voice input means to accept and record the user's voice.

[0313] Step 3:

[0314] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[0315] Step 4:

[0316] The terminal transmits the converted text data to the server.

[0317] Step 5:

[0318] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[0319] Step 6:

[0320] The server runs an emotion engine that analyzes the tone, speaking rate, and context of the received voice data to estimate the user's emotional state.

[0321] Step 7:

[0322] The server uses natural language generation to generate additional questions that take emotional information into account. For example, if the user is relaxed, the server generates questions such as "Do you need a handle?" or "Do you have a color preference?"

[0323] Step 8:

[0324] The server transmits the generated question to the terminal as text data.

[0325] Step 9:

[0326] The terminal presents the question to the user via a user interface or audio output.

[0327] Step 10:

[0328] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[0329] Step 11:

[0330] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[0331] Step 12:

[0332] The terminal transmits the text data to the server.

[0333] Step 13:

[0334] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[0335] Step 14:

[0336] The server sends the generated 3D design blueprint to the terminal.

[0337] Step 15:

[0338] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[0339] Step 16:

[0340] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[0341] Step 17:

[0342] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[0343] Step 18:

[0344] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[0345] Step 19:

[0346] The user makes a final check and approves, saying "This is fine."

[0347] Step 20:

[0348] The server converts the final design into data for the 3D printer.

[0349] Step 21:

[0350] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[0351] Step 22:

[0352] The 3D printer begins printing the object based on the received data.

[0353] Step 23:

[0354] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[0355] Step 24:

[0356] When the terminal completes printing, it notifies the user of the completion.

[0357] Specific examples

[0358] For example, if the user says "Create a new design for a blue coffee cup with a handle," the following happens:

[0359] 1. The user issues a voice command.

[0360] 2. The device converts the voice into text data and sends it to the server.

[0361] 3. The server parses the text data and understands that the user wants to create a custom coffee cup.

[0362] 4. The server uses the emotion engine to estimate the user's emotional state and generate follow-up questions.

[0363] 5. The terminal presents the user with additional questions.

[0364] 6. The user responds to the question.

[0365] 7. The terminal converts the response into text data and sends it to the server.

[0366] 8. The server parses the response and generates a 3D design blueprint.

[0367] 9. The device presents the blueprint to the user and asks for confirmation.

[0368] 10. The user checks the design drawings and indicates any necessary modifications.

[0369] 11. The server reflects the correction instructions and generates the final design.

[0370] 12. The server converts the final design drawing into data for the 3D printer and sends it to the 3D printer via the print control means.

[0371] 13. The 3D printer begins printing the object.

[0372] 14. The device will notify you of the progress and notify you when the print is complete.

[0373] In this way, the system of the present invention can efficiently and accurately perform 3D design and printing based on the user's voice instructions and emotional information.

[0374] Example 2

[0375] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional 3D printing systems require users to have advanced design knowledge and specialized software operation, making it difficult to easily create custom designs. Furthermore, the design process is not optimized to take user feelings into consideration, which can lead to low user satisfaction. Furthermore, the printing progress cannot be checked in real time, making it difficult for users to grasp the progress.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0378] In this invention, the server includes a recognition means for converting voice instructions into text data, a processing means for analyzing the text data to understand the user's intention, and an emotion engine for recognizing the user's emotional state. This allows users to create custom designs based on voice instructions and emotion information without requiring specialized knowledge or skills, and to 3D print them while monitoring the progress in real time.

[0379] A "voice instruction" is an instruction given by the user to the system using voice.

[0380] An "input means" is a device or module that accepts voice instructions.

[0381] The "recognition means" is a processing device or software that converts voice instructions into text data.

[0382] "Processing means" refers to algorithms or engines that analyze text data and understand the user's intent.

[0383] "Generation means" is a function that creates additional questions based on the analysis results.

[0384] An "emotion engine" is a device or software that analyzes voice tone, speech rate, and context to determine a user's emotional state.

[0385] The "presentation means" is a means for visually or audibly presenting the generated question to the user.

[0386] The "response receiving means" is a mechanism for receiving and processing responses from users.

[0387] A "design generator" is a system or software that creates a 3D design blueprint based on user responses.

[0388] The "design confirmation means" is a means for presenting the created 3D design drawing to the user and obtaining confirmation.

[0389] The "conversion means" refers to an algorithm or tool that converts the confirmed design drawings into data for a three-dimensional printer.

[0390] The "control means" is a function that sends the data for the three-dimensional printer to the three-dimensional printer and starts printing.

[0391] "Notification means" refers to a method or device that notifies the user of the progress of the three-dimensional printer in real time.

[0392] A "generative AI model" is an artificial intelligence algorithm that has been trained to perform a specific task.

[0393] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[0394] This system allows users to create and print their desired 3D designs based on voice commands. Its unique feature is that it combines an emotion engine that recognizes the user's emotions to optimize the design process according to the user's emotional state.

[0395] The user issues commands to the system through a voice input means. For example, the user might say, "Please create a custom coffee cup with a new design." The device has a built-in microphone as a voice input means and accepts this voice command. The device then converts the voice into text data using a voice recognition means such as the Google Speech-to-Text API. This converted text data is then sent to the server.

[0396] The server analyzes the received text data using natural language processing tools such as OpenAI's GPT-3 to understand the user's intent. During the analysis process, an emotion engine runs in parallel on the server, which analyzes the voice tone, speaking rate, and context to estimate the user's emotional state. For example, it determines whether the user is excited or relaxed. The emotion engine's inference results are passed to a natural language generation tool, which generates appropriate follow-up questions.

[0397] The server sends the generated follow-up questions as text data to the device, and the device presents the questions to the user. For example, specific questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are displayed. The user answers these questions verbally, and the answers are again converted into text data by the device's voice recognition means and sent to the server. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[0398] Based on the user's responses, the server uses CAD software such as Autodesk Fusion 360 to generate a 3D design blueprint. This blueprint is generated as an STL file and sent to the terminal where it is displayed to the user. The user can review the blueprint and make any necessary corrections by voice. The corrections are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[0399] Finally, when the user approves the design drawing, the server uses a data conversion means to convert it into G-code data for the 3D printer and sends this data to the 3D printer (e.g., Ultimaker 3). The 3D printer prints the object based on the received data. The terminal receives progress information from the 3D printer in real time and notifies the user via a progress notification means. When the print is complete, the terminal notifies the user of the completion.

[0400] For example, if a user says, "Please create a new design for a blue coffee cup with a handle," the voice is converted into text data and sent to the server. The server analyzes the instruction using its emotion engine, and if it determines that the user is relaxed, it generates a specific question such as, "What shade of blue cup would be good?" and presents it to the user via their device. If the user answers, "A deep blue would be good," a more detailed blueprint is generated based on that information.

[0401] An example of a prompt is, "If the user instructs you to create a new design of blue coffee cup with a handle, generate additional questions based on the emotional state analyzed by the emotion engine and explain the process of creating the design."

[0402] In this way, the system of the present invention enables users to quickly create their ideal 3D printed object based on voice instructions and emotional information, without the need for specialized knowledge or skills.

[0403] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0404] Step 1:

[0405] The user inputs a voice command.

[0406] Specific operation: The user speaks into the device's microphone, saying, "Create a new custom coffee cup design."

[0407] Input: User's voice

[0408] Output: Audio data

[0409] Step 2:

[0410] The device converts the voice into text data.

[0411] Specific operation: The device uses the Google Speech-to-Text API to convert voice data into text data in real time.

[0412] Input: Audio data

[0413] Output: Text data

[0414] Step 3:

[0415] The server parses the text data.

[0416] How it works: The server uses a natural language processing engine such as OpenAI's GPT-3 to analyze the text data and understand the user's intent. The analysis results are structured in JSON format.

[0417] Input: Text data

[0418] Output: Analysis results including user intent (JSON format)

[0419] Step 4:

[0420] The server uses an emotion engine to determine the user's emotion.

[0421] How it works: The server's emotion engine analyzes the voice tone, speech rate, and context to estimate the user's emotional state, for example, determining whether the user is excited or relaxed.

[0422] Input: Audio data, text data

[0423] Output: Emotional information including emotional state

[0424] Step 5:

[0425] The server generates a follow-up question.

[0426] Specific operation: Based on the user's intention and emotion information, the server uses natural language generation to generate a follow-up question, such as "Do you need a steering wheel?"

[0427] Input: Analysis results including user intent, emotional information

[0428] Output: Additional questions (text data)

[0429] Step 6:

[0430] The terminal presents the user with a follow-up question.

[0431] Specific operation: The terminal displays the generated follow-up question on the display and presents it to the user.

[0432] Input: Additional question (text data)

[0433] Output: Question displayed

[0434] Step 7:

[0435] The user answers the follow-up questions.

[0436] Specific operation: The user answers the follow-up question by saying something like, "Yes, I need a steering wheel." This answer is input into the device's microphone.

[0437] Input: User's spoken response

[0438] Output: Audio data

[0439] Step 8:

[0440] The device converts the answer into text data.

[0441] Specific operation: The device again uses the Google Speech-to-Text API to convert the voice response into text data, which is then sent to the server.

[0442] Input: Audio data

[0443] Output: Text data

[0444] Step 9:

[0445] The server generates the blueprint.

[0446] How it works: Based on the user's answers, the server generates a 3D design blueprint using CAD software such as Autodesk Fusion 360. This blueprint is saved as an STL file.

[0447] Input: Text data

[0448] Output: 3D design blueprint (STL format)

[0449] Step 10:

[0450] The terminal displays the blueprint to the user.

[0451] Specific operation: The device displays a 3D visualization of the generated design on the screen and allows the user to check it.

[0452] Input: 3D design blueprint (STL format)

[0453] Output: Displayed blueprint

[0454] Step 11:

[0455] The user inputs correction instructions by voice.

[0456] Specific operation: The user checks the design drawings and inputs correction instructions by voice, such as "Please change the shape of the handle."

[0457] Input: User's voice

[0458] Output: Audio data

[0459] Step 12:

[0460] The terminal converts the correction instructions into text data.

[0461] How it works: The device uses the Google Speech-to-Text API to convert the correction instructions into text data, which is then sent to the server.

[0462] Input: Audio data

[0463] Output: Text data

[0464] Step 13:

[0465] The server updates the blueprint to reflect the modifications.

[0466] Specific operation: The server updates the blueprint according to the correction instructions and generates a new STL file, which is then resent to the terminal.

[0467] Input: Text data

[0468] Output: Updated 3D design blueprints (STL format)

[0469] Step 14:

[0470] The user approves the design.

[0471] Specific operation: The user reviews the final design and issues a voice command to approve it: "I approve this."

[0472] Input: User's voice

[0473] Output: Audio data

[0474] Step 15:

[0475] The server converts it into data for 3D printing.

[0476] How it works: The server converts the approved design into G-code data for the 3D printer. This data conforms to the protocol for communicating with the 3D printer.

[0477] Input: 3D design blueprint (STL format)

[0478] Output: G-code data

[0479] Step 16:

[0480] The server sends the data to the 3D printer.

[0481] Specific operation: The server sends G-code data to the 3D printer wirelessly or via a wired connection.

[0482] Input: G-code data

[0483] Output: Data sent to the 3D printer

[0484] Step 17:

[0485] A 3D printer prints an object.

[0486] Specific operation: The 3D printer prints a coffee cup with the specified design based on the received G-code data.

[0487] Input: Data to send to the 3D printer

[0488] Output: Printed matter (coffee cup)

[0489] Step 18:

[0490] The device will notify you of progress in real time.

[0491] Specific operation: The device periodically obtains progress information from the 3D printer and notifies the user of the progress in real time.

[0492] Input: Progress data from the 3D printer

[0493] Output: Progress notification

[0494] Step 19:

[0495] The terminal notifies the user that printing is complete.

[0496] Specific operation: When the print is complete, the terminal notifies the user that "Printing is complete." The user is then ready to remove the finished object.

[0497] Input: Print completion data

[0498] Output: Completion notification

[0499] (Application example 2)

[0500] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0501] Conventional 3D design generation systems generate designs simply based on voice instructions without considering the user's emotional state, which can result in low user satisfaction. Furthermore, in situations where customization based on the user's mood or emotion is required, particularly in services such as food delivery, conventional systems have difficulty responding.

[0502] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's intention, a natural language generation means for generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a 3D design based on the user's responses, a design confirmation means for presenting the 3D design to the user and obtaining confirmation, a data conversion means for converting the confirmed design into data for a 3D printer, a print control means for sending the data for a 3D printer to the 3D printer and starting printing, an emotion analysis means for analyzing the user's voice instructions and emotional state, and an emotion-responsive question generation means for generating follow-up questions based on the user's emotional state based on the emotion analysis means. This enables the generation of customized 3D designs that correspond to the user's mood and emotions.

[0503] "Voice input means" refers to a device or software that accepts voice instructions from the user.

[0504] A "voice recognition means" is a device or algorithm that converts voice instructions into text data.

[0505] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[0506] "Natural language generation means" refers to technology or devices that generate follow-up questions for the user based on the analysis results.

[0507] The "response receiving means" is a device or software that presents the generated question to the user and receives the response.

[0508] "Design generation means" refers to technology or equipment that generates 3D design blueprints based on user responses.

[0509] "Design confirmation means" refers to technology or equipment that presents the generated 3D design drawing to the user for confirmation.

[0510] "Data conversion means" refers to the technology or device that converts the confirmed design drawings into data for a 3D printer.

[0511] "Print control means" refers to the technology or device that sends 3D printer data to the 3D printer and starts printing.

[0512] "Emotion analysis means" refers to technology or devices that analyze the user's voice instructions and emotional state.

[0513] The "emotion-responsive question generation means" is a technique or device that generates a follow-up question according to the user's emotional state based on emotion analysis means.

[0514] The system for implementing this invention is mainly composed of three entities: a server, a terminal, and a user. This system generates 3D designs based on the user's voice instructions and emotional state, and customizes them for specific purposes (such as food delivery).

[0515] First, the user issues a voice instruction to the terminal. For example, the user may say, "Today is a special day, so I'd like a gorgeous package." This voice instruction is accepted by a voice input means installed in the terminal.

[0516] Next, the terminal uses a voice recognition means to convert the voice instruction into text data. The converted text data is sent to the server. The server uses a natural language processing means to analyze the text data and understand the user's intention. At the same time, the server uses an emotion analysis means to analyze the user's emotional state. This analysis determines, for example, whether a user's statement "gorgeous" contains a sense of exhilaration or specialness.

[0517] Based on the analysis results, the server generates a further question using the emotion-aware question generation means. For example, a specific question such as "What color of gorgeous packaging do you like?" is generated. This question is then presented to the user via the terminal.

[0518] The user's response is again received by the voice input means, converted into text data by the voice recognition means, and sent to the server. The server uses the design generation means to generate a 3D design drawing based on this response data. The generated drawing is presented to the user via the terminal, and the user confirms it using the design confirmation means. If necessary, the user can make corrections to the design, and the system will generate the design again.

[0519] Finally, the confirmed design drawing is converted into data for the 3D printer using the data conversion means and sent to the 3D printer through the print control means. The 3D printer prints the object based on the received data.

[0520] This system can generate customized 3D designs according to the user's mood and emotions. For example, if a user says, "I'm feeling good today, so a colorful package would be good," the system will generate a design with vibrant colors based on the user's emotional state. Similarly, if a user says, "Today is a special day, so a gorgeous package would be good," the system will generate a luxurious design with a gold base.

[0521] This allows users to easily use custom-designed packaging that matches their emotional state in food delivery services without requiring specialized knowledge or skills.

[0522] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0523] Step 1:

[0524] The user issues a voice command to the terminal. For example, the user might say, "Today is a special day, so a gorgeous package would be nice." The terminal accepts this voice using a voice input means. The input is the user's voice, and the output is voice data.

[0525] Step 2:

[0526] The device uses a speech recognition means to convert the voice data obtained in step 1 into text data. This conversion is performed using the Google Speech Recognition API or similar. The input is voice data, and the output is text data.

[0527] Step 3:

[0528] Text data is sent from the device to a server. The server uses natural language processing to analyze the text data and understand the user's intent. For example, a generative AI model such as BERT is used for the analysis. The input is text data, and the output is the user's intent as an analysis result.

[0529] Step 4:

[0530] At the same time, the server uses an emotion analysis tool to analyze the user's emotional state from the text data. The analysis takes into account speech tone and speed, and uses an algorithm such as the Emotion API. The input is text data, and the output is the user's emotional state.

[0531] Step 5:

[0532] The server generates follow-up questions using natural language generation based on the analysis results and the emotional state. For example, a question such as "What is a good color for gorgeous packaging?" is generated. A generative AI model (e.g., GPT-3) is used for this generation. The input is the analysis results and the emotional state, and the output is the text of the follow-up question.

[0533] Step 6:

[0534] The server sends the generated follow-up question to the terminal, which then presents it to the user. The user again answers by voice, and the terminal accepts the answer via a voice input means. The input is the text of the follow-up question and the user's voice answer, and the output is voice data.

[0535] Step 7:

[0536] The terminal uses a voice recognition means to convert the user's voice response into text data and transmits it to the server. The input is voice data and the output is text data.

[0537] Step 8:

[0538] The server generates a 3D design blueprint based on the user's answers using a design generation tool. CAD software or a custom design API is used to generate the blueprint. The input is the text data of the user's answers, and the output is a 3D design blueprint.

[0539] Step 9:

[0540] The server sends the generated 3D design blueprint to the terminal, which then presents it to the user. The user can check the blueprint and make corrections as necessary. The input is the 3D design blueprint and user feedback, and the output is the final 3D design blueprint.

[0541] Step 10:

[0542] The server converts the final confirmed design drawing into data for the 3D printer using a data conversion means and sends it to the 3D printer via a print control means. The input is the final 3D design drawing, and the output is the data for the 3D printer.

[0543] Step 11:

[0544] The 3D printer prints the object based on the received data, and the terminal receives the print progress status in real time and notifies the user. The input is the data for the 3D printer, and the output is the object printed by the 3D printer and a progress notification.

[0545] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0546] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0547] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0548] [Second embodiment]

[0549] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0550] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0551] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0552] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0553] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0554] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0555] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0556] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0557] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0558] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0559] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0560] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0561] The specific configuration and functions of a system according to an embodiment of the present invention will be described below.

[0562] This system allows users to design 3D prints by giving voice instructions. The user gives voice instructions to a terminal, which is accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal.

[0563] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, it understands the user's intent and extracts specific design requirements. The server then uses natural language generation to generate any necessary follow-up questions and create text data to ask the user.

[0564] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[0565] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses the design generation means to generate a 3D design blueprint. The generated blueprint is sent to the terminal and requested for user confirmation.

[0566] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[0567] Finally, after the user approves the design drawing, the server uses the data conversion means to convert the design drawing into data for the 3D printer, which is then sent to the 3D printer through the print control means, and printing begins.

[0568] The terminal receives the progress status of the 3D printer in real time and notifies the user of the progress status via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[0569] As a concrete example, consider the case where a user requests, "Please create a new design blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design blueprint based on the response. Finally, once the user confirms and the blueprint is finalized, the server sends the data to a 3D printer, which prints the custom coffee cup.

[0570] In this way, the system of the present invention allows users to 3D print their ideal products simply through voice instructions, without requiring specialized knowledge.

[0571] The processing flow will be explained below.

[0572] Step 1:

[0573] The user speaks to the device, for example, "Create a new custom coffee cup design."

[0574] Step 2:

[0575] The terminal uses the voice input means to accept and record the user's voice.

[0576] Step 3:

[0577] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[0578] Step 4:

[0579] The terminal transmits the converted text data to the server.

[0580] Step 5:

[0581] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[0582] Step 6:

[0583] The server uses natural language generation to generate questions to ascertain any additional details needed from the user, such as "Do I need a handle?" or "Do I need a color preference?"

[0584] Step 7:

[0585] The server transmits the generated question to the terminal as text data.

[0586] Step 8:

[0587] The terminal presents the question to the user via a user interface or audio output.

[0588] Step 9:

[0589] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[0590] Step 10:

[0591] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[0592] Step 11:

[0593] The terminal transmits the text data to the server.

[0594] Step 12:

[0595] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[0596] Step 13:

[0597] The server sends the generated 3D design blueprint to the terminal.

[0598] Step 14:

[0599] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[0600] Step 15:

[0601] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[0602] Step 16:

[0603] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[0604] Step 17:

[0605] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[0606] Step 18:

[0607] The user makes a final check and approves, saying "This is fine."

[0608] Step 19:

[0609] The server converts the final design into data for the 3D printer.

[0610] Step 20:

[0611] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[0612] Step 21:

[0613] The 3D printer begins printing the object based on the received data.

[0614] Step 22:

[0615] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[0616] Step 23:

[0617] When the terminal completes printing, it notifies the user of the completion.

[0618] Example 1

[0619] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] The design process using conventional 3D printing devices requires specialized knowledge and skills, making it difficult for many general users. Furthermore, creating the desired design requires operating complex software and understanding technical terminology, often resulting in design errors and trial and error. The present invention aims to solve these problems by providing a system that allows users to intuitively create 3D designs using voice commands and then print them directly on a 3D printing device.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0622] In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's instructions, a natural language generation means for automatically generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a three-dimensional design drawing based on the user's responses, a design confirmation means for presenting the generated three-dimensional design drawing to the user and obtaining confirmation, a data conversion means for converting the confirmed three-dimensional design drawing into three-dimensional printing data, and a print control means for transmitting the three-dimensional printing data to a three-dimensional printing device and starting printing. This allows users to easily create and print their ideal three-dimensional design through voice instructions without requiring specialized knowledge.

[0623] The "voice input means" is a device or module that has the function of accepting voice instructions from the user.

[0624] "Speech recognition means" refers to software or hardware that has the function of converting voice data into text data.

[0625] "Natural language processing means" refers to software or algorithms that have the ability to analyze text data and understand the user's intent.

[0626] A "natural language generation means" is software or an algorithm that has the function of automatically generating follow-up questions for the user based on the analysis results.

[0627] The "response receiving means" is a device or module that has the function of receiving a user's response to a generated question.

[0628] A "design generator" is software or algorithm capable of generating a three-dimensional design drawing based on user responses.

[0629] The "design confirmation means" is a device or module that has the function of presenting the generated three-dimensional design drawing to the user and obtaining confirmation.

[0630] "Data conversion means" refers to software or algorithms that have the function of converting the confirmed three-dimensional design drawings into data for three-dimensional printing.

[0631] "Printing control means" refers to software or hardware that has the function of sending three-dimensional printing data to a three-dimensional printing device and starting printing.

[0632] The "progress notification means" is a device or module that has the function of notifying the user of the progress of the three-dimensional printing device in real time.

[0633] An "artificial intelligence algorithm" is an algorithm that performs speech recognition and natural language processing using technologies such as machine learning and neural networks.

[0634] The specific configuration and functions of a system for implementing the present invention will be described below. This system allows a user to design a 3D print by issuing voice instructions. The user issues voice instructions to a terminal, which are accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal. Specifically, a microphone is used as the voice input means, and a voice recognition API (e.g., Google Speech-to-Text API) is used as the voice recognition means.

[0635] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, the server understands the user's intent and extracts specific design requirements. A natural language processing model (e.g., OpenAI ChatGPT) is used for the natural language processing.

[0636] The server then uses a natural language generation means to generate any necessary follow-up questions and create text data to ask the user. A natural language generation model (e.g., OpenAI GPT-3) is also used as the natural language generation means.

[0637] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[0638] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses a design generation tool to generate a 3D design drawing. The design generation tool uses a 3D modeling software API (e.g., Autodesk Fusion 360 API). The generated design drawing is sent to the terminal and the user is asked to confirm it.

[0639] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[0640] Finally, after the user approves the design drawing, the server uses a data conversion means to convert the design drawing into data for 3D printing. This data conversion includes converting the data into a standard 3D printer file format (e.g., STL format). This data is sent to the 3D printing device via a print control means, and printing begins. Specifically, a communication module is used to link with the 3D printing device via a network.

[0641] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user of this via the progress status notification means. When printing is completed, the terminal notifies the user of the completion.

[0642] As a concrete example, consider the case where a user requests, "Please create a new design for a blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design for a blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design based on the response. Finally, once the user confirms and the design is finalized, the server sends the data to a 3D printing device, which prints the custom coffee cup.

[0643] In this way, the system of the present invention allows users to three-dimensionally print their ideal products simply through voice instructions, without requiring specialized knowledge.

[0644] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0645] Step 1:

[0646] The user inputs a voice command.

[0647] The user speaks instructions into the device regarding the desired three-dimensional design, for example, "Create a new design for a blue coffee cup with a handle."

[0648] Input: Voice commands

[0649] Output: Audio data recorded on the device

[0650] Specific operation: The user speaks into the device's microphone.

[0651] Step 2:

[0652] The device converts the speech to text.

[0653] The device's built-in speech recognition means (e.g., Google Speech-to-Text API) converts voice input into text data.

[0654] Input: Audio data

[0655] Output: Text data

[0656] Specific operation: The device analyzes the voice data and outputs it as text data.

[0657] Step 3:

[0658] The terminal transmits the text data to the server.

[0659] The terminal transmits the converted text data to the server via the network.

[0660] Input: Text data

[0661] Output: Text data sent to the server

[0662] Specific operation: The device uploads text data to the server via the network.

[0663] Step 4:

[0664] The server analyzes the text data and understands the intent.

[0665] The server analyzes the text data using natural language processing tools (e.g., OpenAI ChatGPT) to understand the user's intent.

[0666] Input: Text data

[0667] Output: Analysis results that understand the user's intent

[0668] Specific operation: The server receives the text data and analyzes it using a natural language processing algorithm.

[0669] Step 5:

[0670] The server generates a follow-up question.

[0671] Based on the analysis, the server generates additional questions to obtain the required details.

[0672] Input: Analysis results

[0673] Output: Additional question text data

[0674] Specific operation: The server generates a question using a natural language generation means.

[0675] Step 6:

[0676] The terminal presents the user with a follow-up question.

[0677] The terminal displays the question received from the server to the user or presents it to the user by voice.

[0678] Input: Additional question text data

[0679] Output: The question presented to the user

[0680] What happens: Your device displays the question on the screen or uses text-to-speech to read the question aloud.

[0681] Step 7:

[0682] The user responds to additional questions.

[0683] The user responds verbally to the questions posed, for example, "The height is 10 cm and the diameter is 8 cm."

[0684] Input: Additional Question

[0685] Output: User's voice response

[0686] Specific operation: The user speaks the response into the device's microphone.

[0687] Step 8:

[0688] The device converts the voice response into text and sends it to the server.

[0689] The terminal converts the user's voice response into text data and sends it to the server.

[0690] Input: Voice response

[0691] Output: User response as text data

[0692] Specific operation: The device converts the voice into text and sends it to the server over the network.

[0693] Step 9:

[0694] The server determines the user's detailed requirements.

[0695] The server then re-analyzes the received text data to determine the user's specific requirements.

[0696] Input: User's detailed text response

[0697] Output: Detailed design requirements

[0698] Specific operation: The server re-analyzes the text data using natural language processing means.

[0699] Step 10:

[0700] The server generates the three-dimensional design drawings.

[0701] The server uses design generation tools (e.g. Autodesk Fusion 360 API) to generate 3D design drawings based on the user's requirements.

[0702] Input: Detailed design requirements

[0703] Output: 3D design data

[0704] Specific operation: The server calls the API to automatically generate a 3D design drawing.

[0705] Step 11:

[0706] The server sends the generated design drawing to the terminal and asks the user for confirmation.

[0707] The server transmits the three-dimensional design drawing data to the terminal and sends a message to the user requesting confirmation.

[0708] Input: 3D design data

[0709] Output: The design presented to the user, along with a message requesting confirmation

[0710] Specific behavior: The device displays a preview of the blueprint and presents a confirmation message to the user.

[0711] Step 12:

[0712] The user checks the design drawings and gives instructions for corrections if necessary.

[0713] The user checks the design drawings and, if any corrections are necessary, gives voice instructions.

[0714] Input: Design drawings and modification instructions

[0715] Output: Audio prompts for corrections

[0716] Specific operation: The user speaks the corrections into the terminal.

[0717] Step 13:

[0718] The terminal converts the correction instructions into text and sends it to the server.

[0719] The terminal converts the user's correction instructions into text data and transmits it to the server.

[0720] Input: Voice commands

[0721] Output: Correction instructions as text data

[0722] Specific operation: The device converts the voice into text and sends it to the server over the network.

[0723] Step 14:

[0724] The server updates the blueprint again.

[0725] The server updates the design drawings again based on the received correction instructions.

[0726] Input: Correction instructions as text data

[0727] Output: Corrected 3D design data

[0728] Specific operation: The server regenerates the design drawing using the design generation means.

[0729] Step 15:

[0730] The user provides final confirmation, and the server converts the data for 3D printing.

[0731] After the user finally approves the design drawing, the server uses a data conversion means to convert it into data for three-dimensional printing (e.g., STL format).

[0732] Input: Final confirmed 3D design data

[0733] Output: 3D printing data

[0734] Specific operation: The server converts the blueprint data into the appropriate file format.

[0735] Step 16:

[0736] The server sends the 3D printing data to the printing device and begins printing.

[0737] The server sends the converted data to the three-dimensional printing device and initiates the printing process using a printing control means.

[0738] Input: 3D printing data

[0739] Output: Print execution

[0740] Specific operations: The server sends data to the printing device and issues a command to start printing.

[0741] Step 17:

[0742] The terminal receives real-time progress and notifies the user.

[0743] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user using the progress status notification means.

[0744] Input: Progress data

[0745] Output: Progress notification

[0746] Specific operation: The device receives the progress data, displays it on the screen, and notifies the user.

[0747] Step 18:

[0748] The terminal notifies the user that printing is complete.

[0749] When printing is complete, the terminal notifies the user of the completion.

[0750] Input: Print completion data

[0751] Output: Completion notification

[0752] Specific operation: The device will display a completion message and notify you with an alarm sound or vibration.

[0753] (Application example 1)

[0754] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Conventional 3D design systems required specialized knowledge and skills, making it difficult to quickly design and manufacture custom parts on the factory floor. This limited factory productivity and flexibility. Furthermore, voice-based systems were unable to accurately grasp the user's intent and were unable to meet the user's specific design requirements. Furthermore, there was no way to know the progress in real time, making it difficult to constantly monitor whether the print was running properly.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0757] In this invention, the server includes an input device that accepts voice instructions, a voice analysis device that converts the voice instructions into text data, a natural language processing device that analyzes the text data to understand the user's intent, a response acceptance device that presents generated questions to the user and accepts their responses, a design generation device that generates a 3D design blueprint based on the user's responses, a data conversion device that converts the confirmed blueprint into 3D printer data, a print control device that sends the 3D printer data to the 3D printer and starts printing, and a means for issuing design instructions for commercial parts in a factory via voice instructions, generating blueprints based on those instructions, and directly controlling printing on the 3D printer. This allows even users without specialized knowledge to quickly and easily design custom parts in a factory using voice instructions and manufacture them on a 3D printer. Furthermore, progress is monitored in real time and the user is notified, making it easy to confirm whether printing is proceeding normally.

[0758] An "input device" is a means for receiving voice instructions from a user.

[0759] A "voice analysis device" is a means for converting voice instructions into text data.

[0760] A "natural language processing device" is a means for analyzing text data and understanding the user's intent.

[0761] A "natural language generation device" is a means for generating follow-up questions for the user based on the analysis results.

[0762] The "response accepting device" is a means for presenting the generated question to the user and accepting the response thereto.

[0763] A "design drawing generator" is a means for generating a 3D design drawing based on a user's response.

[0764] The "verification device" is a means for presenting the generated 3D design blueprint to the user and obtaining their confirmation.

[0765] The "data conversion device" is a means for converting the confirmed design drawings into data for a 3D printer.

[0766] A "print control device" is a means for sending 3D printer data to a 3D printer and starting printing.

[0767] A "progress notification device" is a means of notifying the user of the progress of a 3D printer in real time.

[0768] MODE FOR CARRYING OUT THE INVENTION

[0769] In this embodiment, we will explain the specific configuration and operation of a system that uses voice instructions to create 3D designs in a factory and quickly create custom parts. The main components used here and their roles are as follows:

[0770] Key components of the system

[0771] 1. Input Devices

[0772] This is a device with a microphone that allows the user to input voice commands. The microphone built into a smartphone or tablet is used.

[0773] 2. Voice analysis device

[0774] This is a device that has the function of converting user voice instructions into text data. Specifically, it uses Google's voice recognition service to convert voice into text.

[0775] 3. Natural Language Processing Systems

[0776] This is a device that analyzes text data sent from a voice analysis device and understands what the user is looking for. It uses natural language processing algorithms, especially generative AI models.

[0777] 4. Natural language generator

[0778] This is a device that generates follow-up questions for the user based on the analysis results of the natural language processing device. For example, it creates questions to confirm details of the design.

[0779] 5. Response and Reception Device

[0780] This device presents the generated questions to the user and collects their responses. It reuses voice input and converts it into text data.

[0781] 6. Design Drawing Generator

[0782] This device generates 3D design blueprints based on user responses. Design software is used to create blueprints based on specific shapes and dimensions.

[0783] 7. Verification Device

[0784] This device presents the generated 3D design blueprint to the user for confirmation, and displays it on a smartphone or tablet screen.

[0785] 8. Data Conversion Device

[0786] This is a device that converts the confirmed design drawings into data for 3D printers. It converts the design drawings into a format that can be read by 3D printers, such as STL format.

[0787] 9. Printing Control Device

[0788] This device sends the converted 3D data to the 3D printer and starts printing.

[0789] 10. Progress notification device

[0790] This device notifies users of the progress of their 3D printer in real time, sending notifications to their smartphones and tablets.

[0791] Example of operation

[0792] If an employee voice-inputs, "I'd like to create a custom wrench with a new design," the input device collects the speech and the speech analysis device converts it into text data. The natural language processing device analyzes the data, and the natural language generation device generates follow-up questions about the specific size and shape of the wrench. For example, a question such as, "What is the length and width of the wrench?" is generated. If the user responds, "It's 20 cm long and 5 cm wide," the response acceptance device collects the data and converts it back into text.

[0793] The blueprint generation device then generates a 3D design blueprint based on the response, and the confirmation device presents the blueprint to the user. Once the user reviews and approves the blueprint, the data conversion device converts the blueprint into data for the 3D printer. This data is then sent to the 3D printer via the print control device, and the printing process begins. The user is notified of the printing progress in real time via the progress notification device.

[0794] Prompt Sentence Examples

[0795] "Imagine an application where a user gives voice instructions for designing custom tools to be used in a factory, converts those instructions into text, sends it to a server, and generates a 3D design. As a concrete example, explain the process by which an employee might say into their smartphone, 'I want to create a new custom wrench design,' and then specify the exact size and shape of the wrench by voice."

[0796] In a specific implementation of this invention, factory robots can implement voice-activated design instructions and rapidly 3D print custom-made parts, improving productivity and providing a flexible manufacturing process.

[0797] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0798] Step 1:

[0799] The user issues a voice command into the input device, which then collects the voice as a digital voice signal. This digital voice signal is the input for step 1, and its output is digital voice data. Specifically, the smartphone's microphone collects the user's voice and stores it as digital data.

[0800] Step 2:

[0801] The device uses a voice analyzer to convert the digital voice data from step 1 into text data. The input is digital voice data and the output is text data. Specifically, the Google speech recognition service API is used to convert the collected voice data into text format.

[0802] Step 3:

[0803] The server uses a natural language processing device to analyze the text data from step 2 and understand the user's intent. The input is text data, and the output is the analysis result that includes the user's intent. Specifically, the generative AI model analyzes the text data and understands the "custom wrench design."

[0804] Step 4:

[0805] The server uses a natural language generator to generate follow-up questions for the user based on the analysis results. The input is the analysis results, and the output is text data containing the follow-up questions. Specifically, the AI ​​model generates questions such as "Please specify the length and width of the wrench."

[0806] Step 5:

[0807] The terminal uses the response reception device to present the follow-up question of step 4 to the user and collects the user's voice response. The input is the text data of the follow-up question, and the output is the user's voice response. Specifically, the question is displayed on the smartphone screen, and the user's voice response is collected by the microphone.

[0808] Step 6:

[0809] The terminal uses a voice analysis device to convert the response voice from step 5 back into text data. The input is the response voice, and the output is the response text data. Specifically, the voice recognition service API is used again to convert the voice response into text.

[0810] Step 7:

[0811] The server uses a blueprint generator to generate a 3D design blueprint based on the response text data from step 6. The input is the response text data, and the output is the 3D design blueprint. Specifically, dedicated design software is used to generate a blueprint of the wrench based on the specified length and width.

[0812] Step 8:

[0813] The server uses a confirmation device to present the generated 3D design blueprint to the user and ask for confirmation. The input is the 3D design blueprint, and the output is the user's confirmation response. Specifically, the blueprint is displayed on the smartphone screen, and an interface is provided for the user to confirm.

[0814] Step 9:

[0815] The server converts the user-confirmed design drawings into data for a 3D printer using a data converter. The input is the confirmed 3D design drawing, and the output is data for a 3D printer. Specifically, the design drawing is converted into a data format for a 3D printer, such as STL format.

[0816] Step 10:

[0817] The server uses the print control device to send the 3D printer data from step 9 to the 3D printer and start printing. The input is the data for the 3D printer, and the output is an instruction to start printing. In concrete terms, the 3D printer receives the data and prints based on the specified design.

[0818] Step 11:

[0819] The server uses a progress notification device to notify the user of the 3D printer's progress in real time. The input is the 3D printer's progress data, and the output is a progress notification. Specifically, the progress is notified to the smartphone, allowing the user to check the print progress.

[0820] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0821] This invention relates to a system that creates and prints 3D designs based on user voice commands. In particular, it is characterized by its ability to optimize the design process according to the user's emotional state by incorporating an emotion engine that recognizes the user's emotions.

[0822] The system starts with the user inputting a voice command into the terminal, such as "Please create a new custom coffee cup design." The voice command is then accepted using a voice input means.

[0823] The terminal uses a speech recognition means to convert the user's speech into text data. This text data is sent to a server. The server uses a natural language processing means to analyze the text data and understand the user's intention. During this analysis process, the server runs an emotion engine in parallel to recognize the user's emotions.

[0824] The emotion engine analyzes the user's tone, speech rate, context, etc. to estimate the user's emotional state. For example, it can identify whether the user is excited or relaxed. The estimated emotional information is passed to the natural language generator, which generates appropriate follow-up questions.

[0825] The server sends the generated follow-up questions to the terminal as text data and presents them to the user. For example, questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are presented to the user, and the user's answers are converted back into text data via speech recognition. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[0826] The server generates a 3D design blueprint based on the user's responses. The generated blueprint is sent to the device and displayed to the user. The user can check the blueprint and make any necessary corrections by voice. The correction instructions are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[0827] Finally, when the user approves the design drawing, the server converts it into data for the 3D printer using the data conversion means and sends it to the 3D printer via the print control means. The 3D printer prints an object based on the received data.

[0828] The terminal also receives the progress status of the 3D printer in real time and notifies the user via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[0829] For example, if a user says, "Please create a new design of blue coffee cup with a handle," the device converts the voice to text data and sends it to the server, which uses its emotion engine to analyze the instruction and determine whether the user is excited or relaxed.

[0830] If the emotion engine determines that the user is relaxed, it generates specific questions such as, "What shade of blue cup would you like?" and presents them to the user via the device. As the user provides more detailed answers, the blueprint is generated in a form that is closer to the user's ideal.

[0831] In this way, the system of the present invention allows users to 3D print their ideal product based on voice instructions and emotional information, without the need for specialized knowledge or skills and in a short amount of time.

[0832] The processing flow will be explained below.

[0833] Step 1:

[0834] The user speaks to the device, for example, "Create a new custom coffee cup design."

[0835] Step 2:

[0836] The terminal uses the voice input means to accept and record the user's voice.

[0837] Step 3:

[0838] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[0839] Step 4:

[0840] The terminal transmits the converted text data to the server.

[0841] Step 5:

[0842] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[0843] Step 6:

[0844] The server runs an emotion engine that analyzes the tone, speaking rate, and context of the received voice data to estimate the user's emotional state.

[0845] Step 7:

[0846] The server uses natural language generation to generate additional questions that take emotional information into account. For example, if the user is relaxed, the server generates questions such as "Do you need a handle?" or "Do you have a color preference?"

[0847] Step 8:

[0848] The server transmits the generated question to the terminal as text data.

[0849] Step 9:

[0850] The terminal presents the question to the user via a user interface or audio output.

[0851] Step 10:

[0852] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[0853] Step 11:

[0854] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[0855] Step 12:

[0856] The terminal transmits the text data to the server.

[0857] Step 13:

[0858] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[0859] Step 14:

[0860] The server sends the generated 3D design blueprint to the terminal.

[0861] Step 15:

[0862] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[0863] Step 16:

[0864] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[0865] Step 17:

[0866] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[0867] Step 18:

[0868] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[0869] Step 19:

[0870] The user makes a final check and approves, saying "This is fine."

[0871] Step 20:

[0872] The server converts the final design into data for the 3D printer.

[0873] Step 21:

[0874] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[0875] Step 22:

[0876] The 3D printer begins printing the object based on the received data.

[0877] Step 23:

[0878] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[0879] Step 24:

[0880] When the terminal completes printing, it notifies the user of the completion.

[0881] Specific examples

[0882] For example, if the user says "Create a new design for a blue coffee cup with a handle," the following happens:

[0883] 1. The user issues a voice command.

[0884] 2. The device converts the voice into text data and sends it to the server.

[0885] 3. The server parses the text data and understands that the user wants to create a custom coffee cup.

[0886] 4. The server uses the emotion engine to estimate the user's emotional state and generate follow-up questions.

[0887] 5. The terminal presents the user with additional questions.

[0888] 6. The user responds to the question.

[0889] 7. The terminal converts the response into text data and sends it to the server.

[0890] 8. The server parses the response and generates a 3D design blueprint.

[0891] 9. The device presents the blueprint to the user and asks for confirmation.

[0892] 10. The user checks the design drawings and indicates any necessary modifications.

[0893] 11. The server reflects the correction instructions and generates the final design.

[0894] 12. The server converts the final design drawing into data for the 3D printer and sends it to the 3D printer via the print control means.

[0895] 13. The 3D printer begins printing the object.

[0896] 14. The device will notify you of the progress and notify you when the print is complete.

[0897] In this way, the system of the present invention can efficiently and accurately perform 3D design and printing based on the user's voice instructions and emotional information.

[0898] Example 2

[0899] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0900] Conventional 3D printing systems require users to have advanced design knowledge and specialized software operation, making it difficult to easily create custom designs. Furthermore, the design process is not optimized to take user feelings into consideration, which can lead to low user satisfaction. Furthermore, the printing progress cannot be checked in real time, making it difficult for users to grasp the progress.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0902] In this invention, the server includes a recognition means for converting voice instructions into text data, a processing means for analyzing the text data to understand the user's intention, and an emotion engine for recognizing the user's emotional state. This allows users to create custom designs based on voice instructions and emotion information without requiring specialized knowledge or skills, and to 3D print them while monitoring the progress in real time.

[0903] A "voice instruction" is an instruction given by the user to the system using voice.

[0904] An "input means" is a device or module that accepts voice instructions.

[0905] The "recognition means" is a processing device or software that converts voice instructions into text data.

[0906] "Processing means" refers to algorithms or engines that analyze text data and understand the user's intent.

[0907] "Generation means" is a function that creates additional questions based on the analysis results.

[0908] An "emotion engine" is a device or software that analyzes voice tone, speech rate, and context to determine a user's emotional state.

[0909] The "presentation means" is a means for visually or audibly presenting the generated question to the user.

[0910] The "response receiving means" is a mechanism for receiving and processing responses from users.

[0911] A "design generator" is a system or software that creates a 3D design blueprint based on user responses.

[0912] The "design confirmation means" is a means for presenting the created 3D design drawing to the user and obtaining confirmation.

[0913] The "conversion means" refers to an algorithm or tool that converts the confirmed design drawings into data for a three-dimensional printer.

[0914] The "control means" is a function that sends the data for the three-dimensional printer to the three-dimensional printer and starts printing.

[0915] "Notification means" refers to a method or device that notifies the user of the progress of the three-dimensional printer in real time.

[0916] A "generative AI model" is an artificial intelligence algorithm that has been trained to perform a specific task.

[0917] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[0918] This system allows users to create and print their desired 3D designs based on voice commands. Its unique feature is that it combines an emotion engine that recognizes the user's emotions to optimize the design process according to the user's emotional state.

[0919] The user issues commands to the system through a voice input means. For example, the user might say, "Please create a custom coffee cup with a new design." The device has a built-in microphone as a voice input means and accepts this voice command. The device then converts the voice into text data using a voice recognition means such as the Google Speech-to-Text API. This converted text data is then sent to the server.

[0920] The server analyzes the received text data using natural language processing tools such as OpenAI's GPT-3 to understand the user's intent. During the analysis process, an emotion engine runs in parallel on the server, which analyzes the voice tone, speaking rate, and context to estimate the user's emotional state. For example, it determines whether the user is excited or relaxed. The emotion engine's inference results are passed to a natural language generation tool, which generates appropriate follow-up questions.

[0921] The server sends the generated follow-up questions as text data to the device, and the device presents the questions to the user. For example, specific questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are displayed. The user answers these questions verbally, and the answers are again converted into text data by the device's voice recognition means and sent to the server. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[0922] Based on the user's responses, the server uses CAD software such as Autodesk Fusion 360 to generate a 3D design blueprint. This blueprint is generated as an STL file and sent to the terminal where it is displayed to the user. The user can review the blueprint and make any necessary corrections by voice. The corrections are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[0923] Finally, when the user approves the design drawing, the server uses a data conversion means to convert it into G-code data for the 3D printer and sends this data to the 3D printer (e.g., Ultimaker 3). The 3D printer prints the object based on the received data. The terminal receives progress information from the 3D printer in real time and notifies the user via a progress notification means. When the print is complete, the terminal notifies the user of the completion.

[0924] For example, if a user says, "Please create a new design for a blue coffee cup with a handle," the voice is converted into text data and sent to the server. The server analyzes the instruction using its emotion engine, and if it determines that the user is relaxed, it generates a specific question such as, "What shade of blue cup would be good?" and presents it to the user via their device. If the user answers, "A deep blue would be good," a more detailed blueprint is generated based on that information.

[0925] An example of a prompt is, "If the user instructs you to create a new design of blue coffee cup with a handle, generate additional questions based on the emotional state analyzed by the emotion engine and explain the process of creating the design."

[0926] In this way, the system of the present invention enables users to quickly create their ideal 3D printed object based on voice instructions and emotional information, without the need for specialized knowledge or skills.

[0927] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0928] Step 1:

[0929] The user inputs a voice command.

[0930] Specific operation: The user speaks into the device's microphone, saying, "Create a new custom coffee cup design."

[0931] Input: User's voice

[0932] Output: Audio data

[0933] Step 2:

[0934] The device converts the voice into text data.

[0935] Specific operation: The device uses the Google Speech-to-Text API to convert voice data into text data in real time.

[0936] Input: Audio data

[0937] Output: Text data

[0938] Step 3:

[0939] The server parses the text data.

[0940] How it works: The server uses a natural language processing engine such as OpenAI's GPT-3 to analyze the text data and understand the user's intent. The analysis results are structured in JSON format.

[0941] Input: Text data

[0942] Output: Analysis results including user intent (JSON format)

[0943] Step 4:

[0944] The server uses an emotion engine to determine the user's emotion.

[0945] How it works: The server's emotion engine analyzes the voice tone, speech rate, and context to estimate the user's emotional state, for example, determining whether the user is excited or relaxed.

[0946] Input: Audio data, text data

[0947] Output: Emotional information including emotional state

[0948] Step 5:

[0949] The server generates a follow-up question.

[0950] Specific operation: Based on the user's intention and emotion information, the server uses natural language generation to generate a follow-up question, such as "Do you need a steering wheel?"

[0951] Input: Analysis results including user intent, emotional information

[0952] Output: Additional questions (text data)

[0953] Step 6:

[0954] The terminal presents the user with a follow-up question.

[0955] Specific operation: The terminal displays the generated follow-up question on the display and presents it to the user.

[0956] Input: Additional question (text data)

[0957] Output: Question displayed

[0958] Step 7:

[0959] The user answers the follow-up questions.

[0960] Specific operation: The user answers the follow-up question by saying something like, "Yes, I need a steering wheel." This answer is input into the device's microphone.

[0961] Input: User's spoken response

[0962] Output: Audio data

[0963] Step 8:

[0964] The device converts the answer into text data.

[0965] Specific operation: The device again uses the Google Speech-to-Text API to convert the voice response into text data, which is then sent to the server.

[0966] Input: Audio data

[0967] Output: Text data

[0968] Step 9:

[0969] The server generates the blueprint.

[0970] How it works: Based on the user's answers, the server generates a 3D design blueprint using CAD software such as Autodesk Fusion 360. This blueprint is saved as an STL file.

[0971] Input: Text data

[0972] Output: 3D design blueprint (STL format)

[0973] Step 10:

[0974] The terminal displays the blueprint to the user.

[0975] Specific operation: The device displays a 3D visualization of the generated design on the screen and allows the user to check it.

[0976] Input: 3D design blueprint (STL format)

[0977] Output: Displayed blueprint

[0978] Step 11:

[0979] The user inputs correction instructions by voice.

[0980] Specific operation: The user checks the design drawings and inputs correction instructions by voice, such as "Please change the shape of the handle."

[0981] Input: User's voice

[0982] Output: Audio data

[0983] Step 12:

[0984] The terminal converts the correction instructions into text data.

[0985] How it works: The device uses the Google Speech-to-Text API to convert the correction instructions into text data, which is then sent to the server.

[0986] Input: Audio data

[0987] Output: Text data

[0988] Step 13:

[0989] The server updates the blueprint to reflect the modifications.

[0990] Specific operation: The server updates the blueprint according to the correction instructions and generates a new STL file, which is then resent to the terminal.

[0991] Input: Text data

[0992] Output: Updated 3D design blueprints (STL format)

[0993] Step 14:

[0994] The user approves the design.

[0995] Specific operation: The user reviews the final design and issues a voice command to approve it: "I approve this."

[0996] Input: User's voice

[0997] Output: Audio data

[0998] Step 15:

[0999] The server converts it into data for 3D printing.

[1000] How it works: The server converts the approved design into G-code data for the 3D printer. This data conforms to the protocol for communicating with the 3D printer.

[1001] Input: 3D design blueprint (STL format)

[1002] Output: G-code data

[1003] Step 16:

[1004] The server sends the data to the 3D printer.

[1005] Specific operation: The server sends G-code data to the 3D printer wirelessly or via a wired connection.

[1006] Input: G-code data

[1007] Output: Data sent to the 3D printer

[1008] Step 17:

[1009] A 3D printer prints an object.

[1010] Specific operation: The 3D printer prints a coffee cup with the specified design based on the received G-code data.

[1011] Input: Data to send to the 3D printer

[1012] Output: Printed matter (coffee cup)

[1013] Step 18:

[1014] The device will notify you of progress in real time.

[1015] Specific operation: The device periodically obtains progress information from the 3D printer and notifies the user of the progress in real time.

[1016] Input: Progress data from the 3D printer

[1017] Output: Progress notification

[1018] Step 19:

[1019] The terminal notifies the user that printing is complete.

[1020] Specific operation: When the print is complete, the terminal notifies the user that "Printing is complete." The user is then ready to remove the finished object.

[1021] Input: Print completion data

[1022] Output: Completion notification

[1023] (Application example 2)

[1024] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1025] Conventional 3D design generation systems generate designs simply based on voice instructions without considering the user's emotional state, which can result in low user satisfaction. Furthermore, in situations where customization based on the user's mood or emotion is required, particularly in services such as food delivery, conventional systems have difficulty responding.

[1026] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's intention, a natural language generation means for generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a 3D design based on the user's responses, a design confirmation means for presenting the 3D design to the user and obtaining confirmation, a data conversion means for converting the confirmed design into data for a 3D printer, a print control means for sending the data for a 3D printer to the 3D printer and starting printing, an emotion analysis means for analyzing the user's voice instructions and emotional state, and an emotion-responsive question generation means for generating follow-up questions based on the user's emotional state based on the emotion analysis means. This enables the generation of customized 3D designs that correspond to the user's mood and emotions.

[1027] "Voice input means" refers to a device or software that accepts voice instructions from the user.

[1028] A "voice recognition means" is a device or algorithm that converts voice instructions into text data.

[1029] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[1030] "Natural language generation means" refers to technology or devices that generate follow-up questions for the user based on the analysis results.

[1031] The "response receiving means" is a device or software that presents the generated question to the user and receives the response.

[1032] "Design generation means" refers to technology or equipment that generates 3D design blueprints based on user responses.

[1033] "Design confirmation means" refers to technology or equipment that presents the generated 3D design drawing to the user for confirmation.

[1034] "Data conversion means" refers to the technology or device that converts the confirmed design drawings into data for a 3D printer.

[1035] "Print control means" refers to the technology or device that sends 3D printer data to the 3D printer and starts printing.

[1036] "Emotion analysis means" refers to technology or devices that analyze the user's voice instructions and emotional state.

[1037] The "emotion-responsive question generation means" is a technique or device that generates a follow-up question according to the user's emotional state based on emotion analysis means.

[1038] The system for implementing this invention is mainly composed of three entities: a server, a terminal, and a user. This system generates 3D designs based on the user's voice instructions and emotional state, and customizes them for specific purposes (such as food delivery).

[1039] First, the user issues a voice instruction to the terminal. For example, the user may say, "Today is a special day, so I'd like a gorgeous package." This voice instruction is accepted by a voice input means installed in the terminal.

[1040] Next, the terminal uses a voice recognition means to convert the voice instruction into text data. The converted text data is sent to the server. The server uses a natural language processing means to analyze the text data and understand the user's intention. At the same time, the server uses an emotion analysis means to analyze the user's emotional state. This analysis determines, for example, whether a user's statement "gorgeous" contains a sense of exhilaration or specialness.

[1041] Based on the analysis results, the server generates a further question using the emotion-aware question generation means. For example, a specific question such as "What color of gorgeous packaging do you like?" is generated. This question is then presented to the user via the terminal.

[1042] The user's response is again received by the voice input means, converted into text data by the voice recognition means, and sent to the server. The server uses the design generation means to generate a 3D design drawing based on this response data. The generated drawing is presented to the user via the terminal, and the user confirms it using the design confirmation means. If necessary, the user can make corrections to the design, and the system will generate the design again.

[1043] Finally, the confirmed design drawing is converted into data for the 3D printer using the data conversion means and sent to the 3D printer through the print control means. The 3D printer prints the object based on the received data.

[1044] This system can generate customized 3D designs according to the user's mood and emotions. For example, if a user says, "I'm feeling good today, so a colorful package would be good," the system will generate a design with vibrant colors based on the user's emotional state. Similarly, if a user says, "Today is a special day, so a gorgeous package would be good," the system will generate a luxurious design with a gold base.

[1045] This allows users to easily use custom-designed packaging that matches their emotional state in food delivery services without requiring specialized knowledge or skills.

[1046] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1047] Step 1:

[1048] The user issues a voice command to the terminal. For example, the user might say, "Today is a special day, so a gorgeous package would be nice." The terminal accepts this voice using a voice input means. The input is the user's voice, and the output is voice data.

[1049] Step 2:

[1050] The device uses a speech recognition means to convert the voice data obtained in step 1 into text data. This conversion is performed using the Google Speech Recognition API or similar. The input is voice data, and the output is text data.

[1051] Step 3:

[1052] Text data is sent from the device to a server. The server uses natural language processing to analyze the text data and understand the user's intent. For example, a generative AI model such as BERT is used for the analysis. The input is text data, and the output is the user's intent as an analysis result.

[1053] Step 4:

[1054] At the same time, the server uses an emotion analysis tool to analyze the user's emotional state from the text data. The analysis takes into account speech tone and speed, and uses an algorithm such as the Emotion API. The input is text data, and the output is the user's emotional state.

[1055] Step 5:

[1056] The server generates follow-up questions using natural language generation based on the analysis results and the emotional state. For example, a question such as "What is a good color for gorgeous packaging?" is generated. A generative AI model (e.g., GPT-3) is used for this generation. The input is the analysis results and the emotional state, and the output is the text of the follow-up question.

[1057] Step 6:

[1058] The server sends the generated follow-up question to the terminal, which then presents it to the user. The user again answers by voice, and the terminal accepts the answer via a voice input means. The input is the text of the follow-up question and the user's voice answer, and the output is voice data.

[1059] Step 7:

[1060] The terminal uses a voice recognition means to convert the user's voice response into text data and transmits it to the server. The input is voice data and the output is text data.

[1061] Step 8:

[1062] The server generates a 3D design blueprint based on the user's answers using a design generation tool. CAD software or a custom design API is used to generate the blueprint. The input is the text data of the user's answers, and the output is a 3D design blueprint.

[1063] Step 9:

[1064] The server sends the generated 3D design blueprint to the terminal, which then presents it to the user. The user can check the blueprint and make corrections as necessary. The input is the 3D design blueprint and user feedback, and the output is the final 3D design blueprint.

[1065] Step 10:

[1066] The server converts the final confirmed design drawing into data for the 3D printer using a data conversion means and sends it to the 3D printer via a print control means. The input is the final 3D design drawing, and the output is the data for the 3D printer.

[1067] Step 11:

[1068] The 3D printer prints the object based on the received data, and the terminal receives the print progress status in real time and notifies the user. The input is the data for the 3D printer, and the output is the object printed by the 3D printer and a progress notification.

[1069] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1070] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1071] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1072] [Third embodiment]

[1073] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1074] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1075] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1076] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1077] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1078] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1079] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1080] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1081] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1082] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1083] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1084] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1085] The specific configuration and functions of a system according to an embodiment of the present invention will be described below.

[1086] This system allows users to design 3D prints by giving voice instructions. The user gives voice instructions to a terminal, which is accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal.

[1087] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, it understands the user's intent and extracts specific design requirements. The server then uses natural language generation to generate any necessary follow-up questions and create text data to ask the user.

[1088] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[1089] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses the design generation means to generate a 3D design blueprint. The generated blueprint is sent to the terminal and requested for user confirmation.

[1090] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[1091] Finally, after the user approves the design drawing, the server uses the data conversion means to convert the design drawing into data for the 3D printer, which is then sent to the 3D printer through the print control means, and printing begins.

[1092] The terminal receives the progress status of the 3D printer in real time and notifies the user of the progress status via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[1093] As a concrete example, consider the case where a user requests, "Please create a new design blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design blueprint based on the response. Finally, once the user confirms and the blueprint is finalized, the server sends the data to a 3D printer, which prints the custom coffee cup.

[1094] In this way, the system of the present invention allows users to 3D print their ideal products simply through voice instructions, without requiring specialized knowledge.

[1095] The processing flow will be explained below.

[1096] Step 1:

[1097] The user speaks to the device, for example, "Create a new custom coffee cup design."

[1098] Step 2:

[1099] The terminal uses the voice input means to accept and record the user's voice.

[1100] Step 3:

[1101] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[1102] Step 4:

[1103] The terminal transmits the converted text data to the server.

[1104] Step 5:

[1105] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[1106] Step 6:

[1107] The server uses natural language generation to generate questions to ascertain any additional details needed from the user, such as "Do I need a handle?" or "Do I need a color preference?"

[1108] Step 7:

[1109] The server transmits the generated question to the terminal as text data.

[1110] Step 8:

[1111] The terminal presents the question to the user via a user interface or audio output.

[1112] Step 9:

[1113] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[1114] Step 10:

[1115] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[1116] Step 11:

[1117] The terminal transmits the text data to the server.

[1118] Step 12:

[1119] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[1120] Step 13:

[1121] The server sends the generated 3D design blueprint to the terminal.

[1122] Step 14:

[1123] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[1124] Step 15:

[1125] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[1126] Step 16:

[1127] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[1128] Step 17:

[1129] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[1130] Step 18:

[1131] The user makes a final check and approves, saying "This is fine."

[1132] Step 19:

[1133] The server converts the final design into data for the 3D printer.

[1134] Step 20:

[1135] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[1136] Step 21:

[1137] The 3D printer begins printing the object based on the received data.

[1138] Step 22:

[1139] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[1140] Step 23:

[1141] When the terminal completes printing, it notifies the user of the completion.

[1142] Example 1

[1143] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1144] The design process using conventional 3D printing devices requires specialized knowledge and skills, making it difficult for many general users. Furthermore, creating the desired design requires operating complex software and understanding technical terminology, often resulting in design errors and trial and error. The present invention aims to solve these problems by providing a system that allows users to intuitively create 3D designs using voice commands and then print them directly on a 3D printing device.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1146] In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's instructions, a natural language generation means for automatically generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a three-dimensional design drawing based on the user's responses, a design confirmation means for presenting the generated three-dimensional design drawing to the user and obtaining confirmation, a data conversion means for converting the confirmed three-dimensional design drawing into three-dimensional printing data, and a print control means for transmitting the three-dimensional printing data to a three-dimensional printing device and starting printing. This allows users to easily create and print their ideal three-dimensional design through voice instructions without requiring specialized knowledge.

[1147] The "voice input means" is a device or module that has the function of accepting voice instructions from the user.

[1148] "Speech recognition means" refers to software or hardware that has the function of converting voice data into text data.

[1149] "Natural language processing means" refers to software or algorithms that have the ability to analyze text data and understand the user's intent.

[1150] A "natural language generation means" is software or an algorithm that has the function of automatically generating follow-up questions for the user based on the analysis results.

[1151] The "response receiving means" is a device or module that has the function of receiving a user's response to a generated question.

[1152] A "design generator" is software or algorithm capable of generating a three-dimensional design drawing based on user responses.

[1153] The "design confirmation means" is a device or module that has the function of presenting the generated three-dimensional design drawing to the user and obtaining confirmation.

[1154] "Data conversion means" refers to software or algorithms that have the function of converting the confirmed three-dimensional design drawings into data for three-dimensional printing.

[1155] "Printing control means" refers to software or hardware that has the function of sending three-dimensional printing data to a three-dimensional printing device and starting printing.

[1156] The "progress notification means" is a device or module that has the function of notifying the user of the progress of the three-dimensional printing device in real time.

[1157] An "artificial intelligence algorithm" is an algorithm that performs speech recognition and natural language processing using technologies such as machine learning and neural networks.

[1158] The specific configuration and functions of a system for implementing the present invention will be described below. This system allows a user to design a 3D print by issuing voice instructions. The user issues voice instructions to a terminal, which are accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal. Specifically, a microphone is used as the voice input means, and a voice recognition API (e.g., Google Speech-to-Text API) is used as the voice recognition means.

[1159] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, the server understands the user's intent and extracts specific design requirements. A natural language processing model (e.g., OpenAI ChatGPT) is used for the natural language processing.

[1160] The server then uses a natural language generation means to generate any necessary follow-up questions and create text data to ask the user. A natural language generation model (e.g., OpenAI GPT-3) is also used as the natural language generation means.

[1161] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[1162] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses a design generation tool to generate a 3D design drawing. The design generation tool uses a 3D modeling software API (e.g., Autodesk Fusion 360 API). The generated design drawing is sent to the terminal and the user is asked to confirm it.

[1163] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[1164] Finally, after the user approves the design drawing, the server uses a data conversion means to convert the design drawing into data for 3D printing. This data conversion includes converting the data into a standard 3D printer file format (e.g., STL format). This data is sent to the 3D printing device via a print control means, and printing begins. Specifically, a communication module is used to link with the 3D printing device via a network.

[1165] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user of this via the progress status notification means. When printing is completed, the terminal notifies the user of the completion.

[1166] As a concrete example, consider the case where a user requests, "Please create a new design for a blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design for a blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design based on the response. Finally, once the user confirms and the design is finalized, the server sends the data to a 3D printing device, which prints the custom coffee cup.

[1167] In this way, the system of the present invention allows users to three-dimensionally print their ideal products simply through voice instructions, without requiring specialized knowledge.

[1168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1169] Step 1:

[1170] The user inputs a voice command.

[1171] The user speaks instructions into the device regarding the desired three-dimensional design, for example, "Create a new design for a blue coffee cup with a handle."

[1172] Input: Voice commands

[1173] Output: Audio data recorded on the device

[1174] Specific operation: The user speaks into the device's microphone.

[1175] Step 2:

[1176] The device converts the speech to text.

[1177] The device's built-in speech recognition means (e.g., Google Speech-to-Text API) converts voice input into text data.

[1178] Input: Audio data

[1179] Output: Text data

[1180] Specific operation: The device analyzes the voice data and outputs it as text data.

[1181] Step 3:

[1182] The terminal transmits the text data to the server.

[1183] The terminal transmits the converted text data to the server via the network.

[1184] Input: Text data

[1185] Output: Text data sent to the server

[1186] Specific operation: The device uploads text data to the server via the network.

[1187] Step 4:

[1188] The server analyzes the text data and understands the intent.

[1189] The server analyzes the text data using natural language processing tools (e.g., OpenAI ChatGPT) to understand the user's intent.

[1190] Input: Text data

[1191] Output: Analysis results that understand the user's intent

[1192] Specific operation: The server receives the text data and analyzes it using a natural language processing algorithm.

[1193] Step 5:

[1194] The server generates a follow-up question.

[1195] Based on the analysis, the server generates additional questions to obtain the required details.

[1196] Input: Analysis results

[1197] Output: Additional question text data

[1198] Specific operation: The server generates a question using a natural language generation means.

[1199] Step 6:

[1200] The terminal presents the user with a follow-up question.

[1201] The terminal displays the question received from the server to the user or presents it to the user by voice.

[1202] Input: Additional question text data

[1203] Output: The question presented to the user

[1204] What happens: Your device displays the question on the screen or uses text-to-speech to read the question aloud.

[1205] Step 7:

[1206] The user responds to additional questions.

[1207] The user responds verbally to the questions posed, for example, "The height is 10 cm and the diameter is 8 cm."

[1208] Input: Additional Question

[1209] Output: User's voice response

[1210] Specific operation: The user speaks the response into the device's microphone.

[1211] Step 8:

[1212] The device converts the voice response into text and sends it to the server.

[1213] The terminal converts the user's voice response into text data and sends it to the server.

[1214] Input: Voice response

[1215] Output: User response as text data

[1216] Specific operation: The device converts the voice into text and sends it to the server over the network.

[1217] Step 9:

[1218] The server determines the user's detailed requirements.

[1219] The server then re-analyzes the received text data to determine the user's specific requirements.

[1220] Input: User's detailed text response

[1221] Output: Detailed design requirements

[1222] Specific operation: The server re-analyzes the text data using natural language processing means.

[1223] Step 10:

[1224] The server generates the three-dimensional design drawings.

[1225] The server uses design generation tools (e.g. Autodesk Fusion 360 API) to generate 3D design drawings based on the user's requirements.

[1226] Input: Detailed design requirements

[1227] Output: 3D design data

[1228] Specific operation: The server calls the API to automatically generate a 3D design drawing.

[1229] Step 11:

[1230] The server sends the generated design drawing to the terminal and asks the user for confirmation.

[1231] The server transmits the three-dimensional design drawing data to the terminal and sends a message to the user requesting confirmation.

[1232] Input: 3D design data

[1233] Output: The design presented to the user, along with a message requesting confirmation

[1234] Specific behavior: The device displays a preview of the blueprint and presents a confirmation message to the user.

[1235] Step 12:

[1236] The user checks the design drawings and gives instructions for corrections if necessary.

[1237] The user checks the design drawings and, if any corrections are necessary, gives voice instructions.

[1238] Input: Design drawings and modification instructions

[1239] Output: Audio prompts for corrections

[1240] Specific operation: The user speaks the corrections into the terminal.

[1241] Step 13:

[1242] The terminal converts the correction instructions into text and sends it to the server.

[1243] The terminal converts the user's correction instructions into text data and transmits it to the server.

[1244] Input: Voice commands

[1245] Output: Correction instructions as text data

[1246] Specific operation: The device converts the voice into text and sends it to the server over the network.

[1247] Step 14:

[1248] The server updates the blueprint again.

[1249] The server updates the design drawings again based on the received correction instructions.

[1250] Input: Correction instructions as text data

[1251] Output: Corrected 3D design data

[1252] Specific operation: The server regenerates the design drawing using the design generation means.

[1253] Step 15:

[1254] The user provides final confirmation, and the server converts the data for 3D printing.

[1255] After the user finally approves the design drawing, the server uses a data conversion means to convert it into data for three-dimensional printing (e.g., STL format).

[1256] Input: Final confirmed 3D design data

[1257] Output: 3D printing data

[1258] Specific operation: The server converts the blueprint data into the appropriate file format.

[1259] Step 16:

[1260] The server sends the 3D printing data to the printing device and begins printing.

[1261] The server sends the converted data to the three-dimensional printing device and initiates the printing process using a printing control means.

[1262] Input: 3D printing data

[1263] Output: Print execution

[1264] Specific operations: The server sends data to the printing device and issues a command to start printing.

[1265] Step 17:

[1266] The terminal receives real-time progress and notifies the user.

[1267] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user using the progress status notification means.

[1268] Input: Progress data

[1269] Output: Progress notification

[1270] Specific operation: The device receives the progress data, displays it on the screen, and notifies the user.

[1271] Step 18:

[1272] The terminal notifies the user that printing is complete.

[1273] When printing is complete, the terminal notifies the user of the completion.

[1274] Input: Print completion data

[1275] Output: Completion notification

[1276] Specific operation: The device will display a completion message and notify you with an alarm sound or vibration.

[1277] (Application example 1)

[1278] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1279] Conventional 3D design systems required specialized knowledge and skills, making it difficult to quickly design and manufacture custom parts on the factory floor. This limited factory productivity and flexibility. Furthermore, voice-based systems were unable to accurately grasp the user's intent and were unable to meet the user's specific design requirements. Furthermore, there was no way to know the progress in real time, making it difficult to constantly monitor whether the print was running properly.

[1280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1281] In this invention, the server includes an input device that accepts voice instructions, a voice analysis device that converts the voice instructions into text data, a natural language processing device that analyzes the text data to understand the user's intent, a response acceptance device that presents generated questions to the user and accepts their responses, a design generation device that generates a 3D design blueprint based on the user's responses, a data conversion device that converts the confirmed blueprint into 3D printer data, a print control device that sends the 3D printer data to the 3D printer and starts printing, and a means for issuing design instructions for commercial parts in a factory via voice instructions, generating blueprints based on those instructions, and directly controlling printing on the 3D printer. This allows even users without specialized knowledge to quickly and easily design custom parts in a factory using voice instructions and manufacture them on a 3D printer. Furthermore, progress is monitored in real time and the user is notified, making it easy to confirm whether printing is proceeding normally.

[1282] An "input device" is a means for receiving voice instructions from a user.

[1283] A "voice analysis device" is a means for converting voice instructions into text data.

[1284] A "natural language processing device" is a means for analyzing text data and understanding the user's intent.

[1285] A "natural language generation device" is a means for generating follow-up questions for the user based on the analysis results.

[1286] The "response accepting device" is a means for presenting the generated question to the user and accepting the response thereto.

[1287] A "design drawing generator" is a means for generating a 3D design drawing based on a user's response.

[1288] The "verification device" is a means for presenting the generated 3D design blueprint to the user and obtaining their confirmation.

[1289] The "data conversion device" is a means for converting the confirmed design drawings into data for a 3D printer.

[1290] A "print control device" is a means for sending 3D printer data to a 3D printer and starting printing.

[1291] A "progress notification device" is a means of notifying the user of the progress of a 3D printer in real time.

[1292] MODE FOR CARRYING OUT THE INVENTION

[1293] In this embodiment, we will explain the specific configuration and operation of a system that uses voice instructions to create 3D designs in a factory and quickly create custom parts. The main components used here and their roles are as follows:

[1294] Key components of the system

[1295] 1. Input Devices

[1296] This is a device with a microphone that allows the user to input voice commands. The microphone built into a smartphone or tablet is used.

[1297] 2. Voice analysis device

[1298] This is a device that has the function of converting user voice instructions into text data. Specifically, it uses Google's voice recognition service to convert voice into text.

[1299] 3. Natural Language Processing Systems

[1300] This is a device that analyzes text data sent from a voice analysis device and understands what the user is looking for. It uses natural language processing algorithms, especially generative AI models.

[1301] 4. Natural language generator

[1302] This is a device that generates follow-up questions for the user based on the analysis results of the natural language processing device. For example, it creates questions to confirm details of the design.

[1303] 5. Response and Reception Device

[1304] This device presents the generated questions to the user and collects their responses. It reuses voice input and converts it into text data.

[1305] 6. Design Drawing Generator

[1306] This device generates 3D design blueprints based on user responses. Design software is used to create blueprints based on specific shapes and dimensions.

[1307] 7. Verification Device

[1308] This device presents the generated 3D design blueprint to the user for confirmation, and displays it on a smartphone or tablet screen.

[1309] 8. Data Conversion Device

[1310] This is a device that converts the confirmed design drawings into data for 3D printers. It converts the design drawings into a format that can be read by 3D printers, such as STL format.

[1311] 9. Printing Control Device

[1312] This device sends the converted 3D data to the 3D printer and starts printing.

[1313] 10. Progress notification device

[1314] This device notifies users of the progress of their 3D printer in real time, sending notifications to their smartphones and tablets.

[1315] Example of operation

[1316] If an employee voice-inputs, "I'd like to create a custom wrench with a new design," the input device collects the speech and the speech analysis device converts it into text data. The natural language processing device analyzes the data, and the natural language generation device generates follow-up questions about the specific size and shape of the wrench. For example, a question such as, "What is the length and width of the wrench?" is generated. If the user responds, "It's 20 cm long and 5 cm wide," the response acceptance device collects the data and converts it back into text.

[1317] The blueprint generation device then generates a 3D design blueprint based on the response, and the confirmation device presents the blueprint to the user. Once the user reviews and approves the blueprint, the data conversion device converts the blueprint into data for the 3D printer. This data is then sent to the 3D printer via the print control device, and the printing process begins. The user is notified of the printing progress in real time via the progress notification device.

[1318] Prompt Sentence Examples

[1319] "Imagine an application where a user gives voice instructions for designing custom tools to be used in a factory, converts those instructions into text, sends it to a server, and generates a 3D design. As a concrete example, explain the process by which an employee might say into their smartphone, 'I want to create a new custom wrench design,' and then specify the exact size and shape of the wrench by voice."

[1320] In a specific implementation of this invention, factory robots can implement voice-activated design instructions and rapidly 3D print custom-made parts, improving productivity and providing a flexible manufacturing process.

[1321] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1322] Step 1:

[1323] The user issues a voice command into the input device, which then collects the voice as a digital voice signal. This digital voice signal is the input for step 1, and its output is digital voice data. Specifically, the smartphone's microphone collects the user's voice and stores it as digital data.

[1324] Step 2:

[1325] The device uses a voice analyzer to convert the digital voice data from step 1 into text data. The input is digital voice data and the output is text data. Specifically, the Google speech recognition service API is used to convert the collected voice data into text format.

[1326] Step 3:

[1327] The server uses a natural language processing device to analyze the text data from step 2 and understand the user's intent. The input is text data, and the output is the analysis result that includes the user's intent. Specifically, the generative AI model analyzes the text data and understands the "custom wrench design."

[1328] Step 4:

[1329] The server uses a natural language generator to generate follow-up questions for the user based on the analysis results. The input is the analysis results, and the output is text data containing the follow-up questions. Specifically, the AI ​​model generates questions such as "Please specify the length and width of the wrench."

[1330] Step 5:

[1331] The terminal uses the response reception device to present the follow-up question of step 4 to the user and collects the user's voice response. The input is the text data of the follow-up question, and the output is the user's voice response. Specifically, the question is displayed on the smartphone screen, and the user's voice response is collected by the microphone.

[1332] Step 6:

[1333] The terminal uses a voice analysis device to convert the response voice from step 5 back into text data. The input is the response voice, and the output is the response text data. Specifically, the voice recognition service API is used again to convert the voice response into text.

[1334] Step 7:

[1335] The server uses a blueprint generator to generate a 3D design blueprint based on the response text data from step 6. The input is the response text data, and the output is the 3D design blueprint. Specifically, dedicated design software is used to generate a blueprint of the wrench based on the specified length and width.

[1336] Step 8:

[1337] The server uses a confirmation device to present the generated 3D design blueprint to the user and ask for confirmation. The input is the 3D design blueprint, and the output is the user's confirmation response. Specifically, the blueprint is displayed on the smartphone screen, and an interface is provided for the user to confirm.

[1338] Step 9:

[1339] The server converts the user-confirmed design drawings into data for a 3D printer using a data converter. The input is the confirmed 3D design drawing, and the output is data for a 3D printer. Specifically, the design drawing is converted into a data format for a 3D printer, such as STL format.

[1340] Step 10:

[1341] The server uses the print control device to send the 3D printer data from step 9 to the 3D printer and start printing. The input is the data for the 3D printer, and the output is an instruction to start printing. In concrete terms, the 3D printer receives the data and prints based on the specified design.

[1342] Step 11:

[1343] The server uses a progress notification device to notify the user of the 3D printer's progress in real time. The input is the 3D printer's progress data, and the output is a progress notification. Specifically, the progress is notified to the smartphone, allowing the user to check the print progress.

[1344] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1345] This invention relates to a system that creates and prints 3D designs based on user voice commands. In particular, it is characterized by its ability to optimize the design process according to the user's emotional state by incorporating an emotion engine that recognizes the user's emotions.

[1346] The system starts with the user inputting a voice command into the terminal, such as "Please create a new custom coffee cup design." The voice command is then accepted using a voice input means.

[1347] The terminal uses a speech recognition means to convert the user's speech into text data. This text data is sent to a server. The server uses a natural language processing means to analyze the text data and understand the user's intention. During this analysis process, the server runs an emotion engine in parallel to recognize the user's emotions.

[1348] The emotion engine analyzes the user's tone, speech rate, context, etc. to estimate the user's emotional state. For example, it can identify whether the user is excited or relaxed. The estimated emotional information is passed to the natural language generator, which generates appropriate follow-up questions.

[1349] The server sends the generated follow-up questions to the terminal as text data and presents them to the user. For example, questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are presented to the user, and the user's answers are converted back into text data via speech recognition. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[1350] The server generates a 3D design blueprint based on the user's responses. The generated blueprint is sent to the device and displayed to the user. The user can check the blueprint and make any necessary corrections by voice. The correction instructions are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[1351] Finally, when the user approves the design drawing, the server converts it into data for the 3D printer using the data conversion means and sends it to the 3D printer via the print control means. The 3D printer prints an object based on the received data.

[1352] The terminal also receives the progress status of the 3D printer in real time and notifies the user via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[1353] For example, if a user says, "Please create a new design of blue coffee cup with a handle," the device converts the voice to text data and sends it to the server, which uses its emotion engine to analyze the instruction and determine whether the user is excited or relaxed.

[1354] If the emotion engine determines that the user is relaxed, it generates specific questions such as, "What shade of blue cup would you like?" and presents them to the user via the device. As the user provides more detailed answers, the blueprint is generated in a form that is closer to the user's ideal.

[1355] In this way, the system of the present invention allows users to 3D print their ideal product based on voice instructions and emotional information, without the need for specialized knowledge or skills and in a short amount of time.

[1356] The processing flow will be explained below.

[1357] Step 1:

[1358] The user speaks to the device, for example, "Create a new custom coffee cup design."

[1359] Step 2:

[1360] The terminal uses the voice input means to accept and record the user's voice.

[1361] Step 3:

[1362] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[1363] Step 4:

[1364] The terminal transmits the converted text data to the server.

[1365] Step 5:

[1366] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[1367] Step 6:

[1368] The server runs an emotion engine that analyzes the tone, speaking rate, and context of the received voice data to estimate the user's emotional state.

[1369] Step 7:

[1370] The server uses natural language generation to generate additional questions that take emotional information into account. For example, if the user is relaxed, the server generates questions such as "Do you need a handle?" or "Do you have a color preference?"

[1371] Step 8:

[1372] The server transmits the generated question to the terminal as text data.

[1373] Step 9:

[1374] The terminal presents the question to the user via a user interface or audio output.

[1375] Step 10:

[1376] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[1377] Step 11:

[1378] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[1379] Step 12:

[1380] The terminal transmits the text data to the server.

[1381] Step 13:

[1382] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[1383] Step 14:

[1384] The server sends the generated 3D design blueprint to the terminal.

[1385] Step 15:

[1386] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[1387] Step 16:

[1388] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[1389] Step 17:

[1390] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[1391] Step 18:

[1392] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[1393] Step 19:

[1394] The user makes a final check and approves, saying "This is fine."

[1395] Step 20:

[1396] The server converts the final design into data for the 3D printer.

[1397] Step 21:

[1398] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[1399] Step 22:

[1400] The 3D printer begins printing the object based on the received data.

[1401] Step 23:

[1402] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[1403] Step 24:

[1404] When the terminal completes printing, it notifies the user of the completion.

[1405] Specific examples

[1406] For example, if the user says "Create a new design for a blue coffee cup with a handle," the following happens:

[1407] 1. The user issues a voice command.

[1408] 2. The device converts the voice into text data and sends it to the server.

[1409] 3. The server parses the text data and understands that the user wants to create a custom coffee cup.

[1410] 4. The server uses the emotion engine to estimate the user's emotional state and generate follow-up questions.

[1411] 5. The terminal presents the user with additional questions.

[1412] 6. The user responds to the question.

[1413] 7. The terminal converts the response into text data and sends it to the server.

[1414] 8. The server parses the response and generates a 3D design blueprint.

[1415] 9. The device presents the blueprint to the user and asks for confirmation.

[1416] 10. The user checks the design drawings and indicates any necessary modifications.

[1417] 11. The server reflects the correction instructions and generates the final design.

[1418] 12. The server converts the final design drawing into data for the 3D printer and sends it to the 3D printer via the print control means.

[1419] 13. The 3D printer begins printing the object.

[1420] 14. The device will notify you of the progress and notify you when the print is complete.

[1421] In this way, the system of the present invention can efficiently and accurately perform 3D design and printing based on the user's voice instructions and emotional information.

[1422] Example 2

[1423] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1424] Conventional 3D printing systems require users to have advanced design knowledge and specialized software operation, making it difficult to easily create custom designs. Furthermore, the design process is not optimized to take user feelings into consideration, which can lead to low user satisfaction. Furthermore, the printing progress cannot be checked in real time, making it difficult for users to grasp the progress.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1426] In this invention, the server includes a recognition means for converting voice instructions into text data, a processing means for analyzing the text data to understand the user's intention, and an emotion engine for recognizing the user's emotional state. This allows users to create custom designs based on voice instructions and emotion information without requiring specialized knowledge or skills, and to 3D print them while monitoring the progress in real time.

[1427] A "voice instruction" is an instruction given by the user to the system using voice.

[1428] An "input means" is a device or module that accepts voice instructions.

[1429] The "recognition means" is a processing device or software that converts voice instructions into text data.

[1430] "Processing means" refers to algorithms or engines that analyze text data and understand the user's intent.

[1431] "Generation means" is a function that creates additional questions based on the analysis results.

[1432] An "emotion engine" is a device or software that analyzes voice tone, speech rate, and context to determine a user's emotional state.

[1433] The "presentation means" is a means for visually or audibly presenting the generated question to the user.

[1434] The "response receiving means" is a mechanism for receiving and processing responses from users.

[1435] A "design generator" is a system or software that creates a 3D design blueprint based on user responses.

[1436] The "design confirmation means" is a means for presenting the created 3D design drawing to the user and obtaining confirmation.

[1437] The "conversion means" refers to an algorithm or tool that converts the confirmed design drawings into data for a three-dimensional printer.

[1438] The "control means" is a function that sends the data for the three-dimensional printer to the three-dimensional printer and starts printing.

[1439] "Notification means" refers to a method or device that notifies the user of the progress of the three-dimensional printer in real time.

[1440] A "generative AI model" is an artificial intelligence algorithm that has been trained to perform a specific task.

[1441] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[1442] This system allows users to create and print their desired 3D designs based on voice commands. Its unique feature is that it combines an emotion engine that recognizes the user's emotions to optimize the design process according to the user's emotional state.

[1443] The user issues commands to the system through a voice input means. For example, the user might say, "Please create a custom coffee cup with a new design." The device has a built-in microphone as a voice input means and accepts this voice command. The device then converts the voice into text data using a voice recognition means such as the Google Speech-to-Text API. This converted text data is then sent to the server.

[1444] The server analyzes the received text data using natural language processing tools such as OpenAI's GPT-3 to understand the user's intent. During the analysis process, an emotion engine runs in parallel on the server, which analyzes the voice tone, speaking rate, and context to estimate the user's emotional state. For example, it determines whether the user is excited or relaxed. The emotion engine's inference results are passed to a natural language generation tool, which generates appropriate follow-up questions.

[1445] The server sends the generated follow-up questions as text data to the device, and the device presents the questions to the user. For example, specific questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are displayed. The user answers these questions verbally, and the answers are again converted into text data by the device's voice recognition means and sent to the server. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[1446] Based on the user's responses, the server uses CAD software such as Autodesk Fusion 360 to generate a 3D design blueprint. This blueprint is generated as an STL file and sent to the terminal where it is displayed to the user. The user can review the blueprint and make any necessary corrections by voice. The corrections are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[1447] Finally, when the user approves the design drawing, the server uses a data conversion means to convert it into G-code data for the 3D printer and sends this data to the 3D printer (e.g., Ultimaker 3). The 3D printer prints the object based on the received data. The terminal receives progress information from the 3D printer in real time and notifies the user via a progress notification means. When the print is complete, the terminal notifies the user of the completion.

[1448] For example, if a user says, "Please create a new design for a blue coffee cup with a handle," the voice is converted into text data and sent to the server. The server analyzes the instruction using its emotion engine, and if it determines that the user is relaxed, it generates a specific question such as, "What shade of blue cup would be good?" and presents it to the user via their device. If the user answers, "A deep blue would be good," a more detailed blueprint is generated based on that information.

[1449] An example of a prompt is, "If the user instructs you to create a new design of blue coffee cup with a handle, generate additional questions based on the emotional state analyzed by the emotion engine and explain the process of creating the design."

[1450] In this way, the system of the present invention enables users to quickly create their ideal 3D printed object based on voice instructions and emotional information, without the need for specialized knowledge or skills.

[1451] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1452] Step 1:

[1453] The user inputs a voice command.

[1454] Specific operation: The user speaks into the device's microphone, saying, "Create a new custom coffee cup design."

[1455] Input: User's voice

[1456] Output: Audio data

[1457] Step 2:

[1458] The device converts the voice into text data.

[1459] Specific operation: The device uses the Google Speech-to-Text API to convert voice data into text data in real time.

[1460] Input: Audio data

[1461] Output: Text data

[1462] Step 3:

[1463] The server parses the text data.

[1464] How it works: The server uses a natural language processing engine such as OpenAI's GPT-3 to analyze the text data and understand the user's intent. The analysis results are structured in JSON format.

[1465] Input: Text data

[1466] Output: Analysis results including user intent (JSON format)

[1467] Step 4:

[1468] The server uses an emotion engine to determine the user's emotion.

[1469] How it works: The server's emotion engine analyzes the voice tone, speech rate, and context to estimate the user's emotional state, for example, determining whether the user is excited or relaxed.

[1470] Input: Audio data, text data

[1471] Output: Emotional information including emotional state

[1472] Step 5:

[1473] The server generates a follow-up question.

[1474] Specific operation: Based on the user's intention and emotion information, the server uses natural language generation to generate a follow-up question, such as "Do you need a steering wheel?"

[1475] Input: Analysis results including user intent, emotional information

[1476] Output: Additional questions (text data)

[1477] Step 6:

[1478] The terminal presents the user with a follow-up question.

[1479] Specific operation: The terminal displays the generated follow-up question on the display and presents it to the user.

[1480] Input: Additional question (text data)

[1481] Output: Question displayed

[1482] Step 7:

[1483] The user answers the follow-up questions.

[1484] Specific operation: The user answers the follow-up question by saying something like, "Yes, I need a steering wheel." This answer is input into the device's microphone.

[1485] Input: User's spoken response

[1486] Output: Audio data

[1487] Step 8:

[1488] The device converts the answer into text data.

[1489] Specific operation: The device again uses the Google Speech-to-Text API to convert the voice response into text data, which is then sent to the server.

[1490] Input: Audio data

[1491] Output: Text data

[1492] Step 9:

[1493] The server generates the blueprint.

[1494] How it works: Based on the user's answers, the server generates a 3D design blueprint using CAD software such as Autodesk Fusion 360. This blueprint is saved as an STL file.

[1495] Input: Text data

[1496] Output: 3D design blueprint (STL format)

[1497] Step 10:

[1498] The terminal displays the blueprint to the user.

[1499] Specific operation: The device displays a 3D visualization of the generated design on the screen and allows the user to check it.

[1500] Input: 3D design blueprint (STL format)

[1501] Output: Displayed blueprint

[1502] Step 11:

[1503] The user inputs correction instructions by voice.

[1504] Specific operation: The user checks the design drawings and inputs correction instructions by voice, such as "Please change the shape of the handle."

[1505] Input: User's voice

[1506] Output: Audio data

[1507] Step 12:

[1508] The terminal converts the correction instructions into text data.

[1509] How it works: The device uses the Google Speech-to-Text API to convert the correction instructions into text data, which is then sent to the server.

[1510] Input: Audio data

[1511] Output: Text data

[1512] Step 13:

[1513] The server updates the blueprint to reflect the modifications.

[1514] Specific operation: The server updates the blueprint according to the correction instructions and generates a new STL file, which is then resent to the terminal.

[1515] Input: Text data

[1516] Output: Updated 3D design blueprints (STL format)

[1517] Step 14:

[1518] The user approves the design.

[1519] Specific operation: The user reviews the final design and issues a voice command to approve it: "I approve this."

[1520] Input: User's voice

[1521] Output: Audio data

[1522] Step 15:

[1523] The server converts it into data for 3D printing.

[1524] How it works: The server converts the approved design into G-code data for the 3D printer. This data conforms to the protocol for communicating with the 3D printer.

[1525] Input: 3D design blueprint (STL format)

[1526] Output: G-code data

[1527] Step 16:

[1528] The server sends the data to the 3D printer.

[1529] Specific operation: The server sends G-code data to the 3D printer wirelessly or via a wired connection.

[1530] Input: G-code data

[1531] Output: Data sent to the 3D printer

[1532] Step 17:

[1533] A 3D printer prints an object.

[1534] Specific operation: The 3D printer prints a coffee cup with the specified design based on the received G-code data.

[1535] Input: Data to send to the 3D printer

[1536] Output: Printed matter (coffee cup)

[1537] Step 18:

[1538] The device will notify you of progress in real time.

[1539] Specific operation: The device periodically obtains progress information from the 3D printer and notifies the user of the progress in real time.

[1540] Input: Progress data from the 3D printer

[1541] Output: Progress notification

[1542] Step 19:

[1543] The terminal notifies the user that printing is complete.

[1544] Specific operation: When the print is complete, the terminal notifies the user that "Printing is complete." The user is then ready to remove the finished object.

[1545] Input: Print completion data

[1546] Output: Completion notification

[1547] (Application example 2)

[1548] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1549] Conventional 3D design generation systems generate designs simply based on voice instructions without considering the user's emotional state, which can result in low user satisfaction. Furthermore, in situations where customization based on the user's mood or emotion is required, particularly in services such as food delivery, conventional systems have difficulty responding.

[1550] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's intention, a natural language generation means for generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a 3D design based on the user's responses, a design confirmation means for presenting the 3D design to the user and obtaining confirmation, a data conversion means for converting the confirmed design into data for a 3D printer, a print control means for sending the data for a 3D printer to the 3D printer and starting printing, an emotion analysis means for analyzing the user's voice instructions and emotional state, and an emotion-responsive question generation means for generating follow-up questions based on the user's emotional state based on the emotion analysis means. This enables the generation of customized 3D designs that correspond to the user's mood and emotions.

[1551] "Voice input means" refers to a device or software that accepts voice instructions from the user.

[1552] A "voice recognition means" is a device or algorithm that converts voice instructions into text data.

[1553] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[1554] "Natural language generation means" refers to technology or devices that generate follow-up questions for the user based on the analysis results.

[1555] The "response receiving means" is a device or software that presents the generated question to the user and receives the response.

[1556] "Design generation means" refers to technology or equipment that generates 3D design blueprints based on user responses.

[1557] "Design confirmation means" refers to technology or equipment that presents the generated 3D design drawing to the user for confirmation.

[1558] "Data conversion means" refers to the technology or device that converts the confirmed design drawings into data for a 3D printer.

[1559] "Print control means" refers to the technology or device that sends 3D printer data to the 3D printer and starts printing.

[1560] "Emotion analysis means" refers to technology or devices that analyze the user's voice instructions and emotional state.

[1561] The "emotion-responsive question generation means" is a technique or device that generates a follow-up question according to the user's emotional state based on emotion analysis means.

[1562] The system for implementing this invention is mainly composed of three entities: a server, a terminal, and a user. This system generates 3D designs based on the user's voice instructions and emotional state, and customizes them for specific purposes (such as food delivery).

[1563] First, the user issues a voice instruction to the terminal. For example, the user may say, "Today is a special day, so I'd like a gorgeous package." This voice instruction is accepted by a voice input means installed in the terminal.

[1564] Next, the terminal uses a voice recognition means to convert the voice instruction into text data. The converted text data is sent to the server. The server uses a natural language processing means to analyze the text data and understand the user's intention. At the same time, the server uses an emotion analysis means to analyze the user's emotional state. This analysis determines, for example, whether a user's statement "gorgeous" contains a sense of exhilaration or specialness.

[1565] Based on the analysis results, the server generates a further question using the emotion-aware question generation means. For example, a specific question such as "What color of gorgeous packaging do you like?" is generated. This question is then presented to the user via the terminal.

[1566] The user's response is again received by the voice input means, converted into text data by the voice recognition means, and sent to the server. The server uses the design generation means to generate a 3D design drawing based on this response data. The generated drawing is presented to the user via the terminal, and the user confirms it using the design confirmation means. If necessary, the user can make corrections to the design, and the system will generate the design again.

[1567] Finally, the confirmed design drawing is converted into data for the 3D printer using the data conversion means and sent to the 3D printer through the print control means. The 3D printer prints the object based on the received data.

[1568] This system can generate customized 3D designs according to the user's mood and emotions. For example, if a user says, "I'm feeling good today, so a colorful package would be good," the system will generate a design with vibrant colors based on the user's emotional state. Similarly, if a user says, "Today is a special day, so a gorgeous package would be good," the system will generate a luxurious design with a gold base.

[1569] This allows users to easily use custom-designed packaging that matches their emotional state in food delivery services without requiring specialized knowledge or skills.

[1570] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1571] Step 1:

[1572] The user issues a voice command to the terminal. For example, the user might say, "Today is a special day, so a gorgeous package would be nice." The terminal accepts this voice using a voice input means. The input is the user's voice, and the output is voice data.

[1573] Step 2:

[1574] The device uses a speech recognition means to convert the voice data obtained in step 1 into text data. This conversion is performed using the Google Speech Recognition API or similar. The input is voice data, and the output is text data.

[1575] Step 3:

[1576] Text data is sent from the device to a server. The server uses natural language processing to analyze the text data and understand the user's intent. For example, a generative AI model such as BERT is used for the analysis. The input is text data, and the output is the user's intent as an analysis result.

[1577] Step 4:

[1578] At the same time, the server uses an emotion analysis tool to analyze the user's emotional state from the text data. The analysis takes into account speech tone and speed, and uses an algorithm such as the Emotion API. The input is text data, and the output is the user's emotional state.

[1579] Step 5:

[1580] The server generates follow-up questions using natural language generation based on the analysis results and the emotional state. For example, a question such as "What is a good color for gorgeous packaging?" is generated. A generative AI model (e.g., GPT-3) is used for this generation. The input is the analysis results and the emotional state, and the output is the text of the follow-up question.

[1581] Step 6:

[1582] The server sends the generated follow-up question to the terminal, which then presents it to the user. The user again answers by voice, and the terminal accepts the answer via a voice input means. The input is the text of the follow-up question and the user's voice answer, and the output is voice data.

[1583] Step 7:

[1584] The terminal uses a voice recognition means to convert the user's voice response into text data and transmits it to the server. The input is voice data and the output is text data.

[1585] Step 8:

[1586] The server generates a 3D design blueprint based on the user's answers using a design generation tool. CAD software or a custom design API is used to generate the blueprint. The input is the text data of the user's answers, and the output is a 3D design blueprint.

[1587] Step 9:

[1588] The server sends the generated 3D design blueprint to the terminal, which then presents it to the user. The user can check the blueprint and make corrections as necessary. The input is the 3D design blueprint and user feedback, and the output is the final 3D design blueprint.

[1589] Step 10:

[1590] The server converts the final confirmed design drawing into data for the 3D printer using a data conversion means and sends it to the 3D printer via a print control means. The input is the final 3D design drawing, and the output is the data for the 3D printer.

[1591] Step 11:

[1592] The 3D printer prints the object based on the received data, and the terminal receives the print progress status in real time and notifies the user. The input is the data for the 3D printer, and the output is the object printed by the 3D printer and a progress notification.

[1593] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1594] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1595] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1596] [Fourth embodiment]

[1597] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1598] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1599] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1600] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1601] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1602] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1603] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1604] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1605] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1606] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1607] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1608] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1609] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1610] The specific configuration and functions of a system according to an embodiment of the present invention will be described below.

[1611] This system allows users to design 3D prints by giving voice instructions. The user gives voice instructions to a terminal, which is accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal.

[1612] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, it understands the user's intent and extracts specific design requirements. The server then uses natural language generation to generate any necessary follow-up questions and create text data to ask the user.

[1613] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[1614] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses the design generation means to generate a 3D design blueprint. The generated blueprint is sent to the terminal and requested for user confirmation.

[1615] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[1616] Finally, after the user approves the design drawing, the server uses the data conversion means to convert the design drawing into data for the 3D printer, which is then sent to the 3D printer through the print control means, and printing begins.

[1617] The terminal receives the progress status of the 3D printer in real time and notifies the user of the progress status via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[1618] As a concrete example, consider the case where a user requests, "Please create a new design blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design blueprint based on the response. Finally, once the user confirms and the blueprint is finalized, the server sends the data to a 3D printer, which prints the custom coffee cup.

[1619] In this way, the system of the present invention allows users to 3D print their ideal products simply through voice instructions, without requiring specialized knowledge.

[1620] The processing flow will be explained below.

[1621] Step 1:

[1622] The user speaks to the device, for example, "Create a new custom coffee cup design."

[1623] Step 2:

[1624] The terminal uses the voice input means to accept and record the user's voice.

[1625] Step 3:

[1626] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[1627] Step 4:

[1628] The terminal transmits the converted text data to the server.

[1629] Step 5:

[1630] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[1631] Step 6:

[1632] The server uses natural language generation to generate questions to ascertain any additional details needed from the user, such as "Do I need a handle?" or "Do I need a color preference?"

[1633] Step 7:

[1634] The server transmits the generated question to the terminal as text data.

[1635] Step 8:

[1636] The terminal presents the question to the user via a user interface or audio output.

[1637] Step 9:

[1638] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[1639] Step 10:

[1640] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[1641] Step 11:

[1642] The terminal transmits the text data to the server.

[1643] Step 12:

[1644] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[1645] Step 13:

[1646] The server sends the generated 3D design blueprint to the terminal.

[1647] Step 14:

[1648] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[1649] Step 15:

[1650] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[1651] Step 16:

[1652] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[1653] Step 17:

[1654] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[1655] Step 18:

[1656] The user makes a final check and approves, saying "This is fine."

[1657] Step 19:

[1658] The server converts the final design into data for the 3D printer.

[1659] Step 20:

[1660] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[1661] Step 21:

[1662] The 3D printer begins printing the object based on the received data.

[1663] Step 22:

[1664] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[1665] Step 23:

[1666] When the terminal completes printing, it notifies the user of the completion.

[1667] Example 1

[1668] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1669] The design process using conventional 3D printing devices requires specialized knowledge and skills, making it difficult for many general users. Furthermore, creating the desired design requires operating complex software and understanding technical terminology, often resulting in design errors and trial and error. The present invention aims to solve these problems by providing a system that allows users to intuitively create 3D designs using voice commands and then print them directly on a 3D printing device.

[1670] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1671] In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's instructions, a natural language generation means for automatically generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a three-dimensional design drawing based on the user's responses, a design confirmation means for presenting the generated three-dimensional design drawing to the user and obtaining confirmation, a data conversion means for converting the confirmed three-dimensional design drawing into three-dimensional printing data, and a print control means for transmitting the three-dimensional printing data to a three-dimensional printing device and starting printing. This allows users to easily create and print their ideal three-dimensional design through voice instructions without requiring specialized knowledge.

[1672] The "voice input means" is a device or module that has the function of accepting voice instructions from the user.

[1673] "Speech recognition means" refers to software or hardware that has the function of converting voice data into text data.

[1674] "Natural language processing means" refers to software or algorithms that have the ability to analyze text data and understand the user's intent.

[1675] A "natural language generation means" is software or an algorithm that has the function of automatically generating follow-up questions for the user based on the analysis results.

[1676] The "response receiving means" is a device or module that has the function of receiving a user's response to a generated question.

[1677] A "design generator" is software or algorithm capable of generating a three-dimensional design drawing based on user responses.

[1678] The "design confirmation means" is a device or module that has the function of presenting the generated three-dimensional design drawing to the user and obtaining confirmation.

[1679] "Data conversion means" refers to software or algorithms that have the function of converting the confirmed three-dimensional design drawings into data for three-dimensional printing.

[1680] "Printing control means" refers to software or hardware that has the function of sending three-dimensional printing data to a three-dimensional printing device and starting printing.

[1681] The "progress notification means" is a device or module that has the function of notifying the user of the progress of the three-dimensional printing device in real time.

[1682] An "artificial intelligence algorithm" is an algorithm that performs speech recognition and natural language processing using technologies such as machine learning and neural networks.

[1683] The specific configuration and functions of a system for implementing the present invention will be described below. This system allows a user to design a 3D print by issuing voice instructions. The user issues voice instructions to a terminal, which are accepted through a voice input means. The voice instructions are converted into text data by a voice recognition means installed in the terminal. Specifically, a microphone is used as the voice input means, and a voice recognition API (e.g., Google Speech-to-Text API) is used as the voice recognition means.

[1684] The server receives the text data sent from the device and analyzes it using natural language processing. During this analysis, the server understands the user's intent and extracts specific design requirements. A natural language processing model (e.g., OpenAI ChatGPT) is used for the natural language processing.

[1685] The server then uses a natural language generation means to generate any necessary follow-up questions and create text data to ask the user. A natural language generation model (e.g., OpenAI GPT-3) is also used as the natural language generation means.

[1686] The terminal presents the question received from the server to the user. The user responds to the question, and the response is again input as voice. The voice recognition means converts this new voice into text data again and sends it to the server.

[1687] The server then analyzes the received text data again to determine the user's detailed requirements. Once all the user requirements are gathered, the server uses a design generation tool to generate a 3D design drawing. The design generation tool uses a 3D modeling software API (e.g., Autodesk Fusion 360 API). The generated design drawing is sent to the terminal and the user is asked to confirm it.

[1688] The user checks the blueprint and makes corrections if necessary. Correction instructions are also input as voice, and are sent to the server after going through voice recognition and natural language processing. The server then updates the blueprint again.

[1689] Finally, after the user approves the design drawing, the server uses a data conversion means to convert the design drawing into data for 3D printing. This data conversion includes converting the data into a standard 3D printer file format (e.g., STL format). This data is sent to the 3D printing device via a print control means, and printing begins. Specifically, a communication module is used to link with the 3D printing device via a network.

[1690] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user of this via the progress status notification means. When printing is completed, the terminal notifies the user of the completion.

[1691] As a concrete example, consider the case where a user requests, "Please create a new design for a blue coffee cup with a handle." In this case, the device converts the voice to text data and sends the text data, "Please create a new design for a blue coffee cup with a handle," to the server. The server analyzes this request and generates additional questions to confirm the necessary details. These questions are about specifications such as the height and diameter of the cup, and are presented to the user. If the user responds, "The height is 10 cm, and the diameter is 8 cm," the server generates a 3D design based on the response. Finally, once the user confirms and the design is finalized, the server sends the data to a 3D printing device, which prints the custom coffee cup.

[1692] In this way, the system of the present invention allows users to three-dimensionally print their ideal products simply through voice instructions, without requiring specialized knowledge.

[1693] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1694] Step 1:

[1695] The user inputs a voice command.

[1696] The user speaks instructions into the device regarding the desired three-dimensional design, for example, "Create a new design for a blue coffee cup with a handle."

[1697] Input: Voice commands

[1698] Output: Audio data recorded on the device

[1699] Specific operation: The user speaks into the device's microphone.

[1700] Step 2:

[1701] The device converts the speech to text.

[1702] The device's built-in speech recognition means (e.g., Google Speech-to-Text API) converts voice input into text data.

[1703] Input: Audio data

[1704] Output: Text data

[1705] Specific operation: The device analyzes the voice data and outputs it as text data.

[1706] Step 3:

[1707] The terminal transmits the text data to the server.

[1708] The terminal transmits the converted text data to the server via the network.

[1709] Input: Text data

[1710] Output: Text data sent to the server

[1711] Specific operation: The device uploads text data to the server via the network.

[1712] Step 4:

[1713] The server analyzes the text data and understands the intent.

[1714] The server analyzes the text data using natural language processing tools (e.g., OpenAI ChatGPT) to understand the user's intent.

[1715] Input: Text data

[1716] Output: Analysis results that understand the user's intent

[1717] Specific operation: The server receives the text data and analyzes it using a natural language processing algorithm.

[1718] Step 5:

[1719] The server generates a follow-up question.

[1720] Based on the analysis, the server generates additional questions to obtain the required details.

[1721] Input: Analysis results

[1722] Output: Additional question text data

[1723] Specific operation: The server generates a question using a natural language generation means.

[1724] Step 6:

[1725] The terminal presents the user with a follow-up question.

[1726] The terminal displays the question received from the server to the user or presents it to the user by voice.

[1727] Input: Additional question text data

[1728] Output: The question presented to the user

[1729] What happens: Your device displays the question on the screen or uses text-to-speech to read the question aloud.

[1730] Step 7:

[1731] The user responds to additional questions.

[1732] The user responds verbally to the questions posed, for example, "The height is 10 cm and the diameter is 8 cm."

[1733] Input: Additional Question

[1734] Output: User's voice response

[1735] Specific operation: The user speaks the response into the device's microphone.

[1736] Step 8:

[1737] The device converts the voice response into text and sends it to the server.

[1738] The terminal converts the user's voice response into text data and sends it to the server.

[1739] Input: Voice response

[1740] Output: User response as text data

[1741] Specific operation: The device converts the voice into text and sends it to the server over the network.

[1742] Step 9:

[1743] The server determines the user's detailed requirements.

[1744] The server then re-analyzes the received text data to determine the user's specific requirements.

[1745] Input: User's detailed text response

[1746] Output: Detailed design requirements

[1747] Specific operation: The server re-analyzes the text data using natural language processing means.

[1748] Step 10:

[1749] The server generates the three-dimensional design drawings.

[1750] The server uses design generation tools (e.g. Autodesk Fusion 360 API) to generate 3D design drawings based on the user's requirements.

[1751] Input: Detailed design requirements

[1752] Output: 3D design data

[1753] Specific operation: The server calls the API to automatically generate a 3D design drawing.

[1754] Step 11:

[1755] The server sends the generated design drawing to the terminal and asks the user for confirmation.

[1756] The server transmits the three-dimensional design drawing data to the terminal and sends a message to the user requesting confirmation.

[1757] Input: 3D design data

[1758] Output: The design presented to the user, along with a message requesting confirmation

[1759] Specific behavior: The device displays a preview of the blueprint and presents a confirmation message to the user.

[1760] Step 12:

[1761] The user checks the design drawings and gives instructions for corrections if necessary.

[1762] The user checks the design drawings and, if any corrections are necessary, gives voice instructions.

[1763] Input: Design drawings and modification instructions

[1764] Output: Audio prompts for corrections

[1765] Specific operation: The user speaks the corrections into the terminal.

[1766] Step 13:

[1767] The terminal converts the correction instructions into text and sends it to the server.

[1768] The terminal converts the user's correction instructions into text data and transmits it to the server.

[1769] Input: Voice commands

[1770] Output: Correction instructions as text data

[1771] Specific operation: The device converts the voice into text and sends it to the server over the network.

[1772] Step 14:

[1773] The server updates the blueprint again.

[1774] The server updates the design drawings again based on the received correction instructions.

[1775] Input: Correction instructions as text data

[1776] Output: Corrected 3D design data

[1777] Specific operation: The server regenerates the design drawing using the design generation means.

[1778] Step 15:

[1779] The user provides final confirmation, and the server converts the data for 3D printing.

[1780] After the user finally approves the design drawing, the server uses a data conversion means to convert it into data for three-dimensional printing (e.g., STL format).

[1781] Input: Final confirmed 3D design data

[1782] Output: 3D printing data

[1783] Specific operation: The server converts the blueprint data into the appropriate file format.

[1784] Step 16:

[1785] The server sends the 3D printing data to the printing device and begins printing.

[1786] The server sends the converted data to the three-dimensional printing device and initiates the printing process using a printing control means.

[1787] Input: 3D printing data

[1788] Output: Print execution

[1789] Specific operations: The server sends data to the printing device and issues a command to start printing.

[1790] Step 17:

[1791] The terminal receives real-time progress and notifies the user.

[1792] The terminal receives the progress status of the three-dimensional printing device in real time and notifies the user using the progress status notification means.

[1793] Input: Progress data

[1794] Output: Progress notification

[1795] Specific operation: The device receives the progress data, displays it on the screen, and notifies the user.

[1796] Step 18:

[1797] The terminal notifies the user that printing is complete.

[1798] When printing is complete, the terminal notifies the user of the completion.

[1799] Input: Print completion data

[1800] Output: Completion notification

[1801] Specific operation: The device will display a completion message and notify you with an alarm sound or vibration.

[1802] (Application example 1)

[1803] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1804] Conventional 3D design systems required specialized knowledge and skills, making it difficult to quickly design and manufacture custom parts on the factory floor. This limited factory productivity and flexibility. Furthermore, voice-based systems were unable to accurately grasp the user's intent and were unable to meet the user's specific design requirements. Furthermore, there was no way to know the progress in real time, making it difficult to constantly monitor whether the print was running properly.

[1805] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1806] In this invention, the server includes an input device that accepts voice instructions, a voice analysis device that converts the voice instructions into text data, a natural language processing device that analyzes the text data to understand the user's intent, a response acceptance device that presents generated questions to the user and accepts their responses, a design generation device that generates a 3D design blueprint based on the user's responses, a data conversion device that converts the confirmed blueprint into 3D printer data, a print control device that sends the 3D printer data to the 3D printer and starts printing, and a means for issuing design instructions for commercial parts in a factory via voice instructions, generating blueprints based on those instructions, and directly controlling printing on the 3D printer. This allows even users without specialized knowledge to quickly and easily design custom parts in a factory using voice instructions and manufacture them on a 3D printer. Furthermore, progress is monitored in real time and the user is notified, making it easy to confirm whether printing is proceeding normally.

[1807] An "input device" is a means for receiving voice instructions from a user.

[1808] A "voice analysis device" is a means for converting voice instructions into text data.

[1809] A "natural language processing device" is a means for analyzing text data and understanding the user's intent.

[1810] A "natural language generation device" is a means for generating follow-up questions for the user based on the analysis results.

[1811] The "response accepting device" is a means for presenting the generated question to the user and accepting the response thereto.

[1812] A "design drawing generator" is a means for generating a 3D design drawing based on a user's response.

[1813] The "verification device" is a means for presenting the generated 3D design blueprint to the user and obtaining their confirmation.

[1814] The "data conversion device" is a means for converting the confirmed design drawings into data for a 3D printer.

[1815] A "print control device" is a means for sending 3D printer data to a 3D printer and starting printing.

[1816] A "progress notification device" is a means of notifying the user of the progress of a 3D printer in real time.

[1817] MODE FOR CARRYING OUT THE INVENTION

[1818] In this embodiment, we will explain the specific configuration and operation of a system that uses voice instructions to create 3D designs in a factory and quickly create custom parts. The main components used here and their roles are as follows:

[1819] Key components of the system

[1820] 1. Input Devices

[1821] This is a device with a microphone that allows the user to input voice commands. The microphone built into a smartphone or tablet is used.

[1822] 2. Voice analysis device

[1823] This is a device that has the function of converting user voice instructions into text data. Specifically, it uses Google's voice recognition service to convert voice into text.

[1824] 3. Natural Language Processing Systems

[1825] This is a device that analyzes text data sent from a voice analysis device and understands what the user is looking for. It uses natural language processing algorithms, especially generative AI models.

[1826] 4. Natural language generator

[1827] This is a device that generates follow-up questions for the user based on the analysis results of the natural language processing device. For example, it creates questions to confirm details of the design.

[1828] 5. Response and Reception Device

[1829] This device presents the generated questions to the user and collects their responses. It reuses voice input and converts it into text data.

[1830] 6. Design Drawing Generator

[1831] This device generates 3D design blueprints based on user responses. Design software is used to create blueprints based on specific shapes and dimensions.

[1832] 7. Verification Device

[1833] This device presents the generated 3D design blueprint to the user for confirmation, and displays it on a smartphone or tablet screen.

[1834] 8. Data Conversion Device

[1835] This is a device that converts the confirmed design drawings into data for 3D printers. It converts the design drawings into a format that can be read by 3D printers, such as STL format.

[1836] 9. Printing Control Device

[1837] This device sends the converted 3D data to the 3D printer and starts printing.

[1838] 10. Progress notification device

[1839] This device notifies users of the progress of their 3D printer in real time, sending notifications to their smartphones and tablets.

[1840] Example of operation

[1841] If an employee voice-inputs, "I'd like to create a custom wrench with a new design," the input device collects the speech and the speech analysis device converts it into text data. The natural language processing device analyzes the data, and the natural language generation device generates follow-up questions about the specific size and shape of the wrench. For example, a question such as, "What is the length and width of the wrench?" is generated. If the user responds, "It's 20 cm long and 5 cm wide," the response acceptance device collects the data and converts it back into text.

[1842] The blueprint generation device then generates a 3D design blueprint based on the response, and the confirmation device presents the blueprint to the user. Once the user reviews and approves the blueprint, the data conversion device converts the blueprint into data for the 3D printer. This data is then sent to the 3D printer via the print control device, and the printing process begins. The user is notified of the printing progress in real time via the progress notification device.

[1843] Prompt Sentence Examples

[1844] "Imagine an application where a user gives voice instructions for designing custom tools to be used in a factory, converts those instructions into text, sends it to a server, and generates a 3D design. As a concrete example, explain the process by which an employee might say into their smartphone, 'I want to create a new custom wrench design,' and then specify the exact size and shape of the wrench by voice."

[1845] In a specific implementation of this invention, factory robots can implement voice-activated design instructions and rapidly 3D print custom-made parts, improving productivity and providing a flexible manufacturing process.

[1846] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1847] Step 1:

[1848] The user issues a voice command into the input device, which then collects the voice as a digital voice signal. This digital voice signal is the input for step 1, and its output is digital voice data. Specifically, the smartphone's microphone collects the user's voice and stores it as digital data.

[1849] Step 2:

[1850] The device uses a voice analyzer to convert the digital voice data from step 1 into text data. The input is digital voice data and the output is text data. Specifically, the Google speech recognition service API is used to convert the collected voice data into text format.

[1851] Step 3:

[1852] The server uses a natural language processing device to analyze the text data from step 2 and understand the user's intent. The input is text data, and the output is the analysis result that includes the user's intent. Specifically, the generative AI model analyzes the text data and understands the "custom wrench design."

[1853] Step 4:

[1854] The server uses a natural language generator to generate follow-up questions for the user based on the analysis results. The input is the analysis results, and the output is text data containing the follow-up questions. Specifically, the AI ​​model generates questions such as "Please specify the length and width of the wrench."

[1855] Step 5:

[1856] The terminal uses the response reception device to present the follow-up question of step 4 to the user and collects the user's voice response. The input is the text data of the follow-up question, and the output is the user's voice response. Specifically, the question is displayed on the smartphone screen, and the user's voice response is collected by the microphone.

[1857] Step 6:

[1858] The terminal uses a voice analysis device to convert the response voice from step 5 back into text data. The input is the response voice, and the output is the response text data. Specifically, the voice recognition service API is used again to convert the voice response into text.

[1859] Step 7:

[1860] The server uses a blueprint generator to generate a 3D design blueprint based on the response text data from step 6. The input is the response text data, and the output is the 3D design blueprint. Specifically, dedicated design software is used to generate a blueprint of the wrench based on the specified length and width.

[1861] Step 8:

[1862] The server uses a confirmation device to present the generated 3D design blueprint to the user and ask for confirmation. The input is the 3D design blueprint, and the output is the user's confirmation response. Specifically, the blueprint is displayed on the smartphone screen, and an interface is provided for the user to confirm.

[1863] Step 9:

[1864] The server converts the user-confirmed design drawings into data for a 3D printer using a data converter. The input is the confirmed 3D design drawing, and the output is data for a 3D printer. Specifically, the design drawing is converted into a data format for a 3D printer, such as STL format.

[1865] Step 10:

[1866] The server uses the print control device to send the 3D printer data from step 9 to the 3D printer and start printing. The input is the data for the 3D printer, and the output is an instruction to start printing. In concrete terms, the 3D printer receives the data and prints based on the specified design.

[1867] Step 11:

[1868] The server uses a progress notification device to notify the user of the 3D printer's progress in real time. The input is the 3D printer's progress data, and the output is a progress notification. Specifically, the progress is notified to the smartphone, allowing the user to check the print progress.

[1869] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1870] This invention relates to a system that creates and prints 3D designs based on user voice commands. In particular, it is characterized by its ability to optimize the design process according to the user's emotional state by incorporating an emotion engine that recognizes the user's emotions.

[1871] The system starts with the user inputting a voice command into the terminal, such as "Please create a new custom coffee cup design." The voice command is then accepted using a voice input means.

[1872] The terminal uses a speech recognition means to convert the user's speech into text data. This text data is sent to a server. The server uses a natural language processing means to analyze the text data and understand the user's intention. During this analysis process, the server runs an emotion engine in parallel to recognize the user's emotions.

[1873] The emotion engine analyzes the user's tone, speech rate, context, etc. to estimate the user's emotional state. For example, it can identify whether the user is excited or relaxed. The estimated emotional information is passed to the natural language generator, which generates appropriate follow-up questions.

[1874] The server sends the generated follow-up questions to the terminal as text data and presents them to the user. For example, questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are presented to the user, and the user's answers are converted back into text data via speech recognition. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[1875] The server generates a 3D design blueprint based on the user's responses. The generated blueprint is sent to the device and displayed to the user. The user can check the blueprint and make any necessary corrections by voice. The correction instructions are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[1876] Finally, when the user approves the design drawing, the server converts it into data for the 3D printer using the data conversion means and sends it to the 3D printer via the print control means. The 3D printer prints an object based on the received data.

[1877] The terminal also receives the progress status of the 3D printer in real time and notifies the user via the progress status notification means. When the print is completed, the terminal notifies the user of the completion.

[1878] For example, if a user says, "Please create a new design of blue coffee cup with a handle," the device converts the voice to text data and sends it to the server, which uses its emotion engine to analyze the instruction and determine whether the user is excited or relaxed.

[1879] If the emotion engine determines that the user is relaxed, it generates specific questions such as, "What shade of blue cup would you like?" and presents them to the user via the device. As the user provides more detailed answers, the blueprint is generated in a form that is closer to the user's ideal.

[1880] In this way, the system of the present invention allows users to 3D print their ideal product based on voice instructions and emotional information, without the need for specialized knowledge or skills and in a short amount of time.

[1881] The processing flow will be explained below.

[1882] Step 1:

[1883] The user speaks to the device, for example, "Create a new custom coffee cup design."

[1884] Step 2:

[1885] The terminal uses the voice input means to accept and record the user's voice.

[1886] Step 3:

[1887] The device uses a speech recognition means to convert the recorded voice into text data, which becomes "Create a new custom coffee cup design."

[1888] Step 4:

[1889] The terminal transmits the converted text data to the server.

[1890] Step 5:

[1891] The server receives the text data and begins analyzing it using natural language processing means, which determines that the user wants to create a custom coffee cup.

[1892] Step 6:

[1893] The server runs an emotion engine that analyzes the tone, speaking rate, and context of the received voice data to estimate the user's emotional state.

[1894] Step 7:

[1895] The server uses natural language generation to generate additional questions that take emotional information into account. For example, if the user is relaxed, the server generates questions such as "Do you need a handle?" or "Do you have a color preference?"

[1896] Step 8:

[1897] The server transmits the generated question to the terminal as text data.

[1898] Step 9:

[1899] The terminal presents the question to the user via a user interface or audio output.

[1900] Step 10:

[1901] The user responds verbally to the question posed, for example, "Yes, I need a steering wheel. I want it blue."

[1902] Step 11:

[1903] The terminal again records the user's voice and converts it into text data using a speech recognition means. The converted text is "Yes, I need a handle. Please make it blue."

[1904] Step 12:

[1905] The terminal transmits the text data to the server.

[1906] Step 13:

[1907] The server analyzes the received text data to understand the user's detailed requirements. Once all the user requirements are gathered, a design generation tool is used to generate a 3D design blueprint.

[1908] Step 14:

[1909] The server sends the generated 3D design blueprint to the terminal.

[1910] Step 15:

[1911] The device displays the 3D design blueprint to the user and asks for confirmation, for example, "Is this the design you want?"

[1912] Step 16:

[1913] The user can review the blueprint and give voice instructions for any corrections that need to be made, such as "Make it a little lower."

[1914] Step 17:

[1915] The terminal converts the correction instruction into text data using a voice recognition means and transmits it to the server.

[1916] Step 18:

[1917] The server analyzes the correction instructions, updates the design blueprint, and sends the updated blueprint to the terminal again, prompting the user for confirmation.

[1918] Step 19:

[1919] The user makes a final check and approves, saying "This is fine."

[1920] Step 20:

[1921] The server converts the final design into data for the 3D printer.

[1922] Step 21:

[1923] The server sends the converted 3D printer data to the 3D printer via the print control means and issues an instruction to start printing.

[1924] Step 22:

[1925] The 3D printer begins printing the object based on the received data.

[1926] Step 23:

[1927] The terminal receives the progress status of the print in real time and notifies the user via the progress status notifying means.

[1928] Step 24:

[1929] When the terminal completes printing, it notifies the user of the completion.

[1930] Specific examples

[1931] For example, if the user says "Create a new design for a blue coffee cup with a handle," the following happens:

[1932] 1. The user issues a voice command.

[1933] 2. The device converts the voice into text data and sends it to the server.

[1934] 3. The server parses the text data and understands that the user wants to create a custom coffee cup.

[1935] 4. The server uses the emotion engine to estimate the user's emotional state and generate follow-up questions.

[1936] 5. The terminal presents the user with additional questions.

[1937] 6. The user responds to the question.

[1938] 7. The terminal converts the response into text data and sends it to the server.

[1939] 8. The server parses the response and generates a 3D design blueprint.

[1940] 9. The device presents the blueprint to the user and asks for confirmation.

[1941] 10. The user checks the design drawings and indicates any necessary modifications.

[1942] 11. The server reflects the correction instructions and generates the final design.

[1943] 12. The server converts the final design drawing into data for the 3D printer and sends it to the 3D printer via the print control means.

[1944] 13. The 3D printer begins printing the object.

[1945] 14. The device will notify you of the progress and notify you when the print is complete.

[1946] In this way, the system of the present invention can efficiently and accurately perform 3D design and printing based on the user's voice instructions and emotional information.

[1947] Example 2

[1948] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1949] Conventional 3D printing systems require users to have advanced design knowledge and specialized software operation, making it difficult to easily create custom designs. Furthermore, the design process is not optimized to take user feelings into consideration, which can lead to low user satisfaction. Furthermore, the printing progress cannot be checked in real time, making it difficult for users to grasp the progress.

[1950] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1951] In this invention, the server includes a recognition means for converting voice instructions into text data, a processing means for analyzing the text data to understand the user's intention, and an emotion engine for recognizing the user's emotional state. This allows users to create custom designs based on voice instructions and emotion information without requiring specialized knowledge or skills, and to 3D print them while monitoring the progress in real time.

[1952] A "voice instruction" is an instruction given by the user to the system using voice.

[1953] An "input means" is a device or module that accepts voice instructions.

[1954] The "recognition means" is a processing device or software that converts voice instructions into text data.

[1955] "Processing means" refers to algorithms or engines that analyze text data and understand the user's intent.

[1956] "Generation means" is a function that creates additional questions based on the analysis results.

[1957] An "emotion engine" is a device or software that analyzes voice tone, speech rate, and context to determine a user's emotional state.

[1958] The "presentation means" is a means for visually or audibly presenting the generated question to the user.

[1959] The "response receiving means" is a mechanism for receiving and processing responses from users.

[1960] A "design generator" is a system or software that creates a 3D design blueprint based on user responses.

[1961] The "design confirmation means" is a means for presenting the created 3D design drawing to the user and obtaining confirmation.

[1962] The "conversion means" refers to an algorithm or tool that converts the confirmed design drawings into data for a three-dimensional printer.

[1963] The "control means" is a function that sends the data for the three-dimensional printer to the three-dimensional printer and starts printing.

[1964] "Notification means" refers to a method or device that notifies the user of the progress of the three-dimensional printer in real time.

[1965] A "generative AI model" is an artificial intelligence algorithm that has been trained to perform a specific task.

[1966] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[1967] This system allows users to create and print their desired 3D designs based on voice commands. Its unique feature is that it combines an emotion engine that recognizes the user's emotions to optimize the design process according to the user's emotional state.

[1968] The user issues commands to the system through a voice input means. For example, the user might say, "Please create a custom coffee cup with a new design." The device has a built-in microphone as a voice input means and accepts this voice command. The device then converts the voice into text data using a voice recognition means such as the Google Speech-to-Text API. This converted text data is then sent to the server.

[1969] The server analyzes the received text data using natural language processing tools such as OpenAI's GPT-3 to understand the user's intent. During the analysis process, an emotion engine runs in parallel on the server, which analyzes the voice tone, speaking rate, and context to estimate the user's emotional state. For example, it determines whether the user is excited or relaxed. The emotion engine's inference results are passed to a natural language generation tool, which generates appropriate follow-up questions.

[1970] The server sends the generated follow-up questions as text data to the device, and the device presents the questions to the user. For example, specific questions such as "Do you need a steering wheel?" or "Do you have a color preference?" are displayed. The user answers these questions verbally, and the answers are again converted into text data by the device's voice recognition means and sent to the server. During this process, the emotion engine continuously monitors the user's emotional state and adjusts the content of the questions and the way they are presented as necessary.

[1971] Based on the user's responses, the server uses CAD software such as Autodesk Fusion 360 to generate a 3D design blueprint. This blueprint is generated as an STL file and sent to the terminal where it is displayed to the user. The user can review the blueprint and make any necessary corrections by voice. The corrections are also sent to the server via voice recognition and natural language processing, and the blueprint is updated.

[1972] Finally, when the user approves the design drawing, the server uses a data conversion means to convert it into G-code data for the 3D printer and sends this data to the 3D printer (e.g., Ultimaker 3). The 3D printer prints the object based on the received data. The terminal receives progress information from the 3D printer in real time and notifies the user via a progress notification means. When the print is complete, the terminal notifies the user of the completion.

[1973] For example, if a user says, "Please create a new design for a blue coffee cup with a handle," the voice is converted into text data and sent to the server. The server analyzes the instruction using its emotion engine, and if it determines that the user is relaxed, it generates a specific question such as, "What shade of blue cup would be good?" and presents it to the user via their device. If the user answers, "A deep blue would be good," a more detailed blueprint is generated based on that information.

[1974] An example of a prompt is, "If the user instructs you to create a new design of blue coffee cup with a handle, generate additional questions based on the emotional state analyzed by the emotion engine and explain the process of creating the design."

[1975] In this way, the system of the present invention enables users to quickly create their ideal 3D printed object based on voice instructions and emotional information, without the need for specialized knowledge or skills.

[1976] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1977] Step 1:

[1978] The user inputs a voice command.

[1979] Specific operation: The user speaks into the device's microphone, saying, "Create a new custom coffee cup design."

[1980] Input: User's voice

[1981] Output: Audio data

[1982] Step 2:

[1983] The device converts the voice into text data.

[1984] Specific operation: The device uses the Google Speech-to-Text API to convert voice data into text data in real time.

[1985] Input: Audio data

[1986] Output: Text data

[1987] Step 3:

[1988] The server parses the text data.

[1989] How it works: The server uses a natural language processing engine such as OpenAI's GPT-3 to analyze the text data and understand the user's intent. The analysis results are structured in JSON format.

[1990] Input: Text data

[1991] Output: Analysis results including user intent (JSON format)

[1992] Step 4:

[1993] The server uses an emotion engine to determine the user's emotion.

[1994] How it works: The server's emotion engine analyzes the voice tone, speech rate, and context to estimate the user's emotional state, for example, determining whether the user is excited or relaxed.

[1995] Input: Audio data, text data

[1996] Output: Emotional information including emotional state

[1997] Step 5:

[1998] The server generates a follow-up question.

[1999] Specific operation: Based on the user's intention and emotion information, the server uses natural language generation to generate a follow-up question, such as "Do you need a steering wheel?"

[2000] Input: Analysis results including user intent, emotional information

[2001] Output: Additional questions (text data)

[2002] Step 6:

[2003] The terminal presents the user with a follow-up question.

[2004] Specific operation: The terminal displays the generated follow-up question on the display and presents it to the user.

[2005] Input: Additional question (text data)

[2006] Output: Question displayed

[2007] Step 7:

[2008] The user answers the follow-up questions.

[2009] Specific operation: The user answers the follow-up question by saying something like, "Yes, I need a steering wheel." This answer is input into the device's microphone.

[2010] Input: User's spoken response

[2011] Output: Audio data

[2012] Step 8:

[2013] The device converts the answer into text data.

[2014] Specific operation: The device again uses the Google Speech-to-Text API to convert the voice response into text data, which is then sent to the server.

[2015] Input: Audio data

[2016] Output: Text data

[2017] Step 9:

[2018] The server generates the blueprint.

[2019] How it works: Based on the user's answers, the server generates a 3D design blueprint using CAD software such as Autodesk Fusion 360. This blueprint is saved as an STL file.

[2020] Input: Text data

[2021] Output: 3D design blueprint (STL format)

[2022] Step 10:

[2023] The terminal displays the blueprint to the user.

[2024] Specific operation: The device displays a 3D visualization of the generated design on the screen and allows the user to check it.

[2025] Input: 3D design blueprint (STL format)

[2026] Output: Displayed blueprint

[2027] Step 11:

[2028] The user inputs correction instructions by voice.

[2029] Specific operation: The user checks the design drawings and inputs correction instructions by voice, such as "Please change the shape of the handle."

[2030] Input: User's voice

[2031] Output: Audio data

[2032] Step 12:

[2033] The terminal converts the correction instructions into text data.

[2034] How it works: The device uses the Google Speech-to-Text API to convert the correction instructions into text data, which is then sent to the server.

[2035] Input: Audio data

[2036] Output: Text data

[2037] Step 13:

[2038] The server updates the blueprint to reflect the modifications.

[2039] Specific operation: The server updates the blueprint according to the correction instructions and generates a new STL file, which is then resent to the terminal.

[2040] Input: Text data

[2041] Output: Updated 3D design blueprints (STL format)

[2042] Step 14:

[2043] The user approves the design.

[2044] Specific operation: The user reviews the final design and issues a voice command to approve it: "I approve this."

[2045] Input: User's voice

[2046] Output: Audio data

[2047] Step 15:

[2048] The server converts it into data for 3D printing.

[2049] How it works: The server converts the approved design into G-code data for the 3D printer. This data conforms to the protocol for communicating with the 3D printer.

[2050] Input: 3D design blueprint (STL format)

[2051] Output: G-code data

[2052] Step 16:

[2053] The server sends the data to the 3D printer.

[2054] Specific operation: The server sends G-code data to the 3D printer wirelessly or via a wired connection.

[2055] Input: G-code data

[2056] Output: Data sent to the 3D printer

[2057] Step 17:

[2058] A 3D printer prints an object.

[2059] Specific operation: The 3D printer prints a coffee cup with the specified design based on the received G-code data.

[2060] Input: Data to send to the 3D printer

[2061] Output: Printed matter (coffee cup)

[2062] Step 18:

[2063] The device will notify you of progress in real time.

[2064] Specific operation: The device periodically obtains progress information from the 3D printer and notifies the user of the progress in real time.

[2065] Input: Progress data from the 3D printer

[2066] Output: Progress notification

[2067] Step 19:

[2068] The terminal notifies the user that printing is complete.

[2069] Specific operation: When the print is complete, the terminal notifies the user that "Printing is complete." The user is then ready to remove the finished object.

[2070] Input: Print completion data

[2071] Output: Completion notification

[2072] (Application example 2)

[2073] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2074] Conventional 3D design generation systems generate designs simply based on voice instructions without considering the user's emotional state, which can result in low user satisfaction. Furthermore, in situations where customization based on the user's mood or emotion is required, particularly in services such as food delivery, conventional systems have difficulty responding.

[2075] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for accepting voice instructions, a voice recognition means for converting the voice instructions into text data, a natural language processing means for analyzing the text data to understand the user's intention, a natural language generation means for generating follow-up questions for the user based on the analysis results, a response receiving means for presenting the generated questions to the user and accepting their responses, a design generation means for generating a 3D design based on the user's responses, a design confirmation means for presenting the 3D design to the user and obtaining confirmation, a data conversion means for converting the confirmed design into data for a 3D printer, a print control means for sending the data for a 3D printer to the 3D printer and starting printing, an emotion analysis means for analyzing the user's voice instructions and emotional state, and an emotion-responsive question generation means for generating follow-up questions based on the user's emotional state based on the emotion analysis means. This enables the generation of customized 3D designs that correspond to the user's mood and emotions.

[2076] "Voice input means" refers to a device or software that accepts voice instructions from the user.

[2077] A "voice recognition means" is a device or algorithm that converts voice instructions into text data.

[2078] "Natural language processing means" is a technology for analyzing text data and understanding the user's intent.

[2079] "Natural language generation means" refers to technology or devices that generate follow-up questions for the user based on the analysis results.

[2080] The "response receiving means" is a device or software that presents the generated question to the user and receives the response.

[2081] "Design generation means" refers to technology or equipment that generates 3D design blueprints based on user responses.

[2082] "Design confirmation means" refers to technology or equipment that presents the generated 3D design drawing to the user for confirmation.

[2083] "Data conversion means" refers to the technology or device that converts the confirmed design drawings into data for a 3D printer.

[2084] "Print control means" refers to the technology or device that sends 3D printer data to the 3D printer and starts printing.

[2085] "Emotion analysis means" refers to technology or devices that analyze the user's voice instructions and emotional state.

[2086] The "emotion-responsive question generation means" is a technique or device that generates a follow-up question according to the user's emotional state based on emotion analysis means.

[2087] The system for implementing this invention is mainly composed of three entities: a server, a terminal, and a user. This system generates 3D designs based on the user's voice instructions and emotional state, and customizes them for specific purposes (such as food delivery).

[2088] First, the user issues a voice instruction to the terminal. For example, the user may say, "Today is a special day, so I'd like a gorgeous package." This voice instruction is accepted by a voice input means installed in the terminal.

[2089] Next, the terminal uses a voice recognition means to convert the voice instruction into text data. The converted text data is sent to the server. The server uses a natural language processing means to analyze the text data and understand the user's intention. At the same time, the server uses an emotion analysis means to analyze the user's emotional state. This analysis determines, for example, whether a user's statement "gorgeous" contains a sense of exhilaration or specialness.

[2090] Based on the analysis results, the server generates a further question using the emotion-aware question generation means. For example, a specific question such as "What color of gorgeous packaging do you like?" is generated. This question is then presented to the user via the terminal.

[2091] The user's response is again received by the voice input means, converted into text data by the voice recognition means, and sent to the server. The server uses the design generation means to generate a 3D design drawing based on this response data. The generated drawing is presented to the user via the terminal, and the user confirms it using the design confirmation means. If necessary, the user can make corrections to the design, and the system will generate the design again.

[2092] Finally, the confirmed design drawing is converted into data for the 3D printer using the data conversion means and sent to the 3D printer through the print control means. The 3D printer prints the object based on the received data.

[2093] This system can generate customized 3D designs according to the user's mood and emotions. For example, if a user says, "I'm feeling good today, so a colorful package would be good," the system will generate a design with vibrant colors based on the user's emotional state. Similarly, if a user says, "Today is a special day, so a gorgeous package would be good," the system will generate a luxurious design with a gold base.

[2094] This allows users to easily use custom-designed packaging that matches their emotional state in food delivery services without requiring specialized knowledge or skills.

[2095] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2096] Step 1:

[2097] The user issues a voice command to the terminal. For example, the user might say, "Today is a special day, so a gorgeous package would be nice." The terminal accepts this voice using a voice input means. The input is the user's voice, and the output is voice data.

[2098] Step 2:

[2099] The device uses a speech recognition means to convert the voice data obtained in step 1 into text data. This conversion is performed using the Google Speech Recognition API or similar. The input is voice data, and the output is text data.

[2100] Step 3:

[2101] Text data is sent from the device to a server. The server uses natural language processing to analyze the text data and understand the user's intent. For example, a generative AI model such as BERT is used for the analysis. The input is text data, and the output is the user's intent as an analysis result.

[2102] Step 4:

[2103] At the same time, the server uses an emotion analysis tool to analyze the user's emotional state from the text data. The analysis takes into account speech tone and speed, and uses an algorithm such as the Emotion API. The input is text data, and the output is the user's emotional state.

[2104] Step 5:

[2105] The server generates follow-up questions using natural language generation based on the analysis results and the emotional state. For example, a question such as "What is a good color for gorgeous packaging?" is generated. A generative AI model (e.g., GPT-3) is used for this generation. The input is the analysis results and the emotional state, and the output is the text of the follow-up question.

[2106] Step 6:

[2107] The server sends the generated follow-up question to the terminal, which then presents it to the user. The user again answers by voice, and the terminal accepts the answer via a voice input means. The input is the text of the follow-up question and the user's voice answer, and the output is voice data.

[2108] Step 7:

[2109] The terminal uses a voice recognition means to convert the user's voice response into text data and transmits it to the server. The input is voice data and the output is text data.

[2110] Step 8:

[2111] The server generates a 3D design blueprint based on the user's answers using a design generation tool. CAD software or a custom design API is used to generate the blueprint. The input is the text data of the user's answers, and the output is a 3D design blueprint.

[2112] Step 9:

[2113] The server sends the generated 3D design blueprint to the terminal, which then presents it to the user. The user can check the blueprint and make corrections as necessary. The input is the 3D design blueprint and user feedback, and the output is the final 3D design blueprint.

[2114] Step 10:

[2115] The server converts the final confirmed design drawing into data for the 3D printer using a data conversion means and sends it to the 3D printer via a print control means. The input is the final 3D design drawing, and the output is the data for the 3D printer.

[2116] Step 11:

[2117] The 3D printer prints the object based on the received data, and the terminal receives the print progress status in real time and notifies the user. The input is the data for the 3D printer, and the output is the object printed by the 3D printer and a progress notification.

[2118] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2119] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2120] In the above embodiment, an example was given in which the specific processing is performed by the da...

Claims

1. a voice input means for receiving voice instructions; a voice recognition means for converting voice instructions into text data; natural language processing means for analyzing text data to understand user intent; a natural language generation means for generating follow-up questions for the user based on the analysis results; a response receiving means for presenting the generated question to a user and receiving a response thereto; A design generation means for generating a 3D design blueprint based on the user's response; a design confirmation means for presenting the 3D design drawing to a user for confirmation; A data conversion means for converting the confirmed design drawings into data for a 3D printer; A print control means for transmitting 3D printer data to the 3D printer and starting printing; A system including:

2. The system according to claim 1, further comprising a progress notification means for notifying the user of the progress of the 3D printer in real time.

3. 2. The system of claim 1, wherein the speech recognition means and the natural language processing means use artificial intelligence algorithms.

Images