system

Abstract

We provide the system. [Solution] A depth measurement means for acquiring three-dimensional shape data, A chat interface that receives user requests in natural language, A generation means for automatically generating a 3D model file based on the aforementioned 3D shape data and the aforementioned natural language request, A search method that searches for and presents existing products similar to the generated 3D model, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There are difficulties when people obtain customizable objects suitable for specific sizes and shapes in daily life. In particular, it is difficult for ordinary users without specialized 3D modeling knowledge to easily design and obtain products that meet their individual needs. There is also a problem that it is not easy to check whether similar products already exist in the market.

Means for Solving the Problems

[0005] This invention provides a system that rapidly acquires three-dimensional shape data using depth measurement means and automatically generates a 3D model file using generation means based on requests entered by the user in natural language. This system allows users to easily create customized 3D objects by utilizing machine learning algorithms. Furthermore, it includes means for searching for existing products similar to the generated model, and can efficiently meet the user's needs by presenting suitable options.

[0006] "Three-dimensional shape data" refers to digital data that includes information about the height, width, and depth of an object, and is necessary to construct a 3D model of that object.

[0007] A "depth measurement device" is a technical device that acquires information such as the distance and height of an object or space through the reflection of light or sound waves, and uses this information to generate three-dimensional shape data.

[0008] A "chat interface" is a conversational user interface that allows users to input requests and preferences using natural language.

[0009] "Generation means" refers to a system or algorithm for automatically generating a 3D model file based on acquired data and input requests.

[0010] A "3D model file" is a file format that contains the data necessary to digitally reproduce the shape and characteristics of an object in three-dimensional space.

[0011] A "search method" refers to a system or algorithm that identifies similar product data within a database and processes it to present it to the user.

[0012] A "machine learning algorithm" is a program or process that allows a computer to learn patterns from data and make predictions or decisions based on new data. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be described.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that enables users to easily generate customized three-dimensional objects. A depth measurement mechanism is incorporated into the terminal to quickly and accurately acquire three-dimensional shape data. By using LiDAR sensors and the like, shape data of physical objects and spaces can be acquired.

[0035] The terminal transmits the acquired 3D shape data to the server. The server analyzes the received data in combination with the user's natural language requests, and generates an appropriate 3D model file using a generation mechanism. In this process, the generation mechanism utilizes machine learning algorithms to generate the model that best matches the user's requests.

[0036] Specifically, when a user requires an object of a particular shape or size, the data scanned by the terminal and requests entered by the user in the chat interface, such as "make the height 10cm" or "make it yellow," are sent to the server. In response, the server generates a 3D model file in .obj format via a generation mechanism and provides it to the user's terminal.

[0037] Furthermore, the server searches its database for existing products similar to the generated 3D model. If similar products are found, this information is provided to the user, allowing them to choose between the new model and commercially available products. This enables users to efficiently find the best option to suit their needs.

[0038] In this way, even users without special 3D modeling knowledge can easily design and obtain objects to solve everyday inconveniences.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user uses a device to select an object or space they wish to scan using depth measurement and activates the LiDAR sensor. The device then moves the sensor toward the target object, acquiring its three-dimensional shape data.

[0042] Step 2:

[0043] The terminal converts the acquired 3D shape data into a file format and sends it to a server in the cloud. A secure protocol is used for transmission to protect the confidentiality of the data.

[0044] Step 3:

[0045] Users use the device's chat interface to input their wishes and requests in natural language. This includes object dimensions, color, and other functional requirements.

[0046] Step 4:

[0047] The server analyzes the user's request using a natural language processing engine and converts it into technical specifications. The analyzed data and 3D shape data are then passed to the generation system.

[0048] Step 5:

[0049] The server generation method utilizes machine learning algorithms to generate 3D model files based on the analysis data. This generation uses formats such as .obj, and is optimized to meet user requirements.

[0050] Step 6:

[0051] The server searches for similar products based on the generated 3D model. It refers to existing product databases to identify product information that closely matches the model.

[0052] Step 7:

[0053] The terminal displays a 3D model generated from the server along with information on similar products to the user. The user can preview the generated model and view details of similar products.

[0054] Step 8:

[0055] The user selects a generated .obj file from the provided information and decides whether to print it on a 3D printer or purchase a similar product from the options presented. This allows the user to actually obtain an object that meets their needs.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In modern life, there is a need for users to design and acquire customizable 3D objects without requiring special technical knowledge or expertise. However, conventional 3D modeling methods require specialized software and techniques, making them difficult for the average user to use. Therefore, there is a need for technology that allows even users without specialized knowledge to easily and quickly generate and acquire 3D objects.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes measuring means, interactive interface means, creation means, and search means. This allows users to intuitively communicate requests using natural language without special technical knowledge, efficiently generate three-dimensional objects based on those requests, and compare and select existing similar products.

[0061] "Measurement means" refers to devices and technologies for acquiring three-dimensional shape data, and in particular, it refers to systems that use light to accurately measure the distance and shape of an object.

[0062] An "interactive interface means" refers to a communication tool that receives instructions from users in natural language and links that information with a processing system.

[0063] "Creation method" refers to the technology and algorithms used to automatically generate a 3D structure file based on acquired 3D shape data and user instructions.

[0064] "Search method" refers to a function that searches a database for existing products similar to the generated three-dimensional structure and presents candidates to the user.

[0065] To concretely implement the invention, it is crucial to build a system that allows for cooperation between a server and a terminal. In this system, a distance measuring device utilizing light is used on the terminal as a means of acquiring three-dimensional shape data of physical objects and spaces. Specifically, LiDAR sensors are used. This allows the user to acquire detailed information about the distance and shape of objects.

[0066] The acquired 3D shape data is stored on the terminal, and in parallel, the user enters custom requests about the object in natural language through an interactive interface. This prompt serves as a convenient way to express the user's intent and is sent to the server.

[0067] The server utilizes a generative AI model to automatically generate a 3D structure file by combining the 3D shape data received through the creation method with user instructions. The machine learning algorithm used in this process performs complex data integration and analysis based on the user's requirements to generate the most suitable 3D structure.

[0068] Furthermore, the server can use search mechanisms to find existing products in its database that match or are similar to the generated 3D structure, thereby presenting options to the user. This allows the user to compare the generated object with similar commercially available products and select the optimal option.

[0069] For example, if a user needs a shelf of a specific shape and size, they can scan the wall with their device and input a request in natural language, such as "make the shelf 180cm high and 90cm wide." The server parses this prompt, generates a 3D structure of the corresponding shelf, and proposes it. The user can then receive information on commercially available products similar to the generated shelf to help them make a selection.

[0070] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0071] Step 1:

[0072] The device uses a LiDAR sensor to acquire three-dimensional shape data of physical objects and space. The data obtained by this measurement method is recorded in the device as point cloud data. The input is the physical state of the object, and the output is digital three-dimensional shape data.

[0073] Step 2:

[0074] The user inputs requests about objects in natural language through the terminal's interactive interface. For example, they might give the instruction, "Make the shelf 180cm high and 90cm wide." The input is the user's request and is stored in the terminal as a prompt. This prompt is used for subsequent processing.

[0075] Step 3:

[0076] The terminal sends the acquired 3D shape data and the user's natural language request (prompt) to the server. In this data transfer, the shape data and user request are inputs, and the output is a signal to prompt the server to start processing.

[0077] Step 4:

[0078] The server processes the received 3D shape data and prompt text using a generative AI model. The input consists of data and requests received from the terminal, and the output is design guidelines for a 3D structure that meets the user requirements. The data is integrated by a machine learning algorithm to generate the optimal shape based on the requirements.

[0079] Step 5:

[0080] The server generates a 3D structure file based on the analysis results. Using machine learning techniques, the data is converted to the .obj format, and a model suitable for the user's requirements is generated. The input is design guidelines, and the output is a 3D structure file in .obj format.

[0081] Step 6:

[0082] The server searches for existing products similar to the generated 3D structure. It accesses a database and uses feature extraction techniques to find similar products. The input is the generated 3D structure, and the output is a list of similar products.

[0083] Step 7:

[0084] The server sends the generated 3D structure file and information on similar products found to the terminal. The user receives this information and can compare the generated product with commercially available products to make purchasing or usage decisions. The input is the data of the generated product and information on similar products, and the output is information provided to the user.

[0085] (Application Example 1)

[0086] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] Modern consumers have a need to easily customize items through natural interaction, even in virtual environments. However, conventional 3D modeling techniques and product catalogs require advanced expertise and manual operation, making them not easily accessible to everyone. Furthermore, there is the problem of difficulty in real-time verification of visualization and customization results in virtual space.

[0088] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0089] In this invention, the server includes a distance measuring device for acquiring three-dimensional shape data, a communication means for receiving user requests in natural language, a generation device for automatically generating three-dimensional structure data based on the three-dimensional shape data and the natural language requests, a search device for searching for and presenting existing articles similar to the generated three-dimensional structure data, and a display device for presenting the visualized and adjusted three-dimensional structure data of the articles through a virtual vision device. This makes it possible for users to easily customize three-dimensional objects without specialized knowledge and to visually confirm them in real time in a virtual environment.

[0090] "Three-dimensional shape data" refers to digital data used to represent the three-dimensional structure of an object or space, and includes coordinate and distance information.

[0091] A "distance measuring device" is a device that uses light or other technologies to measure the distance to an object and acquire data on its three-dimensional shape.

[0092] A "communication method" is an interface that receives requests and instructions from users in natural language and processes that information within the system.

[0093] A "generation device" is a device that has the function of automatically generating new three-dimensional structural data or models based on acquired three-dimensional shape data and user requirements.

[0094] A "search device" is a device that finds existing items or information in a database that are similar to the generated three-dimensional structural data, and presents them to the user.

[0095] A "virtual visual device" is a display device that visualizes objects generated based on digital data, allowing users to experience them virtually.

[0096] The present invention provides a system that allows users to easily customize three-dimensional objects in a virtual space. Users can collect data on the three-dimensional shape in real time using a virtual vision device such as smart glasses. Specifically, the following hardware and software are used.

[0097] The server acquires three-dimensional shape data of objects and space through a distance measuring device equipped with a LiDAR sensor. Users input their customization requests in natural language via communication. Natural language processing software then parses these requests. Specifically, natural language processing libraries using Python are often used.

[0098] On the server side, based on the acquired 3D shape data and the user's request, the generation device generates customized 3D structural data using OpenAI's (registered trademark) generation AI model. Based on this generated data, the search device searches the database for similar items and presents them to the user. Through this series of processes, the user can visually confirm what kind of customization is possible.

[0099] The virtual vision system enables real-time display to visualize the generated three-dimensional model. As a result, an environment is created where users can easily check the quality of the model.

[0100] As a concrete example, consider a scenario in a furniture store's virtual showroom where a user visualizes a chair in a virtual space and gives instructions such as, "I want the height lowered by 5 cm and the color changed to blue." Based on this request, the server uses a generative AI model to generate a 3D model according to the instructions. The user can then review this model in the virtual environment and use it as a basis for deciding whether to order the actual product.

[0101] Example of a prompt:

[0102] "Based on the chair data, please generate a new 3D model with the height reduced by 5 centimeters and the color changed to blue."

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] The user uses smart glasses to visually identify a specific object in a virtual space and acquires three-dimensional shape data using a distance measuring device. The input is the physical information of the object, and the output is its three-dimensional shape data. The distance measuring device uses a LiDAR sensor to scan the surface of the object and generates its coordinate information as digital data.

[0106] Step 2:

[0107] The user sends customization requests to the server in natural language via a communication method. The input is a natural language instruction, and the output is the parsed request data. The server uses a natural language processing library to parse the input data and extract the specific changes (e.g., "lower the height by 5cm," "change the color to blue").

[0108] Step 3:

[0109] The server uses a generation device to generate customized 3D structural data based on the acquired 3D shape data and analyzed request data. The input includes 3D shape data and request data, and the output is customized 3D structural data. The AI ​​generation model adjusts the model, and a 3D model matching the requirements is generated in .obj format.

[0110] Step 4:

[0111] The server uses customized three-dimensional structure data to search for similar existing items within the database and presents the information to the user. The input is customized three-dimensional structure data, and the output is information about similar items. The search device executes database queries to find similar items that meet specific criteria.

[0112] Step 5:

[0113] The user uses a virtual vision device to visually verify the generated customized model. The input is the generated three-dimensional structural data, and the output is a visualized object display. The virtual vision device renders the data in real time, providing the user with visual feedback.

[0114] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0115] This invention is a system that generates customized three-dimensional objects that reflect not only the user's requests but also their emotional state. A depth measurement means built into the terminal uses LiDAR to acquire three-dimensional shape data of objects and space, and incorporates this into the shape selected by the user.

[0116] The device also provides a chat interface to receive requests in natural language through interaction with the user. In addition, it incorporates an emotion engine that analyzes the user's emotions. The emotion engine uses facial recognition and voice tone analysis to identify the user's emotions and extract emotional information such as "happiness" or "calmness." This makes it possible to take into account design elements that correspond to the user's mental state.

[0117] The server has a generation mechanism for generating 3D model files based on 3D shape data and natural language requests. Using machine learning algorithms, it creates 3D models that reflect the user's requests and emotions. Emotional information can influence subtle aspects of the design; for example, positive emotions might be associated with brighter colors and more curvilinear designs.

[0118] When a user requests a special product design based on a specific theme, they first scan the target area with LiDAR on their device, and then enter specific requests, such as "I want some small items with a cheerful atmosphere," via a chat interface. When the emotion engine detects joy from the user's facial expressions and voice, that emotion information is also transmitted to the server.

[0119] The server processes the received data to generate a new 3D model, and in the process, it also searches for similar products. This allows the user to review the newly generated custom model and select existing similar products that match their preferences.

[0120] As described above, the present invention allows users to easily obtain emotionally tailored customized products without requiring any technical knowledge.

[0121] The following describes the processing flow.

[0122] Step 1:

[0123] The user selects an object to scan using the device and activates the LiDAR sensor. The device uses depth measurement to acquire three-dimensional shape data of the object and temporarily stores that data.

[0124] Step 2:

[0125] The device receives requests from users in natural language via a chat interface. Users can input specific requests such as "bright design" or "curved shape."

[0126] Step 3:

[0127] The device activates an emotion engine and uses its camera and microphone to recognize the user's emotions. Through facial recognition and voice analysis, it identifies the user's emotional state and acquires that information.

[0128] Step 4:

[0129] The terminal transmits acquired 3D shape data, natural language requests, and sentiment information to the server. This data is securely transmitted through a secure communication protocol.

[0130] Step 5:

[0131] The server analyzes the received natural language request and converts it into technical instructions. Using machine learning algorithms, it generates a 3D model file that combines the analysis results with sentiment information.

[0132] Step 6:

[0133] The server uses the generated 3D model to search the database for similar products. It extracts information on existing products with a high degree of similarity and prepares the data to present to the user.

[0134] Step 7:

[0135] The terminal displays the generated 3D model and similar product information received from the server to the user. The user can view the model in 3D preview and, if necessary, also view details of similar products.

[0136] Step 8:

[0137] Users can select a generated 3D model and either print it using a 3D printer or save it digitally. They can also purchase similar products suggested, expanding their product choices.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0140] Conventional 3D model generation systems have the problem of being unable to customize models to take into account the emotional state of the user, and therefore being unable to generate models that meet the emotions and needs of individual users. Furthermore, there was a lack of a simple way to understand the relationship between the generated 3D models and existing products in the market.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes a measurement means for acquiring three-dimensional shape information, a dialogue means for receiving user requests in natural language, a model generation means for automatically generating a three-dimensional model based on the three-dimensional shape information and the natural language requests, a search means for searching for and presenting existing items similar to the generated three-dimensional model, and an emotion analysis means for analyzing the user's emotional state and reflecting the emotional information in the three-dimensional model. This makes it possible to generate a customized model that matches the user's emotions and requests, and to present similar existing items.

[0143] "Three-dimensional shape information" refers to data that represents and records the physical shape of an object or space in three dimensions.

[0144] "Measurement means" refers to devices and methods used to acquire the three-dimensional shape of an object, utilizing technologies such as light detection and distance measurement.

[0145] A "user" refers to a person who operates this system and has specific requests or objectives.

[0146] "Natural language" refers to the linguistic forms that people use on a daily basis, and the language that users use to communicate requests to a system.

[0147] A "dialogue tool" is an interface for communicating with users and receiving requests in natural language.

[0148] A "3D model" is a digital representation generated based on data of a three-dimensional shape; it is a computer model that mimics the physical shape.

[0149] "Model generation means" refers to devices or methods used to automatically generate three-dimensional models based on input data, and includes machine learning algorithms.

[0150] A "search tool" refers to a device or method for finding existing items similar to the generated three-dimensional model in a database and presenting them to the user.

[0151] "Emotional state" refers to information that indicates the user's current emotional situation, and refers to the psychological state extracted from facial expressions and tone of voice.

[0152] "Emotional analysis means" refers to devices or methods for analyzing and identifying a user's emotions, and utilizes facial recognition and voice analysis technologies.

[0153] This invention relates to a system that generates customized 3D models that reflect the user's requests and emotional state. The system implementation mainly consists of terminals and servers.

[0154] The terminal incorporates sensors that utilize optical detection and ranging technologies as measurement means to acquire three-dimensional shape information. This measurement means efficiently acquires three-dimensional shape data from the user's environment and saves it as point cloud data. The user can input their requests in natural language through a chat interface, which serves as a means of interaction. In this case, specific requests such as "I want some small items that give off a bright atmosphere" are entered as an example of prompt text.

[0155] The device also includes emotion analysis capabilities to analyze the user's emotional state. It uses a camera and microphone to capture facial expressions and voice tone, thereby identifying the user's emotions. For example, feelings of joy and happiness can be detected, and this emotional information is reflected in the model's design.

[0156] The server receives three-dimensional shape data, natural language requests, and sentiment information transmitted from the terminal. It then automatically generates a 3D model based on a generative AI model using machine learning as the generation method. In this process, for example, if positive sentiment information is received, bright colors and curvilinear design elements are incorporated into the model.

[0157] Furthermore, the server searches its database for existing items similar to the generated 3D model and presents them to the user. This allows the user to compare the newly generated custom model with similar commercially available products and make a suitable selection.

[0158] In this way, the system allows users to obtain appropriate and emotionally tailored customized products without requiring any special technical knowledge.

[0159] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0160] Step 1:

[0161] The device activates its LiDAR sensor via user input to acquire three-dimensional shape data of surrounding objects and space. The sensor uses light detection and ranging techniques to collect detailed point cloud data of the target. This input data is stored as a basis for future 3D model generation.

[0162] Step 2:

[0163] The terminal receives user requests in natural language using a chat interface. For example, the user might type, "I want some small items with a bright atmosphere." The terminal records this natural language request as text data and prepares it for subsequent processing.

[0164] Step 3:

[0165] The device uses a camera and microphone to capture the user's facial expressions and voice, and performs emotion analysis. The emotion analysis system processes this input data and generates emotional information such as "happiness" or "calmness." This information is stored as data that influences design elements.

[0166] Step 4:

[0167] The terminal sends acquired 3D shape data, natural language requests, and sentiment information to the server as data packets. Here, accurate data transmission is ensured using communication methods. The output of this step is a complete set of user information for the server to analyze.

[0168] Step 5:

[0169] The server creates a 3D model using a generative AI model based on the received data. The machine learning algorithm designs the 3D model, reflecting user requests and emotional information. Here, bright colors and curvilinear designs are incorporated based on positive emotions. The generated 3D model is then prepared for further processing.

[0170] Step 6:

[0171] The server uses the generated 3D model to search its database for similar existing items. The search extracts information on similar products, which is also prepared for user presentation.

[0172] Step 7:

[0173] The terminal displays the 3D model and similar product information returned from the server to the user. The user can visually review the generated custom model and compare it with suggested similar products. The output of this step provides the user with information to make selections and decisions.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0176] In today's commercial spaces, it is crucial to quickly customize products based on individual customer emotions and needs. However, conventional technologies have struggled to efficiently customize products while considering customer emotions, creating a need for methods to enhance customer satisfaction. Furthermore, there has been a lack of systems capable of providing product recommendations utilizing real-time sentiment analysis.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes depth detection means for acquiring three-dimensional shape information, dialogue interface means for receiving user requests in natural language, and emotion recognition means for analyzing the user's emotional state. This makes it possible to quickly generate and present customized products that meet the customer's emotions and requests.

[0179] "Three-dimensional shape information" refers to data that indicates the three-dimensional shape and position of an object in space.

[0180] A "depth detection method" is a method of acquiring three-dimensional shape information using devices or techniques that measure the distance to an object.

[0181] "Users" refers to customers who use this system to customize products.

[0182] "Natural language" refers to the language that humans use on a daily basis, and is a means of communication that does not rely on special codes or formats.

[0183] A "dialogue interface means" is a technology that provides an interface for receiving and processing user requests in natural language.

[0184] "Emotion identification means" refers to technology that analyzes and identifies emotions from a user's facial expressions, voice, etc.

[0185] A "3D shape file" is a digital file that stores three-dimensional shape data and is used for visualization and 3D printing.

[0186] "Generation method" refers to a system that automatically creates a 3D shape file based on the input data.

[0187] A "search tool" is a system element that has the function of searching for existing items similar to the generated three-dimensional shape and presenting them to the user.

[0188] This invention is a system that generates three-dimensional objects corresponding to the user's emotions. This system utilizes depth detection means, dialogue interface means, and emotion recognition means.

[0189] Hardware and software configuration:

[0190] server:

[0191] The server processes three-dimensional shape information and generates a 3D shape file based on natural language requests and sentiment data. Machine learning algorithms are used for generation. Specifically, Amazon Lex is used for natural language understanding, and Microsoft Azure's Emotion API is used for sentiment analysis.

[0192] Device (e.g., smart glasses):

[0193] The device is equipped with a LiDAR sensor to measure the distance to objects, a camera to capture the user's facial expressions, and a microphone to recognize voice. This allows the user's requests and emotional state to be transmitted to the server in real time. Blender can be used for 3D rendering.

[0194] Specific example:

[0195] As users move through a commercial space, they can input prompts through smart glasses, such as "I want furniture with a relaxing atmosphere." The smart glasses detect the user's calm facial expression and send the emotion "relaxation" to a server. Based on this information, the server generates 3D models of custom furniture with bright colors and rounded designs. Users can then view the generated designs on the spot and select the items they want.

[0196] Example of a prompt:

[0197] "I want to give this product as a gift to a friend, so I'd like the design to convey my joy."

[0198] "I'd like the design to use warmer colors."

[0199] This invention allows users to obtain customized products that cater to their emotions at that moment, without requiring any technical knowledge.

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The user wears smart glasses and points the camera at an object of interest. The device uses a LiDAR sensor to acquire three-dimensional shape information. This information is stored on the device as data representing the shape and position of the object in space.

[0203] Step 2:

[0204] The user inputs their request in natural language into the microphone of the smart glasses. For example, they might request "make the design brighter." The device converts the voice into text and saves the request as digital data through a dialogue interface.

[0205] Step 3:

[0206] The device captures the user's facial expressions with its camera and analyzes the user's emotional state using emotion recognition technology. The analyzed emotion data (e.g., "joy" or "calmness") is sent to the server as numerical data.

[0207] Step 4:

[0208] The server receives 3D shape information from LiDAR, request data from voice input, and emotion data. Based on this data, a generative AI model within the server generates a customized 3D shape file. Machine learning algorithms are used in this process.

[0209] Step 5:

[0210] The generated 3D shape file is sent from the server to the terminal and displayed on the smart glasses' screen. The user can view the proposed custom design through the smart glasses.

[0211] Step 6:

[0212] The server then searches for existing items similar to the generated 3D shape file and presents that information to the user. This allows the user to select not only the newly generated design but also existing products.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] This invention is a system that enables users to easily generate customized three-dimensional objects. A depth measurement mechanism is incorporated into the terminal to quickly and accurately acquire three-dimensional shape data. By using LiDAR sensors and the like, shape data of physical objects and spaces can be acquired.

[0230] The terminal transmits the acquired 3D shape data to the server. The server analyzes the received data in combination with the user's natural language requests, and generates an appropriate 3D model file using a generation mechanism. In this process, the generation mechanism utilizes machine learning algorithms to generate the model that best matches the user's requests.

[0231] Specifically, when a user requires an object of a particular shape or size, the data scanned by the terminal and requests entered by the user in the chat interface, such as "make the height 10cm" or "make it yellow," are sent to the server. In response, the server generates a 3D model file in .obj format via a generation mechanism and provides it to the user's terminal.

[0232] Furthermore, the server searches its database for existing products similar to the generated 3D model. If similar products are found, this information is provided to the user, allowing them to choose between the new model and commercially available products. This enables users to efficiently find the best option to suit their needs.

[0233] In this way, even users without special 3D modeling knowledge can easily design and obtain objects to solve everyday inconveniences.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] The user uses a device to select an object or space they wish to scan using depth measurement and activates the LiDAR sensor. The device then moves the sensor toward the target object, acquiring its three-dimensional shape data.

[0237] Step 2:

[0238] The terminal converts the acquired 3D shape data into a file format and sends it to a server in the cloud. A secure protocol is used for transmission to protect the confidentiality of the data.

[0239] Step 3:

[0240] Users use the device's chat interface to input their wishes and requests in natural language. This includes object dimensions, color, and other functional requirements.

[0241] Step 4:

[0242] The server analyzes the user's request using a natural language processing engine and converts it into technical specifications. The analyzed data and 3D shape data are then passed to the generation system.

[0243] Step 5:

[0244] The server generation method utilizes machine learning algorithms to generate 3D model files based on the analysis data. This generation uses formats such as .obj, and is optimized to meet user requirements.

[0245] Step 6:

[0246] The server searches for similar products based on the generated 3D model. It refers to existing product databases to identify product information that closely matches the model.

[0247] Step 7:

[0248] The terminal displays a 3D model generated from the server along with information on similar products to the user. The user can preview the generated model and view details of similar products.

[0249] Step 8:

[0250] The user selects a generated .obj file from the provided information and decides whether to print it on a 3D printer or purchase a similar product from the options presented. This allows the user to actually obtain an object that meets their needs.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] In modern life, there is a need for users to design and acquire customizable 3D objects without requiring special technical knowledge or expertise. However, conventional 3D modeling methods require specialized software and techniques, making them difficult for the average user to use. Therefore, there is a need for technology that allows even users without specialized knowledge to easily and quickly generate and acquire 3D objects.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes measuring means, interactive interface means, creation means, and search means. This allows users to intuitively communicate requests using natural language without special technical knowledge, efficiently generate three-dimensional objects based on those requests, and compare and select existing similar products.

[0256] "Measurement means" refers to devices and technologies for acquiring three-dimensional shape data, and in particular, it refers to systems that use light to accurately measure the distance and shape of an object.

[0257] An "interactive interface means" refers to a communication tool that receives instructions from users in natural language and links that information with a processing system.

[0258] "Creation method" refers to the technology and algorithms used to automatically generate a 3D structure file based on acquired 3D shape data and user instructions.

[0259] "Search method" refers to a function that searches a database for existing products similar to the generated three-dimensional structure and presents candidates to the user.

[0260] To concretely implement the invention, it is crucial to build a system that allows for cooperation between a server and a terminal. In this system, a distance measuring device utilizing light is used on the terminal as a means of acquiring three-dimensional shape data of physical objects and spaces. Specifically, LiDAR sensors are used. This allows the user to acquire detailed information about the distance and shape of objects.

[0261] The acquired 3D shape data is stored on the terminal, and in parallel, the user enters custom requests about the object in natural language through an interactive interface. This prompt serves as a convenient way to express the user's intent and is sent to the server.

[0262] The server utilizes a generative AI model to automatically generate a 3D structure file by combining the 3D shape data received through the creation method with user instructions. The machine learning algorithm used in this process performs complex data integration and analysis based on the user's requirements to generate the most suitable 3D structure.

[0263] Furthermore, the server can use search mechanisms to find existing products in its database that match or are similar to the generated 3D structure, thereby presenting options to the user. This allows the user to compare the generated object with similar commercially available products and select the optimal option.

[0264] For example, if a user needs a shelf of a specific shape and size, they can scan the wall with their device and input a request in natural language, such as "make the shelf 180cm high and 90cm wide." The server parses this prompt, generates a 3D structure of the corresponding shelf, and proposes it. The user can then receive information on commercially available products similar to the generated shelf to help them make a selection.

[0265] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0266] Step 1:

[0267] The device uses a LiDAR sensor to acquire three-dimensional shape data of physical objects and space. The data obtained by this measurement method is recorded in the device as point cloud data. The input is the physical state of the object, and the output is digital three-dimensional shape data.

[0268] Step 2:

[0269] The user inputs requests about objects in natural language through the terminal's interactive interface. For example, they might give the instruction, "Make the shelf 180cm high and 90cm wide." The input is the user's request and is stored in the terminal as a prompt. This prompt is used for subsequent processing.

[0270] Step 3:

[0271] The terminal sends the acquired 3D shape data and the user's natural language request (prompt) to the server. In this data transfer, the shape data and user request are inputs, and the output is a signal to prompt the server to start processing.

[0272] Step 4:

[0273] The server processes the received 3D shape data and prompt text using a generative AI model. The input consists of data and requests received from the terminal, and the output is design guidelines for a 3D structure that meets the user requirements. The data is integrated by a machine learning algorithm to generate the optimal shape based on the requirements.

[0274] Step 5:

[0275] The server generates a 3D structure file based on the analysis results. Using machine learning techniques, the data is converted to the .obj format, and a model suitable for the user's requirements is generated. The input is design guidelines, and the output is a 3D structure file in .obj format.

[0276] Step 6:

[0277] The server searches for existing products similar to the generated 3D structure. It accesses a database and uses feature extraction techniques to find similar products. The input is the generated 3D structure, and the output is a list of similar products.

[0278] Step 7:

[0279] The server sends the generated 3D structure file and information on similar products found to the terminal. The user receives this information and can compare the generated product with commercially available products to make purchasing or usage decisions. The input is the data of the generated product and information on similar products, and the output is information provided to the user.

[0280] (Application Example 1)

[0281] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0282] Modern consumers have a need to easily customize items through natural interactions even in a virtual environment. However, ordinary three-dimensional modeling technologies and product catalogs require advanced expertise and manual operations, and are not easily accessible to everyone. There is also a problem that it is difficult to confirm in real time the visualization and customization results in the virtual space.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0284] In this invention, the server includes a distance measuring device for acquiring three-dimensional shape data, a communication means for receiving a request from a user as natural language, a generating device for automatically generating three-dimensional structure data based on the three-dimensional shape data and the natural language request, a search device for searching for and presenting existing items similar to the generated three-dimensional structure data, and a display device for presenting the visualization of the item and the adjusted three-dimensional structure data through a virtual visual device. Thereby, even without expertise, the user can easily customize three-dimensional objects and visually confirm them in real time in a virtual environment.

[0285] "Three-dimensional shape data" is digital data for expressing the three-dimensional structure of an object or space, and includes coordinate and distance information.

[0286] "Distance measuring device" is a device that measures the distance to an object using light or other technologies and acquires three-dimensional shape data.

[0287] "Communication means" is an interface for receiving requests and instructions from a user as natural language and processing that information within the system.

[0288] "Generating device" is a device having a function of automatically generating new three-dimensional structure data or models based on the acquired three-dimensional shape data and the user's request.

[0289] A "search device" is a device that finds existing items or information in a database that are similar to the generated three-dimensional structural data, and presents them to the user.

[0290] A "virtual visual device" is a display device that visualizes objects generated based on digital data, allowing users to experience them virtually.

[0291] The present invention provides a system that allows users to easily customize three-dimensional objects in a virtual space. Users can collect data on the three-dimensional shape in real time using a virtual vision device such as smart glasses. Specifically, the following hardware and software are used.

[0292] The server acquires three-dimensional shape data of objects and space through a distance measuring device equipped with a LiDAR sensor. Users input their customization requests in natural language via communication. Natural language processing software then parses these requests. Specifically, natural language processing libraries using Python are often used.

[0293] On the server side, based on the acquired 3D shape data and the user's request, the generation device uses OpenAI's generation AI model to generate customized 3D structural data. Based on this generated data, the search device searches the database for similar items and presents them to the user. Through this series of processes, the user can visually confirm what kind of customization is possible.

[0294] The virtual vision system enables real-time display to visualize the generated three-dimensional model. As a result, an environment is created where users can easily check the quality of the model.

[0295] As a concrete example, consider a scenario in a furniture store's virtual showroom where a user visualizes a chair in a virtual space and gives instructions such as, "I want the height lowered by 5 cm and the color changed to blue." Based on this request, the server uses a generative AI model to generate a 3D model according to the instructions. The user can then review this model in the virtual environment and use it as a basis for deciding whether to order the actual product.

[0296] Example of a prompt:

[0297] "Based on the chair data, please generate a new 3D model with the height reduced by 5 centimeters and the color changed to blue."

[0298] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0299] Step 1:

[0300] The user uses smart glasses to visually identify a specific object in a virtual space and acquires three-dimensional shape data using a distance measuring device. The input is the physical information of the object, and the output is its three-dimensional shape data. The distance measuring device uses a LiDAR sensor to scan the surface of the object and generates its coordinate information as digital data.

[0301] Step 2:

[0302] The user sends customization requests to the server in natural language via a communication method. The input is a natural language instruction, and the output is the parsed request data. The server uses a natural language processing library to parse the input data and extract the specific changes (e.g., "lower the height by 5cm," "change the color to blue").

[0303] Step 3:

[0304] Based on the acquired three-dimensional shape data and the analyzed request data, the server generates customized three-dimensional structure data using a generation device. The input includes the three-dimensional shape data and the request data, and the output is the customized three-dimensional structure data. The generation AI model adjusts the model, and a three-dimensional model as required is generated in.obj format.

[0305] Step 4:

[0306] Using the customized three-dimensional structure data, the server searches for similar existing items from the database and presents the information to the user. The input is the customized three-dimensional structure data, and the output is the similar item information. The search device executes a database query to find similar ones under specific conditions.

[0307] Step 5:

[0308] The user visually checks the generated customized model using a virtual visual device. The input is the generated three-dimensional structure data, and the output is the visualized object display. The virtual visual device renders the data in real time and provides visual feedback to the user.

[0309] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0310] This invention is a system that generates a customized three-dimensional object that reflects not only the user's request but also the emotional state. The depth measurement means incorporated in the terminal uses LiDAR to acquire the three-dimensional shape data of an object or space and incorporates it into the shape selected by the user.

[0311] The device also provides a chat interface to receive requests in natural language through interaction with the user. In addition, it incorporates an emotion engine that analyzes the user's emotions. The emotion engine uses facial recognition and voice tone analysis to identify the user's emotions and extract emotional information such as "happiness" or "calmness." This makes it possible to take into account design elements that correspond to the user's mental state.

[0312] The server has a generation mechanism for generating 3D model files based on 3D shape data and natural language requests. Using machine learning algorithms, it creates 3D models that reflect the user's requests and emotions. Emotional information can influence subtle aspects of the design; for example, positive emotions might be associated with brighter colors and more curvilinear designs.

[0313] When a user requests a special product design based on a specific theme, they first scan the target area with LiDAR on their device, and then enter specific requests, such as "I want some small items with a cheerful atmosphere," via a chat interface. When the emotion engine detects joy from the user's facial expressions and voice, that emotion information is also transmitted to the server.

[0314] The server processes the received data to generate a new 3D model, and in the process, it also searches for similar products. This allows the user to review the newly generated custom model and select existing similar products that match their preferences.

[0315] As described above, the present invention allows users to easily obtain emotionally tailored customized products without requiring any technical knowledge.

[0316] The following describes the processing flow.

[0317] Step 1:

[0318] The user selects an object to scan using the device and activates the LiDAR sensor. The device uses depth measurement to acquire three-dimensional shape data of the object and temporarily stores that data.

[0319] Step 2:

[0320] The device receives requests from users in natural language via a chat interface. Users can input specific requests such as "bright design" or "curved shape."

[0321] Step 3:

[0322] The device activates an emotion engine and uses its camera and microphone to recognize the user's emotions. Through facial recognition and voice analysis, it identifies the user's emotional state and acquires that information.

[0323] Step 4:

[0324] The terminal transmits acquired 3D shape data, natural language requests, and sentiment information to the server. This data is securely transmitted through a secure communication protocol.

[0325] Step 5:

[0326] The server analyzes the received natural language request and converts it into technical instructions. Using machine learning algorithms, it generates a 3D model file that combines the analysis results with sentiment information.

[0327] Step 6:

[0328] The server uses the generated 3D model to search the database for similar products. It extracts information on existing products with a high degree of similarity and prepares the data to present to the user.

[0329] Step 7:

[0330] The terminal displays the generated 3D model and similar product information received from the server to the user. The user can view the model in 3D preview and, if necessary, also view details of similar products.

[0331] Step 8:

[0332] Users can select a generated 3D model and either print it using a 3D printer or save it digitally. They can also purchase similar products suggested, expanding their product choices.

[0333] (Example 2)

[0334] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0335] Conventional 3D model generation systems have the problem of being unable to customize models to take into account the emotional state of the user, and therefore being unable to generate models that meet the emotions and needs of individual users. Furthermore, there was a lack of a simple way to understand the relationship between the generated 3D models and existing products in the market.

[0336] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0337] In this invention, the server includes a measurement means for acquiring three-dimensional shape information, a dialogue means for receiving user requests in natural language, a model generation means for automatically generating a three-dimensional model based on the three-dimensional shape information and the natural language requests, a search means for searching for and presenting existing items similar to the generated three-dimensional model, and an emotion analysis means for analyzing the user's emotional state and reflecting the emotional information in the three-dimensional model. This makes it possible to generate a customized model that matches the user's emotions and requests, and to present similar existing items.

[0338] "Three-dimensional shape information" refers to data that represents and records the physical shape of an object or space in three dimensions.

[0339] "Measurement means" refers to devices and methods used to acquire the three-dimensional shape of an object, utilizing technologies such as light detection and distance measurement.

[0340] A "user" refers to a person who operates this system and has specific requests or objectives.

[0341] "Natural language" refers to the linguistic forms that people use on a daily basis, and the language that users use to communicate requests to a system.

[0342] A "dialogue tool" is an interface for communicating with users and receiving requests in natural language.

[0343] A "3D model" is a digital representation generated based on data of a three-dimensional shape; it is a computer model that mimics the physical shape.

[0344] "Model generation means" refers to devices or methods used to automatically generate three-dimensional models based on input data, and includes machine learning algorithms.

[0345] A "search tool" refers to a device or method for finding existing items similar to the generated three-dimensional model in a database and presenting them to the user.

[0346] "Emotional state" refers to information that indicates the user's current emotional situation, and refers to the psychological state extracted from facial expressions and tone of voice.

[0347] "Emotional analysis means" refers to devices or methods for analyzing and identifying a user's emotions, and utilizes facial recognition and voice analysis technologies.

[0348] This invention relates to a system that generates customized 3D models that reflect the user's requests and emotional state. The system implementation mainly consists of terminals and servers.

[0349] The terminal incorporates sensors that utilize optical detection and ranging technologies as measurement means to acquire three-dimensional shape information. This measurement means efficiently acquires three-dimensional shape data from the user's environment and saves it as point cloud data. The user can input their requests in natural language through a chat interface, which serves as a means of interaction. In this case, specific requests such as "I want some small items that give off a bright atmosphere" are entered as an example of prompt text.

[0350] The device also includes emotion analysis capabilities to analyze the user's emotional state. It uses a camera and microphone to capture facial expressions and voice tone, thereby identifying the user's emotions. For example, feelings of joy and happiness can be detected, and this emotional information is reflected in the model's design.

[0351] The server receives three-dimensional shape data, natural language requests, and sentiment information transmitted from the terminal. It then automatically generates a 3D model based on a generative AI model using machine learning as the generation method. In this process, for example, if positive sentiment information is received, bright colors and curvilinear design elements are incorporated into the model.

[0352] Furthermore, the server searches its database for existing items similar to the generated 3D model and presents them to the user. This allows the user to compare the newly generated custom model with similar commercially available products and make a suitable selection.

[0353] In this way, the system allows users to obtain appropriate and emotionally tailored customized products without requiring any special technical knowledge.

[0354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0355] Step 1:

[0356] The device activates its LiDAR sensor via user input to acquire three-dimensional shape data of surrounding objects and space. The sensor uses light detection and ranging techniques to collect detailed point cloud data of the target. This input data is stored as a basis for future 3D model generation.

[0357] Step 2:

[0358] The terminal receives user requests in natural language using a chat interface. For example, the user might type, "I want some small items with a bright atmosphere." The terminal records this natural language request as text data and prepares it for subsequent processing.

[0359] Step 3:

[0360] The device uses a camera and microphone to capture the user's facial expressions and voice, and performs emotion analysis. The emotion analysis system processes this input data and generates emotional information such as "happiness" or "calmness." This information is stored as data that influences design elements.

[0361] Step 4:

[0362] The terminal sends acquired 3D shape data, natural language requests, and sentiment information to the server as data packets. Here, accurate data transmission is ensured using communication methods. The output of this step is a complete set of user information for the server to analyze.

[0363] Step 5:

[0364] The server creates a 3D model using a generative AI model based on the received data. The machine learning algorithm designs the 3D model, reflecting user requests and emotional information. Here, bright colors and curvilinear designs are incorporated based on positive emotions. The generated 3D model is then prepared for further processing.

[0365] Step 6:

[0366] The server uses the generated 3D model to search its database for similar existing items. The search extracts information on similar products, which is also prepared for user presentation.

[0367] Step 7:

[0368] The terminal displays the 3D model and similar product information returned from the server to the user. The user can visually review the generated custom model and compare it with suggested similar products. The output of this step provides the user with information to make selections and decisions.

[0369] (Application Example 2)

[0370] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0371] In today's commercial spaces, it is crucial to quickly customize products based on individual customer emotions and needs. However, conventional technologies have struggled to efficiently customize products while considering customer emotions, creating a need for methods to enhance customer satisfaction. Furthermore, there has been a lack of systems capable of providing product recommendations utilizing real-time sentiment analysis.

[0372] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0373] In this invention, the server includes depth detection means for acquiring three-dimensional shape information, dialogue interface means for receiving user requests in natural language, and emotion recognition means for analyzing the user's emotional state. This makes it possible to quickly generate and present customized products that meet the customer's emotions and requests.

[0374] "Three-dimensional shape information" refers to data that indicates the three-dimensional shape and position of an object in space.

[0375] A "depth detection method" is a method of acquiring three-dimensional shape information using devices or techniques that measure the distance to an object.

[0376] "Users" refers to customers who use this system to customize products.

[0377] "Natural language" refers to the language that humans use on a daily basis, and is a means of communication that does not rely on special codes or formats.

[0378] A "dialogue interface means" is a technology that provides an interface for receiving and processing user requests in natural language.

[0379] "Emotion identification means" refers to technology that analyzes and identifies emotions from a user's facial expressions, voice, etc.

[0380] A "3D shape file" is a digital file that stores three-dimensional shape data and is used for visualization and 3D printing.

[0381] "Generation method" refers to a system that automatically creates a 3D shape file based on the input data.

[0382] A "search tool" is a system element that has the function of searching for existing items similar to the generated three-dimensional shape and presenting them to the user.

[0383] This invention is a system that generates three-dimensional objects corresponding to the user's emotions. This system utilizes depth detection means, dialogue interface means, and emotion recognition means.

[0384] Hardware and software configuration:

[0385] server:

[0386] The server processes three-dimensional shape information and generates a 3D shape file based on natural language requests and sentiment data. Machine learning algorithms are used for generation. Specifically, Amazon Lex is used for natural language understanding, and Microsoft Azure's Emotion API is used for sentiment analysis.

[0387] Device (e.g., smart glasses):

[0388] The device is equipped with a LiDAR sensor to measure the distance to objects, a camera to capture the user's facial expressions, and a microphone to recognize voice. This allows the user's requests and emotional state to be transmitted to the server in real time. Blender can be used for 3D rendering.

[0389] Specific example:

[0390] As users move through a commercial space, they can input prompts through smart glasses, such as "I want furniture with a relaxing atmosphere." The smart glasses detect the user's calm facial expression and send the emotion "relaxation" to a server. Based on this information, the server generates 3D models of custom furniture with bright colors and rounded designs. Users can then view the generated designs on the spot and select the items they want.

[0391] Example of a prompt:

[0392] "I want to give this product as a gift to a friend, so I'd like the design to convey my joy."

[0393] "I'd like the design to use warmer colors."

[0394] This invention allows users to obtain customized products that cater to their emotions at that moment, without requiring any technical knowledge.

[0395] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0396] Step 1:

[0397] The user wears smart glasses and points the camera at an object of interest. The device uses a LiDAR sensor to acquire three-dimensional shape information. This information is stored on the device as data representing the shape and position of the object in space.

[0398] Step 2:

[0399] The user inputs their request in natural language into the microphone of the smart glasses. For example, they might request "make the design brighter." The device converts the voice into text and saves the request as digital data through a dialogue interface.

[0400] Step 3:

[0401] The device captures the user's facial expressions with its camera and analyzes the user's emotional state using emotion recognition technology. The analyzed emotion data (e.g., "joy" or "calmness") is sent to the server as numerical data.

[0402] Step 4:

[0403] The server receives 3D shape information from LiDAR, request data from voice input, and emotion data. Based on this data, a generative AI model within the server generates a customized 3D shape file. Machine learning algorithms are used in this process.

[0404] Step 5:

[0405] The generated 3D shape file is sent from the server to the terminal and displayed on the smart glasses' screen. The user can view the proposed custom design through the smart glasses.

[0406] Step 6:

[0407] The server then searches for existing items similar to the generated 3D shape file and presents that information to the user. This allows the user to select not only the newly generated design but also existing products.

[0408] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0409] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0410] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0411] [Third Embodiment]

[0412] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0413] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0414] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0415] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0416] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0417] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0418] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0419] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0420] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0421] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0422] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0423] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0424] This invention is a system that enables users to easily generate customized three-dimensional objects. A depth measurement mechanism is incorporated into the terminal to quickly and accurately acquire three-dimensional shape data. By using LiDAR sensors and the like, shape data of physical objects and spaces can be acquired.

[0425] The terminal transmits the acquired 3D shape data to the server. The server analyzes the received data in combination with the user's natural language requests, and generates an appropriate 3D model file using a generation mechanism. In this process, the generation mechanism utilizes machine learning algorithms to generate the model that best matches the user's requests.

[0426] Specifically, when a user requires an object of a particular shape or size, the data scanned by the terminal and requests entered by the user in the chat interface, such as "make the height 10cm" or "make it yellow," are sent to the server. In response, the server generates a 3D model file in .obj format via a generation mechanism and provides it to the user's terminal.

[0427] Furthermore, the server searches its database for existing products similar to the generated 3D model. If similar products are found, this information is provided to the user, allowing them to choose between the new model and commercially available products. This enables users to efficiently find the best option to suit their needs.

[0428] In this way, even users without special 3D modeling knowledge can easily design and obtain objects to solve everyday inconveniences.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The user uses a device to select an object or space they wish to scan using depth measurement and activates the LiDAR sensor. The device then moves the sensor toward the target object, acquiring its three-dimensional shape data.

[0432] Step 2:

[0433] The terminal converts the acquired 3D shape data into a file format and sends it to a server in the cloud. A secure protocol is used for transmission to protect the confidentiality of the data.

[0434] Step 3:

[0435] Users use the device's chat interface to input their wishes and requests in natural language. This includes object dimensions, color, and other functional requirements.

[0436] Step 4:

[0437] The server analyzes the user's request using a natural language processing engine and converts it into technical specifications. The analyzed data and 3D shape data are then passed to the generation system.

[0438] Step 5:

[0439] The server generation method utilizes machine learning algorithms to generate 3D model files based on the analysis data. This generation uses formats such as .obj, and is optimized to meet user requirements.

[0440] Step 6:

[0441] The server searches for similar products based on the generated 3D model. It refers to existing product databases to identify product information that closely matches the model.

[0442] Step 7:

[0443] The terminal displays a 3D model generated from the server along with information on similar products to the user. The user can preview the generated model and view details of similar products.

[0444] Step 8:

[0445] The user selects a generated .obj file from the provided information and decides whether to print it on a 3D printer or purchase a similar product from the options presented. This allows the user to actually obtain an object that meets their needs.

[0446] (Example 1)

[0447] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] In modern life, there is a need for users to design and acquire customizable 3D objects without requiring special technical knowledge or expertise. However, conventional 3D modeling methods require specialized software and techniques, making them difficult for the average user to use. Therefore, there is a need for technology that allows even users without specialized knowledge to easily and quickly generate and acquire 3D objects.

[0449] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0450] In this invention, the server includes measuring means, interactive interface means, creation means, and search means. This allows users to intuitively communicate requests using natural language without special technical knowledge, efficiently generate three-dimensional objects based on those requests, and compare and select existing similar products.

[0451] "Measurement means" refers to devices and technologies for acquiring three-dimensional shape data, and in particular, it refers to systems that use light to accurately measure the distance and shape of an object.

[0452] An "interactive interface means" refers to a communication tool that receives instructions from users in natural language and links that information with a processing system.

[0453] "Creation method" refers to the technology and algorithms used to automatically generate a 3D structure file based on acquired 3D shape data and user instructions.

[0454] "Search method" refers to a function that searches a database for existing products similar to the generated three-dimensional structure and presents candidates to the user.

[0455] To concretely implement the invention, it is crucial to build a system that allows for cooperation between a server and a terminal. In this system, a distance measuring device utilizing light is used on the terminal as a means of acquiring three-dimensional shape data of physical objects and spaces. Specifically, LiDAR sensors are used. This allows the user to acquire detailed information about the distance and shape of objects.

[0456] The acquired 3D shape data is stored on the terminal, and in parallel, the user enters custom requests about the object in natural language through an interactive interface. This prompt serves as a convenient way to express the user's intent and is sent to the server.

[0457] The server utilizes a generative AI model to automatically generate a 3D structure file by combining the 3D shape data received through the creation method with user instructions. The machine learning algorithm used in this process performs complex data integration and analysis based on the user's requirements to generate the most suitable 3D structure.

[0458] Furthermore, the server can use search mechanisms to find existing products in its database that match or are similar to the generated 3D structure, thereby presenting options to the user. This allows the user to compare the generated object with similar commercially available products and select the optimal option.

[0459] For example, if a user needs a shelf of a specific shape and size, they can scan the wall with their device and input a request in natural language, such as "make the shelf 180cm high and 90cm wide." The server parses this prompt, generates a 3D structure of the corresponding shelf, and proposes it. The user can then receive information on commercially available products similar to the generated shelf to help them make a selection.

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] The device uses a LiDAR sensor to acquire three-dimensional shape data of physical objects and space. The data obtained by this measurement method is recorded in the device as point cloud data. The input is the physical state of the object, and the output is digital three-dimensional shape data.

[0463] Step 2:

[0464] The user inputs requests about objects in natural language through the terminal's interactive interface. For example, they might give the instruction, "Make the shelf 180cm high and 90cm wide." The input is the user's request and is stored in the terminal as a prompt. This prompt is used for subsequent processing.

[0465] Step 3:

[0466] The terminal sends the acquired 3D shape data and the user's natural language request (prompt) to the server. In this data transfer, the shape data and user request are inputs, and the output is a signal to prompt the server to start processing.

[0467] Step 4:

[0468] The server processes the received 3D shape data and prompt text using a generative AI model. The input consists of data and requests received from the terminal, and the output is design guidelines for a 3D structure that meets the user requirements. The data is integrated by a machine learning algorithm to generate the optimal shape based on the requirements.

[0469] Step 5:

[0470] The server generates a 3D structure file based on the analysis results. Using machine learning techniques, the data is converted to the .obj format, and a model suitable for the user's requirements is generated. The input is design guidelines, and the output is a 3D structure file in .obj format.

[0471] Step 6:

[0472] The server searches for existing products similar to the generated 3D structure. It accesses a database and uses feature extraction techniques to find similar products. The input is the generated 3D structure, and the output is a list of similar products.

[0473] Step 7:

[0474] The server sends the generated 3D structure file and information on similar products found to the terminal. The user receives this information and can compare the generated product with commercially available products to make purchasing or usage decisions. The input is the data of the generated product and information on similar products, and the output is information provided to the user.

[0475] (Application Example 1)

[0476] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0477] Modern consumers have a need to easily customize items through natural interaction, even in virtual environments. However, conventional 3D modeling techniques and product catalogs require advanced expertise and manual operation, making them not easily accessible to everyone. Furthermore, there is the problem of difficulty in real-time verification of visualization and customization results in virtual space.

[0478] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0479] In this invention, the server includes a distance measuring device for acquiring three-dimensional shape data, a communication means for receiving user requests in natural language, a generation device for automatically generating three-dimensional structure data based on the three-dimensional shape data and the natural language requests, a search device for searching for and presenting existing articles similar to the generated three-dimensional structure data, and a display device for presenting the visualized and adjusted three-dimensional structure data of the articles through a virtual vision device. This makes it possible for users to easily customize three-dimensional objects without specialized knowledge and to visually confirm them in real time in a virtual environment.

[0480] "Three-dimensional shape data" refers to digital data used to represent the three-dimensional structure of an object or space, and includes coordinate and distance information.

[0481] A "distance measuring device" is a device that uses light or other technologies to measure the distance to an object and acquire data on its three-dimensional shape.

[0482] A "communication method" is an interface that receives requests and instructions from users in natural language and processes that information within the system.

[0483] A "generation device" is a device that has the function of automatically generating new three-dimensional structural data or models based on acquired three-dimensional shape data and user requirements.

[0484] A "search device" is a device that finds existing items or information in a database that are similar to the generated three-dimensional structural data, and presents them to the user.

[0485] A "virtual visual device" is a display device that visualizes objects generated based on digital data, allowing users to experience them virtually.

[0486] The present invention provides a system that allows users to easily customize three-dimensional objects in a virtual space. Users can collect data on the three-dimensional shape in real time using a virtual vision device such as smart glasses. Specifically, the following hardware and software are used.

[0487] The server acquires three-dimensional shape data of objects and space through a distance measuring device equipped with a LiDAR sensor. Users input their customization requests in natural language via communication. Natural language processing software then parses these requests. Specifically, natural language processing libraries using Python are often used.

[0488] On the server side, based on the acquired 3D shape data and the user's request, the generation device uses OpenAI's generation AI model to generate customized 3D structural data. Based on this generated data, the search device searches the database for similar items and presents them to the user. Through this series of processes, the user can visually confirm what kind of customization is possible.

[0489] The virtual vision system enables real-time display to visualize the generated three-dimensional model. As a result, an environment is created where users can easily check the quality of the model.

[0490] As a concrete example, consider a scenario in a furniture store's virtual showroom where a user visualizes a chair in a virtual space and gives instructions such as, "I want the height lowered by 5 cm and the color changed to blue." Based on this request, the server uses a generative AI model to generate a 3D model according to the instructions. The user can then review this model in the virtual environment and use it as a basis for deciding whether to order the actual product.

[0491] Example of a prompt:

[0492] "Based on the chair data, please generate a new 3D model with the height reduced by 5 centimeters and the color changed to blue."

[0493] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0494] Step 1:

[0495] The user uses smart glasses to visually identify a specific object in a virtual space and acquires three-dimensional shape data using a distance measuring device. The input is the physical information of the object, and the output is its three-dimensional shape data. The distance measuring device uses a LiDAR sensor to scan the surface of the object and generates its coordinate information as digital data.

[0496] Step 2:

[0497] The user sends customization requests to the server in natural language via a communication method. The input is a natural language instruction, and the output is the parsed request data. The server uses a natural language processing library to parse the input data and extract the specific changes (e.g., "lower the height by 5cm," "change the color to blue").

[0498] Step 3:

[0499] The server uses a generation device to generate customized 3D structural data based on the acquired 3D shape data and analyzed request data. The input includes 3D shape data and request data, and the output is customized 3D structural data. The AI ​​generation model adjusts the model, and a 3D model matching the requirements is generated in .obj format.

[0500] Step 4:

[0501] The server uses customized three-dimensional structure data to search for similar existing items within the database and presents the information to the user. The input is customized three-dimensional structure data, and the output is information about similar items. The search device executes database queries to find similar items that meet specific criteria.

[0502] Step 5:

[0503] The user uses a virtual vision device to visually verify the generated customized model. The input is the generated three-dimensional structural data, and the output is a visualized object display. The virtual vision device renders the data in real time, providing the user with visual feedback.

[0504] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0505] This invention is a system that generates customized three-dimensional objects that reflect not only the user's requests but also their emotional state. A depth measurement means built into the terminal uses LiDAR to acquire three-dimensional shape data of objects and space, and incorporates this into the shape selected by the user.

[0506] The device also provides a chat interface to receive requests in natural language through interaction with the user. In addition, it incorporates an emotion engine that analyzes the user's emotions. The emotion engine uses facial recognition and voice tone analysis to identify the user's emotions and extract emotional information such as "happiness" or "calmness." This makes it possible to take into account design elements that correspond to the user's mental state.

[0507] The server has a generation mechanism for generating 3D model files based on 3D shape data and natural language requests. Using machine learning algorithms, it creates 3D models that reflect the user's requests and emotions. Emotional information can influence subtle aspects of the design; for example, positive emotions might be associated with brighter colors and more curvilinear designs.

[0508] When a user requests a special product design based on a specific theme, they first scan the target area with LiDAR on their device, and then enter specific requests, such as "I want some small items with a cheerful atmosphere," via a chat interface. When the emotion engine detects joy from the user's facial expressions and voice, that emotion information is also transmitted to the server.

[0509] The server processes the received data to generate a new 3D model, and in the process, it also searches for similar products. This allows the user to review the newly generated custom model and select existing similar products that match their preferences.

[0510] As described above, the present invention allows users to easily obtain emotionally tailored customized products without requiring any technical knowledge.

[0511] The following describes the processing flow.

[0512] Step 1:

[0513] The user selects an object to scan using the device and activates the LiDAR sensor. The device uses depth measurement to acquire three-dimensional shape data of the object and temporarily stores that data.

[0514] Step 2:

[0515] The device receives requests from users in natural language via a chat interface. Users can input specific requests such as "bright design" or "curved shape."

[0516] Step 3:

[0517] The device activates an emotion engine and uses its camera and microphone to recognize the user's emotions. Through facial recognition and voice analysis, it identifies the user's emotional state and acquires that information.

[0518] Step 4:

[0519] The terminal transmits acquired 3D shape data, natural language requests, and sentiment information to the server. This data is securely transmitted through a secure communication protocol.

[0520] Step 5:

[0521] The server analyzes the received natural language request and converts it into technical instructions. Using machine learning algorithms, it generates a 3D model file that combines the analysis results with sentiment information.

[0522] Step 6:

[0523] The server uses the generated 3D model to search the database for similar products. It extracts information on existing products with a high degree of similarity and prepares the data to present to the user.

[0524] Step 7:

[0525] The terminal displays the generated 3D model and similar product information received from the server to the user. The user can view the model in 3D preview and, if necessary, also view details of similar products.

[0526] Step 8:

[0527] Users can select a generated 3D model and either print it using a 3D printer or save it digitally. They can also purchase similar products suggested, expanding their product choices.

[0528] (Example 2)

[0529] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0530] Conventional 3D model generation systems have the problem of being unable to customize models to take into account the emotional state of the user, and therefore being unable to generate models that meet the emotions and needs of individual users. Furthermore, there was a lack of a simple way to understand the relationship between the generated 3D models and existing products in the market.

[0531] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0532] In this invention, the server includes a measurement means for acquiring three-dimensional shape information, a dialogue means for receiving user requests in natural language, a model generation means for automatically generating a three-dimensional model based on the three-dimensional shape information and the natural language requests, a search means for searching for and presenting existing items similar to the generated three-dimensional model, and an emotion analysis means for analyzing the user's emotional state and reflecting the emotional information in the three-dimensional model. This makes it possible to generate a customized model that matches the user's emotions and requests, and to present similar existing items.

[0533] "Three-dimensional shape information" refers to data that represents and records the physical shape of an object or space in three dimensions.

[0534] "Measurement means" refers to devices and methods used to acquire the three-dimensional shape of an object, utilizing technologies such as light detection and distance measurement.

[0535] A "user" refers to a person who operates this system and has specific requests or objectives.

[0536] "Natural language" refers to the linguistic forms that people use on a daily basis, and the language that users use to communicate requests to a system.

[0537] A "dialogue tool" is an interface for communicating with users and receiving requests in natural language.

[0538] A "3D model" is a digital representation generated based on data of a three-dimensional shape; it is a computer model that mimics the physical shape.

[0539] "Model generation means" refers to devices or methods used to automatically generate three-dimensional models based on input data, and includes machine learning algorithms.

[0540] A "search tool" refers to a device or method for finding existing items similar to the generated three-dimensional model in a database and presenting them to the user.

[0541] "Emotional state" refers to information that indicates the user's current emotional situation, and refers to the psychological state extracted from facial expressions and tone of voice.

[0542] "Emotional analysis means" refers to devices or methods for analyzing and identifying a user's emotions, and utilizes facial recognition and voice analysis technologies.

[0543] This invention relates to a system that generates customized 3D models that reflect the user's requests and emotional state. The system implementation mainly consists of terminals and servers.

[0544] The terminal incorporates sensors that utilize optical detection and ranging technologies as measurement means to acquire three-dimensional shape information. This measurement means efficiently acquires three-dimensional shape data from the user's environment and saves it as point cloud data. The user can input their requests in natural language through a chat interface, which serves as a means of interaction. In this case, specific requests such as "I want some small items that give off a bright atmosphere" are entered as an example of prompt text.

[0545] The device also includes emotion analysis capabilities to analyze the user's emotional state. It uses a camera and microphone to capture facial expressions and voice tone, thereby identifying the user's emotions. For example, feelings of joy and happiness can be detected, and this emotional information is reflected in the model's design.

[0546] The server receives three-dimensional shape data, natural language requests, and sentiment information transmitted from the terminal. It then automatically generates a 3D model based on a generative AI model using machine learning as the generation method. In this process, for example, if positive sentiment information is received, bright colors and curvilinear design elements are incorporated into the model.

[0547] Furthermore, the server searches its database for existing items similar to the generated 3D model and presents them to the user. This allows the user to compare the newly generated custom model with similar commercially available products and make a suitable selection.

[0548] In this way, the system allows users to obtain appropriate and emotionally tailored customized products without requiring any special technical knowledge.

[0549] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0550] Step 1:

[0551] The device activates its LiDAR sensor via user input to acquire three-dimensional shape data of surrounding objects and space. The sensor uses light detection and ranging techniques to collect detailed point cloud data of the target. This input data is stored as a basis for future 3D model generation.

[0552] Step 2:

[0553] The terminal receives user requests in natural language using a chat interface. For example, the user might type, "I want some small items with a bright atmosphere." The terminal records this natural language request as text data and prepares it for subsequent processing.

[0554] Step 3:

[0555] The device uses a camera and microphone to capture the user's facial expressions and voice, and performs emotion analysis. The emotion analysis system processes this input data and generates emotional information such as "happiness" or "calmness." This information is stored as data that influences design elements.

[0556] Step 4:

[0557] The terminal sends acquired 3D shape data, natural language requests, and sentiment information to the server as data packets. Here, accurate data transmission is ensured using communication methods. The output of this step is a complete set of user information for the server to analyze.

[0558] Step 5:

[0559] The server creates a 3D model using a generative AI model based on the received data. The machine learning algorithm designs the 3D model, reflecting user requests and emotional information. Here, bright colors and curvilinear designs are incorporated based on positive emotions. The generated 3D model is then prepared for further processing.

[0560] Step 6:

[0561] The server uses the generated 3D model to search its database for similar existing items. The search extracts information on similar products, which is also prepared for user presentation.

[0562] Step 7:

[0563] The terminal displays the 3D model and similar product information returned from the server to the user. The user can visually review the generated custom model and compare it with suggested similar products. The output of this step provides the user with information to make selections and decisions.

[0564] (Application Example 2)

[0565] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0566] In today's commercial spaces, it is crucial to quickly customize products based on individual customer emotions and needs. However, conventional technologies have struggled to efficiently customize products while considering customer emotions, creating a need for methods to enhance customer satisfaction. Furthermore, there has been a lack of systems capable of providing product recommendations utilizing real-time sentiment analysis.

[0567] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0568] In this invention, the server includes depth detection means for acquiring three-dimensional shape information, dialogue interface means for receiving user requests in natural language, and emotion recognition means for analyzing the user's emotional state. This makes it possible to quickly generate and present customized products that meet the customer's emotions and requests.

[0569] "Three-dimensional shape information" refers to data that indicates the three-dimensional shape and position of an object in space.

[0570] A "depth detection method" is a method of acquiring three-dimensional shape information using devices or techniques that measure the distance to an object.

[0571] "Users" refers to customers who use this system to customize products.

[0572] "Natural language" refers to the language that humans use on a daily basis, and is a means of communication that does not rely on special codes or formats.

[0573] A "dialogue interface means" is a technology that provides an interface for receiving and processing user requests in natural language.

[0574] "Emotion identification means" refers to technology that analyzes and identifies emotions from a user's facial expressions, voice, etc.

[0575] A "3D shape file" is a digital file that stores three-dimensional shape data and is used for visualization and 3D printing.

[0576] "Generation method" refers to a system that automatically creates a 3D shape file based on the input data.

[0577] A "search tool" is a system element that has the function of searching for existing items similar to the generated three-dimensional shape and presenting them to the user.

[0578] This invention is a system that generates three-dimensional objects corresponding to the user's emotions. This system utilizes depth detection means, dialogue interface means, and emotion recognition means.

[0579] Hardware and software configuration:

[0580] server:

[0581] The server processes three-dimensional shape information and generates a 3D shape file based on natural language requests and sentiment data. Machine learning algorithms are used for generation. Specifically, Amazon Lex is used for natural language understanding, and Microsoft Azure's Emotion API is used for sentiment analysis.

[0582] Device (e.g., smart glasses):

[0583] The device is equipped with a LiDAR sensor to measure the distance to objects, a camera to capture the user's facial expressions, and a microphone to recognize voice. This allows the user's requests and emotional state to be transmitted to the server in real time. Blender can be used for 3D rendering.

[0584] Specific example:

[0585] As users move through a commercial space, they can input prompts through smart glasses, such as "I want furniture with a relaxing atmosphere." The smart glasses detect the user's calm facial expression and send the emotion "relaxation" to a server. Based on this information, the server generates 3D models of custom furniture with bright colors and rounded designs. Users can then view the generated designs on the spot and select the items they want.

[0586] Example of a prompt:

[0587] "I want to give this product as a gift to a friend, so I'd like the design to convey my joy."

[0588] "I'd like the design to use warmer colors."

[0589] This invention allows users to obtain customized products that cater to their emotions at that moment, without requiring any technical knowledge.

[0590] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0591] Step 1:

[0592] The user wears smart glasses and points the camera at an object of interest. The device uses a LiDAR sensor to acquire three-dimensional shape information. This information is stored on the device as data representing the shape and position of the object in space.

[0593] Step 2:

[0594] The user inputs their request in natural language into the microphone of the smart glasses. For example, they might request "make the design brighter." The device converts the voice into text and saves the request as digital data through a dialogue interface.

[0595] Step 3:

[0596] The device captures the user's facial expressions with its camera and analyzes the user's emotional state using emotion recognition technology. The analyzed emotion data (e.g., "joy" or "calmness") is sent to the server as numerical data.

[0597] Step 4:

[0598] The server receives 3D shape information from LiDAR, request data from voice input, and emotion data. Based on this data, a generative AI model within the server generates a customized 3D shape file. Machine learning algorithms are used in this process.

[0599] Step 5:

[0600] The generated 3D shape file is sent from the server to the terminal and displayed on the smart glasses' screen. The user can view the proposed custom design through the smart glasses.

[0601] Step 6:

[0602] The server then searches for existing items similar to the generated 3D shape file and presents that information to the user. This allows the user to select not only the newly generated design but also existing products.

[0603] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0604] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0605] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0606] [Fourth Embodiment]

[0607] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0608] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0609] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0611] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0613] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0614] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0615] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0616] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0617] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0618] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0619] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] This invention is a system that enables users to easily generate customized three-dimensional objects. A depth measurement mechanism is incorporated into the terminal to quickly and accurately acquire three-dimensional shape data. By using LiDAR sensors and the like, shape data of physical objects and spaces can be acquired.

[0621] The terminal transmits the acquired 3D shape data to the server. The server analyzes the received data in combination with the user's natural language requests, and generates an appropriate 3D model file using a generation mechanism. In this process, the generation mechanism utilizes machine learning algorithms to generate the model that best matches the user's requests.

[0622] Specifically, when a user requires an object of a particular shape or size, the data scanned by the terminal and requests entered by the user in the chat interface, such as "make the height 10cm" or "make it yellow," are sent to the server. In response, the server generates a 3D model file in .obj format via a generation mechanism and provides it to the user's terminal.

[0623] Furthermore, the server searches its database for existing products similar to the generated 3D model. If similar products are found, this information is provided to the user, allowing them to choose between the new model and commercially available products. This enables users to efficiently find the best option to suit their needs.

[0624] In this way, even users without special 3D modeling knowledge can easily design and obtain objects to solve everyday inconveniences.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] The user uses a device to select an object or space they wish to scan using depth measurement and activates the LiDAR sensor. The device then moves the sensor toward the target object, acquiring its three-dimensional shape data.

[0628] Step 2:

[0629] The terminal converts the acquired 3D shape data into a file format and sends it to a server in the cloud. A secure protocol is used for transmission to protect the confidentiality of the data.

[0630] Step 3:

[0631] Users use the device's chat interface to input their wishes and requests in natural language. This includes object dimensions, color, and other functional requirements.

[0632] Step 4:

[0633] The server analyzes the user's request using a natural language processing engine and converts it into technical specifications. The analyzed data and 3D shape data are then passed to the generation system.

[0634] Step 5:

[0635] The server generation method utilizes machine learning algorithms to generate 3D model files based on the analysis data. This generation uses formats such as .obj, and is optimized to meet user requirements.

[0636] Step 6:

[0637] The server searches for similar products based on the generated 3D model. It refers to existing product databases to identify product information that closely matches the model.

[0638] Step 7:

[0639] The terminal displays a 3D model generated from the server along with information on similar products to the user. The user can preview the generated model and view details of similar products.

[0640] Step 8:

[0641] The user selects a generated .obj file from the provided information and decides whether to print it on a 3D printer or purchase a similar product from the options presented. This allows the user to actually obtain an object that meets their needs.

[0642] (Example 1)

[0643] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] In modern life, there is a need for users to design and acquire customizable 3D objects without requiring special technical knowledge or expertise. However, conventional 3D modeling methods require specialized software and techniques, making them difficult for the average user to use. Therefore, there is a need for technology that allows even users without specialized knowledge to easily and quickly generate and acquire 3D objects.

[0645] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0646] In this invention, the server includes measuring means, interactive interface means, creation means, and search means. This allows users to intuitively communicate requests using natural language without special technical knowledge, efficiently generate three-dimensional objects based on those requests, and compare and select existing similar products.

[0647] "Measurement means" refers to devices and technologies for acquiring three-dimensional shape data, and in particular, it refers to systems that use light to accurately measure the distance and shape of an object.

[0648] An "interactive interface means" refers to a communication tool that receives instructions from users in natural language and links that information with a processing system.

[0649] "Creation method" refers to the technology and algorithms used to automatically generate a 3D structure file based on acquired 3D shape data and user instructions.

[0650] "Search method" refers to a function that searches a database for existing products similar to the generated three-dimensional structure and presents candidates to the user.

[0651] To concretely implement the invention, it is crucial to build a system that allows for cooperation between a server and a terminal. In this system, a distance measuring device utilizing light is used on the terminal as a means of acquiring three-dimensional shape data of physical objects and spaces. Specifically, LiDAR sensors are used. This allows the user to acquire detailed information about the distance and shape of objects.

[0652] The acquired 3D shape data is stored on the terminal, and in parallel, the user enters custom requests about the object in natural language through an interactive interface. This prompt serves as a convenient way to express the user's intent and is sent to the server.

[0653] The server utilizes a generative AI model to automatically generate a 3D structure file by combining the 3D shape data received through the creation method with user instructions. The machine learning algorithm used in this process performs complex data integration and analysis based on the user's requirements to generate the most suitable 3D structure.

[0654] Furthermore, the server can use search mechanisms to find existing products in its database that match or are similar to the generated 3D structure, thereby presenting options to the user. This allows the user to compare the generated object with similar commercially available products and select the optimal option.

[0655] For example, if a user needs a shelf of a specific shape and size, they can scan the wall with their device and input a request in natural language, such as "make the shelf 180cm high and 90cm wide." The server parses this prompt, generates a 3D structure of the corresponding shelf, and proposes it. The user can then receive information on commercially available products similar to the generated shelf to help them make a selection.

[0656] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0657] Step 1:

[0658] The device uses a LiDAR sensor to acquire three-dimensional shape data of physical objects and space. The data obtained by this measurement method is recorded in the device as point cloud data. The input is the physical state of the object, and the output is digital three-dimensional shape data.

[0659] Step 2:

[0660] The user inputs requests about objects in natural language through the terminal's interactive interface. For example, they might give the instruction, "Make the shelf 180cm high and 90cm wide." The input is the user's request and is stored in the terminal as a prompt. This prompt is used for subsequent processing.

[0661] Step 3:

[0662] The terminal sends the acquired 3D shape data and the user's natural language request (prompt) to the server. In this data transfer, the shape data and user request are inputs, and the output is a signal to prompt the server to start processing.

[0663] Step 4:

[0664] The server processes the received 3D shape data and prompt text using a generative AI model. The input consists of data and requests received from the terminal, and the output is design guidelines for a 3D structure that meets the user requirements. The data is integrated by a machine learning algorithm to generate the optimal shape based on the requirements.

[0665] Step 5:

[0666] The server generates a 3D structure file based on the analysis results. Using machine learning techniques, the data is converted to the .obj format, and a model suitable for the user's requirements is generated. The input is design guidelines, and the output is a 3D structure file in .obj format.

[0667] Step 6:

[0668] The server searches for existing products similar to the generated 3D structure. It accesses a database and uses feature extraction techniques to find similar products. The input is the generated 3D structure, and the output is a list of similar products.

[0669] Step 7:

[0670] The server sends the generated 3D structure file and information on similar products found to the terminal. The user receives this information and can compare the generated product with commercially available products to make purchasing or usage decisions. The input is the data of the generated product and information on similar products, and the output is information provided to the user.

[0671] (Application Example 1)

[0672] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0673] Modern consumers have a need to easily customize items through natural interaction, even in virtual environments. However, conventional 3D modeling techniques and product catalogs require advanced expertise and manual operation, making them not easily accessible to everyone. Furthermore, there is the problem of difficulty in real-time verification of visualization and customization results in virtual space.

[0674] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0675] In this invention, the server includes a distance measuring device for acquiring three-dimensional shape data, a communication means for receiving user requests in natural language, a generation device for automatically generating three-dimensional structure data based on the three-dimensional shape data and the natural language requests, a search device for searching for and presenting existing articles similar to the generated three-dimensional structure data, and a display device for presenting the visualized and adjusted three-dimensional structure data of the articles through a virtual vision device. This makes it possible for users to easily customize three-dimensional objects without specialized knowledge and to visually confirm them in real time in a virtual environment.

[0676] "Three-dimensional shape data" refers to digital data used to represent the three-dimensional structure of an object or space, and includes coordinate and distance information.

[0677] A "distance measuring device" is a device that uses light or other technologies to measure the distance to an object and acquire data on its three-dimensional shape.

[0678] A "communication method" is an interface that receives requests and instructions from users in natural language and processes that information within the system.

[0679] A "generation device" is a device that has the function of automatically generating new three-dimensional structural data or models based on acquired three-dimensional shape data and user requirements.

[0680] A "search device" is a device that finds existing items or information in a database that are similar to the generated three-dimensional structural data, and presents them to the user.

[0681] A "virtual visual device" is a display device that visualizes objects generated based on digital data, allowing users to experience them virtually.

[0682] The present invention provides a system that allows users to easily customize three-dimensional objects in a virtual space. Users can collect data on the three-dimensional shape in real time using a virtual vision device such as smart glasses. Specifically, the following hardware and software are used.

[0683] The server acquires three-dimensional shape data of objects and space through a distance measuring device equipped with a LiDAR sensor. Users input their customization requests in natural language via communication. Natural language processing software then parses these requests. Specifically, natural language processing libraries using Python are often used.

[0684] On the server side, based on the acquired 3D shape data and the user's request, the generation device uses OpenAI's generation AI model to generate customized 3D structural data. Based on this generated data, the search device searches the database for similar items and presents them to the user. Through this series of processes, the user can visually confirm what kind of customization is possible.

[0685] The virtual vision system enables real-time display to visualize the generated three-dimensional model. As a result, an environment is created where users can easily check the quality of the model.

[0686] As a concrete example, consider a scenario in a furniture store's virtual showroom where a user visualizes a chair in a virtual space and gives instructions such as, "I want the height lowered by 5 cm and the color changed to blue." Based on this request, the server uses a generative AI model to generate a 3D model according to the instructions. The user can then review this model in the virtual environment and use it as a basis for deciding whether to order the actual product.

[0687] Example of a prompt:

[0688] "Based on the chair data, please generate a new 3D model with the height reduced by 5 centimeters and the color changed to blue."

[0689] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0690] Step 1:

[0691] The user uses smart glasses to visually identify a specific object in a virtual space and acquires three-dimensional shape data using a distance measuring device. The input is the physical information of the object, and the output is its three-dimensional shape data. The distance measuring device uses a LiDAR sensor to scan the surface of the object and generates its coordinate information as digital data.

[0692] Step 2:

[0693] The user sends customization requests to the server in natural language via a communication method. The input is a natural language instruction, and the output is the parsed request data. The server uses a natural language processing library to parse the input data and extract the specific changes (e.g., "lower the height by 5cm," "change the color to blue").

[0694] Step 3:

[0695] The server uses a generation device to generate customized 3D structural data based on the acquired 3D shape data and analyzed request data. The input includes 3D shape data and request data, and the output is customized 3D structural data. The AI ​​generation model adjusts the model, and a 3D model matching the requirements is generated in .obj format.

[0696] Step 4:

[0697] The server uses customized three-dimensional structure data to search for similar existing items within the database and presents the information to the user. The input is customized three-dimensional structure data, and the output is information about similar items. The search device executes database queries to find similar items that meet specific criteria.

[0698] Step 5:

[0699] The user uses a virtual vision device to visually verify the generated customized model. The input is the generated three-dimensional structural data, and the output is a visualized object display. The virtual vision device renders the data in real time, providing the user with visual feedback.

[0700] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0701] This invention is a system that generates customized three-dimensional objects that reflect not only the user's requests but also their emotional state. A depth measurement means built into the terminal uses LiDAR to acquire three-dimensional shape data of objects and space, and incorporates this into the shape selected by the user.

[0702] The device also provides a chat interface to receive requests in natural language through interaction with the user. In addition, it incorporates an emotion engine that analyzes the user's emotions. The emotion engine uses facial recognition and voice tone analysis to identify the user's emotions and extract emotional information such as "happiness" or "calmness." This makes it possible to take into account design elements that correspond to the user's mental state.

[0703] The server has a generation mechanism for generating 3D model files based on 3D shape data and natural language requests. Using machine learning algorithms, it creates 3D models that reflect the user's requests and emotions. Emotional information can influence subtle aspects of the design; for example, positive emotions might be associated with brighter colors and more curvilinear designs.

[0704] When a user requests a special product design based on a specific theme, they first scan the target area with LiDAR on their device, and then enter specific requests, such as "I want some small items with a cheerful atmosphere," via a chat interface. When the emotion engine detects joy from the user's facial expressions and voice, that emotion information is also transmitted to the server.

[0705] The server processes the received data to generate a new 3D model, and in the process, it also searches for similar products. This allows the user to review the newly generated custom model and select existing similar products that match their preferences.

[0706] As described above, the present invention allows users to easily obtain emotionally tailored customized products without requiring any technical knowledge.

[0707] The following describes the processing flow.

[0708] Step 1:

[0709] The user selects an object to scan using the device and activates the LiDAR sensor. The device uses depth measurement to acquire three-dimensional shape data of the object and temporarily stores that data.

[0710] Step 2:

[0711] The device receives requests from users in natural language via a chat interface. Users can input specific requests such as "bright design" or "curved shape."

[0712] Step 3:

[0713] The device activates an emotion engine and uses its camera and microphone to recognize the user's emotions. Through facial recognition and voice analysis, it identifies the user's emotional state and acquires that information.

[0714] Step 4:

[0715] The terminal transmits acquired 3D shape data, natural language requests, and sentiment information to the server. This data is securely transmitted through a secure communication protocol.

[0716] Step 5:

[0717] The server analyzes the received natural language request and converts it into technical instructions. Using machine learning algorithms, it generates a 3D model file that combines the analysis results with sentiment information.

[0718] Step 6:

[0719] The server uses the generated 3D model to search the database for similar products. It extracts information on existing products with a high degree of similarity and prepares the data to present to the user.

[0720] Step 7:

[0721] The terminal displays the generated 3D model and similar product information received from the server to the user. The user can view the model in 3D preview and, if necessary, also view details of similar products.

[0722] Step 8:

[0723] Users can select a generated 3D model and either print it using a 3D printer or save it digitally. They can also purchase similar products suggested, expanding their product choices.

[0724] (Example 2)

[0725] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0726] Conventional 3D model generation systems have the problem of being unable to customize models to take into account the emotional state of the user, and therefore being unable to generate models that meet the emotions and needs of individual users. Furthermore, there was a lack of a simple way to understand the relationship between the generated 3D models and existing products in the market.

[0727] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0728] In this invention, the server includes a measurement means for acquiring three-dimensional shape information, a dialogue means for receiving user requests in natural language, a model generation means for automatically generating a three-dimensional model based on the three-dimensional shape information and the natural language requests, a search means for searching for and presenting existing items similar to the generated three-dimensional model, and an emotion analysis means for analyzing the user's emotional state and reflecting the emotional information in the three-dimensional model. This makes it possible to generate a customized model that matches the user's emotions and requests, and to present similar existing items.

[0729] "Three-dimensional shape information" refers to data that represents and records the physical shape of an object or space in three dimensions.

[0730] "Measurement means" refers to devices and methods used to acquire the three-dimensional shape of an object, utilizing technologies such as light detection and distance measurement.

[0731] A "user" refers to a person who operates this system and has specific requests or objectives.

[0732] "Natural language" refers to the linguistic forms that people use on a daily basis, and the language that users use to communicate requests to a system.

[0733] A "dialogue tool" is an interface for communicating with users and receiving requests in natural language.

[0734] A "3D model" is a digital representation generated based on data of a three-dimensional shape; it is a computer model that mimics the physical shape.

[0735] "Model generation means" refers to devices or methods used to automatically generate three-dimensional models based on input data, and includes machine learning algorithms.

[0736] A "search tool" refers to a device or method for finding existing items similar to the generated three-dimensional model in a database and presenting them to the user.

[0737] "Emotional state" refers to information that indicates the user's current emotional situation, and refers to the psychological state extracted from facial expressions and tone of voice.

[0738] "Emotional analysis means" refers to devices or methods for analyzing and identifying a user's emotions, and utilizes facial recognition and voice analysis technologies.

[0739] This invention relates to a system that generates customized 3D models that reflect the user's requests and emotional state. The system implementation mainly consists of terminals and servers.

[0740] The terminal incorporates sensors that utilize optical detection and ranging technologies as measurement means to acquire three-dimensional shape information. This measurement means efficiently acquires three-dimensional shape data from the user's environment and saves it as point cloud data. The user can input their requests in natural language through a chat interface, which serves as a means of interaction. In this case, specific requests such as "I want some small items that give off a bright atmosphere" are entered as an example of prompt text.

[0741] The device also includes emotion analysis capabilities to analyze the user's emotional state. It uses a camera and microphone to capture facial expressions and voice tone, thereby identifying the user's emotions. For example, feelings of joy and happiness can be detected, and this emotional information is reflected in the model's design.

[0742] The server receives three-dimensional shape data, natural language requests, and sentiment information transmitted from the terminal. It then automatically generates a 3D model based on a generative AI model using machine learning as the generation method. In this process, for example, if positive sentiment information is received, bright colors and curvilinear design elements are incorporated into the model.

[0743] Furthermore, the server searches its database for existing items similar to the generated 3D model and presents them to the user. This allows the user to compare the newly generated custom model with similar commercially available products and make a suitable selection.

[0744] In this way, the system allows users to obtain appropriate and emotionally tailored customized products without requiring any special technical knowledge.

[0745] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0746] Step 1:

[0747] The device activates its LiDAR sensor via user input to acquire three-dimensional shape data of surrounding objects and space. The sensor uses light detection and ranging techniques to collect detailed point cloud data of the target. This input data is stored as a basis for future 3D model generation.

[0748] Step 2:

[0749] The terminal receives user requests in natural language using a chat interface. For example, the user might type, "I want some small items with a bright atmosphere." The terminal records this natural language request as text data and prepares it for subsequent processing.

[0750] Step 3:

[0751] The device uses a camera and microphone to capture the user's facial expressions and voice, and performs emotion analysis. The emotion analysis system processes this input data and generates emotional information such as "happiness" or "calmness." This information is stored as data that influences design elements.

[0752] Step 4:

[0753] The terminal sends acquired 3D shape data, natural language requests, and sentiment information to the server as data packets. Here, accurate data transmission is ensured using communication methods. The output of this step is a complete set of user information for the server to analyze.

[0754] Step 5:

[0755] The server creates a 3D model using a generative AI model based on the received data. The machine learning algorithm designs the 3D model, reflecting user requests and emotional information. Here, bright colors and curvilinear designs are incorporated based on positive emotions. The generated 3D model is then prepared for further processing.

[0756] Step 6:

[0757] The server uses the generated 3D model to search its database for similar existing items. The search extracts information on similar products, which is also prepared for user presentation.

[0758] Step 7:

[0759] The terminal displays the 3D model and similar product information returned from the server to the user. The user can visually review the generated custom model and compare it with suggested similar products. The output of this step provides the user with information to make selections and decisions.

[0760] (Application Example 2)

[0761] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] In today's commercial spaces, it is crucial to quickly customize products based on individual customer emotions and needs. However, conventional technologies have struggled to efficiently customize products while considering customer emotions, creating a need for methods to enhance customer satisfaction. Furthermore, there has been a lack of systems capable of providing product recommendations utilizing real-time sentiment analysis.

[0763] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0764] In this invention, the server includes depth detection means for acquiring three-dimensional shape information, dialogue interface means for receiving user requests in natural language, and emotion recognition means for analyzing the user's emotional state. This makes it possible to quickly generate and present customized products that meet the customer's emotions and requests.

[0765] "Three-dimensional shape information" refers to data that indicates the three-dimensional shape and position of an object in space.

[0766] A "depth detection method" is a method of acquiring three-dimensional shape information using devices or techniques that measure the distance to an object.

[0767] "Users" refers to customers who use this system to customize products.

[0768] "Natural language" refers to the language that humans use on a daily basis, and is a means of communication that does not rely on special codes or formats.

[0769] A "dialogue interface means" is a technology that provides an interface for receiving and processing user requests in natural language.

[0770] "Emotion identification means" refers to technology that analyzes and identifies emotions from a user's facial expressions, voice, etc.

[0771] A "3D shape file" is a digital file that stores three-dimensional shape data and is used for visualization and 3D printing.

[0772] "Generation method" refers to a system that automatically creates a 3D shape file based on the input data.

[0773] A "search tool" is a system element that has the function of searching for existing items similar to the generated three-dimensional shape and presenting them to the user.

[0774] This invention is a system that generates three-dimensional objects corresponding to the user's emotions. This system utilizes depth detection means, dialogue interface means, and emotion recognition means.

[0775] Hardware and software configuration:

[0776] server:

[0777] The server processes three-dimensional shape information and generates a 3D shape file based on natural language requests and sentiment data. Machine learning algorithms are used for generation. Specifically, Amazon Lex is used for natural language understanding, and Microsoft Azure's Emotion API is used for sentiment analysis.

[0778] Device (e.g., smart glasses):

[0779] The device is equipped with a LiDAR sensor to measure the distance to objects, a camera to capture the user's facial expressions, and a microphone to recognize voice. This allows the user's requests and emotional state to be transmitted to the server in real time. Blender can be used for 3D rendering.

[0780] Specific example:

[0781] As users move through a commercial space, they can input prompts through smart glasses, such as "I want furniture with a relaxing atmosphere." The smart glasses detect the user's calm facial expression and send the emotion "relaxation" to a server. Based on this information, the server generates 3D models of custom furniture with bright colors and rounded designs. Users can then view the generated designs on the spot and select the items they want.

[0782] Example of a prompt:

[0783] "I want to give this product as a gift to a friend, so I'd like the design to convey my joy."

[0784] "I'd like the design to use warmer colors."

[0785] This invention allows users to obtain customized products that cater to their emotions at that moment, without requiring any technical knowledge.

[0786] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0787] Step 1:

[0788] The user wears smart glasses and points the camera at an object of interest. The device uses a LiDAR sensor to acquire three-dimensional shape information. This information is stored on the device as data representing the shape and position of the object in space.

[0789] Step 2:

[0790] The user inputs their request in natural language into the microphone of the smart glasses. For example, they might request "make the design brighter." The device converts the voice into text and saves the request as digital data through a dialogue interface.

[0791] Step 3:

[0792] The device captures the user's facial expressions with its camera and analyzes the user's emotional state using emotion recognition technology. The analyzed emotion data (e.g., "joy" or "calmness") is sent to the server as numerical data.

[0793] Step 4:

[0794] The server receives 3D shape information from LiDAR, request data from voice input, and emotion data. Based on this data, a generative AI model within the server generates a customized 3D shape file. Machine learning algorithms are used in this process.

[0795] Step 5:

[0796] The generated 3D shape file is sent from the server to the terminal and displayed on the smart glasses' screen. The user can view the proposed custom design through the smart glasses.

[0797] Step 6:

[0798] The server then searches for existing items similar to the generated 3D shape file and presents that information to the user. This allows the user to select not only the newly generated design but also existing products.

[0799] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0800] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0801] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0802] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0803] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0804] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0805] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0806] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0807] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0808] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0809] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0810] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0811] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0812] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0813] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0814] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0815] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0816] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0817] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0818] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0819] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0820] The following is further disclosed regarding the embodiments described above.

[0821] (Claim 1)

[0822] A depth measurement means for acquiring three-dimensional shape data,

[0823] A chat interface that receives user requests in natural language,

[0824] A generation means for automatically generating a 3D model file based on the aforementioned 3D shape data and the aforementioned natural language request,

[0825] A search method that searches for and presents existing products similar to the generated 3D model,

[0826] A system that includes this.

[0827] (Claim 2)

[0828] The system according to claim 1, wherein the depth measuring means operates using light detection and ranging techniques.

[0829] (Claim 3)

[0830] The system according to claim 1, wherein the generation means generates a three-dimensional model using a machine learning algorithm.

[0831] "Example 1"

[0832] (Claim 1)

[0833] A measurement means for acquiring three-dimensional shape data,

[0834] An interactive interface means that receives instructions from the user in natural language,

[0835] A creation means for automatically generating a three-dimensional structure file based on the three-dimensional shape data and the natural language instructions,

[0836] A search method for finding and presenting existing products similar to the generated three-dimensional structure,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1, wherein the measuring means operates using light detection and distance measurement techniques.

[0840] (Claim 3)

[0841] The system according to claim 1, wherein the creation means generates a three-dimensional structure using machine learning theory.

[0842] "Application Example 1"

[0843] (Claim 1)

[0844] A distance measuring device for acquiring three-dimensional shape data,

[0845] A communication method that receives user requests in natural language,

[0846] A generation device that automatically generates three-dimensional structural data based on the three-dimensional shape data and the natural language request,

[0847] A search device that searches for and presents existing items similar to the generated three-dimensional structural data,

[0848] A display device that presents three-dimensional structural data of an object after visualization and adjustment through a virtual vision device,

[0849] A system that includes this.

[0850] (Claim 2)

[0851] The system according to claim 1, wherein the distance measuring device operates using light detection and distance measurement techniques.

[0852] (Claim 3)

[0853] The system according to claim 1, wherein the generation device generates three-dimensional structural data using a data analysis algorithm.

[0854] "Example 2 of combining an emotion engine"

[0855] (Claim 1)

[0856] A measurement means for acquiring information on three-dimensional shape,

[0857] A dialogue method that receives requests from users in natural language,

[0858] A model generation means that automatically generates a three-dimensional model based on the three-dimensional shape information and the natural language request,

[0859] A search method that searches for and presents existing items similar to the generated 3D model,

[0860] An emotion analysis method that analyzes the user's emotional state and reflects the emotional information in a three-dimensional model,

[0861] A system that includes this.

[0862] (Claim 2)

[0863] The system according to claim 1, wherein the measurement means operates using light detection and ranging techniques.

[0864] (Claim 3)

[0865] The system according to claim 1, wherein the model generation means generates a three-dimensional model using machine learning techniques.

[0866] "Application example 2 when combining with an emotional engine"

[0867] (Claim 1)

[0868] A depth detection means for acquiring three-dimensional shape information,

[0869] A dialogue interface means that receives requests from users in natural language,

[0870] A means for identifying emotions to analyze the emotional state of a user,

[0871] A generation means for automatically generating a three-dimensional shape file based on the three-dimensional shape information, the natural language request, and the user's emotional state,

[0872] A search method for finding and presenting existing items similar to the generated three-dimensional shape,

[0873] A system that includes this.

[0874] (Claim 2)

[0875] The system according to claim 1, wherein the depth detection means operates using optical detection and distance measurement techniques.

[0876] (Claim 3)

[0877] The system according to claim 1, wherein the generation means generates a three-dimensional shape using a machine learning algorithm. [Explanation of symbols]

[0878] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A depth measurement means for acquiring three-dimensional shape data, A chat interface that receives user requests in natural language, A generation means for automatically generating a 3D model file based on the aforementioned 3D shape data and the aforementioned natural language request, A search method that searches for and presents existing products similar to the generated 3D model, A system that includes this.

2. The system according to claim 1, wherein the depth measuring means operates using light detection and ranging technology.

3. The system according to claim 1, wherein the generation means generates a three-dimensional model using a machine learning algorithm.

Images