System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring a proposal generated by generation means; means for visualizing the generated proposal using augmented reality technology; and means for evaluating the visualized proposal by a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that this invention aims to solve is to improve the current situation where it is not possible to intuitively and accurately evaluate the credibility of plans and strategy proposals generated by generative AI. The problem is that the evaluation of generated proposals often depends on the subjectivity and skill of the person in charge, making it difficult to objectively evaluate them using concrete visuals. [Means for solving the problem]

[0005] In order to solve this problem, the present invention provides the following means: A system including a means for acquiring ideas generated by a generation means, a means for visualizing the generated ideas using augmented reality technology, and a means for a user to evaluate the visualized ideas makes it possible to visually evaluate the generated ideas.

[0006] Furthermore, by including a user evaluation acquisition means for acquiring user evaluations, a modification means for modifying the proposal generated based on the user evaluations, and a means for revising the modified proposals, the system allows users to intuitively evaluate proposals and re-evaluate them to reflect the feedback.

[0007] By further providing a communication means for receiving user input from a terminal and communicating with the generation means, the user can easily generate and evaluate ideas. Furthermore, by providing a format conversion means for converting data of the generated ideas into a format usable in augmented reality technology, the generated ideas can be efficiently visualized. This provides a system that allows users to evaluate ideas through concrete visuals using a virtual reality device or an augmented reality device.

[0008] "Generation means" refers to algorithms and systems that generate plans and strategy proposals based on user input data.

[0009] "Proposal" refers to the specific content of the plan or strategy generated by the generation means.

[0010] "Augmented reality technology" is a general term for technology that displays digital information overlaid on the real world.

[0011] "Visualization means" refers to techniques and devices for visually displaying the generated proposals.

[0012] The 'user evaluation acquisition means' is a means for collecting and recording evaluations made by users on generated proposals.

[0013] "Revisers" are algorithms and systems for revising generated suggestions based on user ratings.

[0014] The "communication means" is a means for transmitting user input data from the terminal to the server and receiving generated proposal data from the server to the terminal.

[0015] The "format conversion means" refers to a technology and system for converting the generated proposal data into a format that can be used in augmented reality technology.

[0016] "Virtual or augmented reality device" refers to a device for visually displaying generated proposals to a user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[0039] System program processing overview

[0040] 1. User Input

[0041] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0042] Example: A user types, "Come up with a new office design."

[0043] 2. Sending input data

[0044] The terminal sends the user's input text to the server.

[0045] Example: The device sends the input text to the server via an HTTP POST request.

[0046] 3. Planning and strategy creation

[0047] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[0048] Example: The server generates a "spacious office space with lots of natural light."

[0049] 4. Format conversion of generated results

[0050] The server converts the generated proposals into a format that can be used with augmented reality technology (e.g., 3D model data).

[0051] Example: The server converts the generated proposal into a data format for 3D modeling.

[0052] 5. Send the conversion results

[0053] The 3D model data generated by the server is sent to the terminal.

[0054] Example: The server encodes 3D model data and sends it back to the device.

[0055] 6. Receipt and processing of data

[0056] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[0057] Example: The device loads 3D model data into the Unity engine and displays it on the VR device.

[0058] 7. User Ratings

[0059] Users visually check and evaluate the generated 3D models using a VR headset.

[0060] Example: A user puts on a VR headset and walks freely around a 3D model of an office.

[0061] 8. Enter your rating

[0062] The user enters feedback into the device's rating UI.

[0063] Example: A user enters feedback such as "I'd like a window added to the right wall."

[0064] 9. Submission of evaluation data

[0065] The terminal sends the user's feedback to the server.

[0066] Example: The device sends feedback to the server via an HTTP POST request.

[0067] 10. Processing Feedback

[0068] The server analyzes the received feedback, and the generating AI reevaluates and corrects it.

[0069] Example: The server executes the modification "add a window to the right wall" and generates the office design again.

[0070] 11. Sending the regeneration results

[0071] The server resends the corrected 3D model data to the device.

[0072] Example: The server sends the modified 3D data back to the device.

[0073] 12. Final Evaluation

[0074] Users review the new model and provide feedback and ratings until they are satisfied.

[0075] Example: Users review and rate new office designs until they are satisfied.

[0076] Specific examples

[0077] When a user types "Please propose a conference room layout" into the device's chat UI, the device sends this text data to the server. The server uses generative AI to generate a proposal for a "conference room with a large table, multiple chairs, and a large display" and converts it into 3D model data. This 3D model data is sent to the device, which displays the received data on the VR device. The user uses a VR headset to check the conference room model and provides feedback such as "I'd like the table to be moved closer to the window." The server then receives the feedback, regenerates the proposal, and sends it to the device, where the new model is displayed. The user can repeat this process until they are finally satisfied.

[0078] The present invention can provide a system that allows users to visually and intuitively evaluate generated ideas and provide highly accurate feedback.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0082] As a concrete example, the user inputs "Please suggest a design for our new office."

[0083] Step 2:

[0084] The device receives the user's input text and sends it to the server using an HTTP POST request.

[0085] The request includes the text data and any required metadata.

[0086] Step 3:

[0087] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[0088] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[0089] Step 4:

[0090] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0091] For example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[0092] Step 5:

[0093] The server sends the converted 3D model data to the terminal.

[0094] This data is encoded and sent back to the device as an HTTP response.

[0095] Step 6:

[0096] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[0097] For example, the device uses the Unity engine to load 3D model data and display it on the VR headset.

[0098] Step 7:

[0099] Users use a VR headset to view and evaluate the generated 3D model.

[0100] Users can freely observe and move around the office layout and design within the virtual space.

[0101] Step 8:

[0102] The user enters feedback into the device's rating UI.

[0103] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[0104] Step 9:

[0105] The device receives the user's feedback and sends it to the server via an HTTP POST request.

[0106] The feedback includes the user's rating and specific correction instructions.

[0107] Step 10:

[0108] The server analyzes the received feedback and uses generative AI to regenerate and revise the proposal.

[0109] For example, the server generates a new office design that reflects the modification "add a window to the right wall."

[0110] Step 11:

[0111] The server sends the modified 3D model data back to the device.

[0112] The corrected data is re-encoded and sent back to the terminal as an HTTP response.

[0113] Step 12:

[0114] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[0115] Allow users to see the new model.

[0116] Step 13:

[0117] The user evaluates the new 3D model again and provides feedback as needed.

[0118] This process is repeated until the user is satisfied.

[0119] This allows for visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] In systems using conventional generation and visualization methods, it was difficult for users to intuitively evaluate the generated proposals. Furthermore, there was a lack of a process for quickly reflecting user evaluations in feedback and regenerating the proposals. This made it difficult to provide highly accurate proposals that met the user's requirements in a short amount of time.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for receiving a prompt sentence based on a user's input and generating a proposal using a generative AI model, means for converting the generated proposal into a format usable in augmented reality technology, and means for displaying the converted proposal on an augmented reality device, thereby enabling the user to intuitively evaluate the generated proposal and quickly modify and regenerate it based on the evaluation.

[0125] "User input" refers to text data that a user inputs using a terminal through a chat UI, etc.

[0126] A "prompt" refers to an instruction or question that is passed to a generative AI model based on user input.

[0127] A "generative AI model" refers to an artificial intelligence model that receives a prompt sentence as input and generates a plan or strategy proposal in the specified format.

[0128] "Means of generation" refers to the means of using a generative AI model to execute the process of generating plans and strategy proposals based on user input.

[0129] "Augmented reality technology" refers to technology that displays digital information overlaid on the physical environment.

[0130] "Means for converting" refers to means for converting ideas generated by a generative AI model into a data format that can be used in augmented reality technology.

[0131] "Means for displaying" refers to means for displaying the converted data on an augmented reality or VR device.

[0132] The term "means for evaluation" refers to the means used by a user to visually check and evaluate the generated proposals using an augmented reality device.

[0133] The "user evaluation acquisition means" refers to a means for collecting evaluations and feedback given by users on generated proposals.

[0134] "Means for correction" refers to the process of correcting the generated proposal based on user evaluations and feedback and then generating it again.

[0135] "Means of communication" refers to the process of sending and receiving data between a terminal and a server.

[0136] This invention provides a system that intuitively evaluates plans and strategies generated based on user input and quickly modifies them based on user feedback. This system is realized by combining a generation unit, augmented reality technology, and a user evaluation acquisition unit.

[0137] First, a user inputs text data into the chat UI using a terminal. For example, the user inputs "Please come up with a design for our new office." This input is sent to the server by the terminal via an HTTP POST request.

[0138] The server analyzes the received text data and generates plans and strategy proposals using a generative AI model. Specifically, in response to the received prompt "New office design," the server uses the generative AI model to generate a proposal for "a spacious office space with plenty of natural light."

[0139] The server then converts the generated proposal into a format that can be used with augmented reality technology, converting the text data into 3D model data (e.g., OBJ format). The converted data is then sent back to the device as an HTTP response.

[0140] The device decodes the received 3D model data and inputs it into an augmented reality display platform such as the Unity engine, which then displays the 3D model on the VR device.

[0141] The user visually checks the generated 3D model using a VR headset. For example, the user puts on the VR headset and checks the generated 3D model of an office in detail. Next, the user enters feedback in an evaluation form, such as "I would like a window added to the right wall." This feedback is then sent back to the server from the device.

[0142] The server analyzes the received feedback and reevaluates and modifies it using the generative AI model. Specifically, it generates a new office design that reflects the modification instruction to "add a window to the right wall." This modified 3D model data is then sent back from the server to the device.

[0143] Users review new models and provide feedback until they are satisfied. Through this cycle, users can intuitively provide highly accurate feedback.

[0144] As a result, this system combines the use of generative AI models based on user input with augmented reality technology to enable highly accurate evaluation and revision of plans and strategic proposals.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1:

[0147] User Input

[0148] The user uses the terminal to input text data related to the plan or strategy into the chat UI.

[0149] Input: Text data entered by the user into the chat UI (e.g., "Please come up with a design for our new office.")

[0150] Specific operation: The user enters text data into the chat input field on the device and presses the send button.

[0151] Step 2:

[0152] Sending input data

[0153] The terminal sends the user's input text to the server as an HTTP POST request.

[0154] Input: The text data entered by the user in step 1

[0155] Output: HTTP POST request sent to the server

[0156] Specific operation: The terminal creates an HTTP request and sends the request including text data to the server.

[0157] Step 3:

[0158] Planning and strategy creation

[0159] The server analyzes the received text data and uses a generative AI model to generate plans and strategy proposals.

[0160] Input: Prompt sent from the terminal (e.g., "Could you come up with a design for our new office?")

[0161] Output: Generated planning and strategy proposals (e.g., "A spacious office space with plenty of natural light")

[0162] Specific operation: The server analyzes the text data and inputs it into a generative AI model to generate suggestions.

[0163] Step 4:

[0164] Format conversion of generated results

[0165] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0166] Input: Text data of plans and strategies output from the generative AI model

[0167] Output: Data format for 3D modeling (e.g. OBJ format)

[0168] Specific operation: The server uses a format conversion engine to convert the generated text data into a 3D model.

[0169] Step 5:

[0170] Sending the conversion results

[0171] The server sends the generated 3D model data to the terminal as an HTTP response.

[0172] Input: Format-converted 3D model data

[0173] Output: HTTP response sent from the server to the device

[0174] Specific operation: The server encodes the 3D model data and sends it to the terminal as an HTTP response.

[0175] Step 6:

[0176] Receiving and processing data

[0177] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[0178] Input: 3D model data sent from the server

[0179] Output: 3D model data loaded into an augmented reality viewing platform

[0180] Specific operation: The device decodes the received data and inputs it into an augmented reality display platform such as the Unity engine.

[0181] Step 7:

[0182] User Rating

[0183] Users can visually check and evaluate the generated 3D models using a VR headset.

[0184] Input: 3D model displayed on an augmented reality viewing platform

[0185] Output: User ratings and feedback

[0186] Specific actions: The user puts on a VR headset and evaluates the 3D model while examining it in detail.

[0187] Step 8:

[0188] Enter your rating

[0189] The user enters feedback into the device's rating UI.

[0190] Input: User rating and feedback (e.g., "I'd like a window added to the right wall")

[0191] Output: Feedback data entered into the terminal

[0192] Specific actions: The user enters feedback into the evaluation form on the device and presses the submit button.

[0193] Step 9:

[0194] Submitting evaluation data

[0195] The device sends the user's feedback to the server as an HTTP POST request.

[0196] Input: Feedback data entered by the user into the rating UI

[0197] Output: HTTP POST request sent to the server

[0198] Specific operation: The device creates an HTTP request containing feedback data and sends it to the server.

[0199] Step 10:

[0200] Processing Feedback

[0201] The server analyzes the received feedback and reevaluates and corrects it using a generative AI model.

[0202] Input: Feedback data sent from the device (e.g., "Please add a window to the right wall.")

[0203] Output: Revised plan / strategy proposal (e.g. "Office space with a window added to the right wall")

[0204] Specific operation: The server analyzes the feedback data and inputs it into the generative AI model to generate revised proposals.

[0205] Step 11:

[0206] Sending regeneration results

[0207] The server retransmits the modified 3D model data to the terminal as an HTTP response.

[0208] Input: 3D model data regenerated from a generative AI model

[0209] Output: HTTP response sent from the server to the device

[0210] Specific operation: The server encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[0211] Step 12:

[0212] Final evaluation

[0213] Users review the new model and provide feedback and ratings until they are satisfied.

[0214] Input: Augmented reality viewing platform displaying the modified 3D model data.

[0215] Output: User's final rating and feedback

[0216] Specific actions: Users make final checks on the new office design and continue rating and providing feedback until they are satisfied.

[0217] (Application example 1)

[0218] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0219] Conventional evaluation systems for store design plans and strategy proposals are limited to simple 2D drawings or still images, making visual evaluation in real space difficult. Furthermore, the process of quickly modifying and reevaluating designs based on user feedback is inefficient, placing a significant burden on store owners and designers. A system that can solve these problems, quickly obtain intuitive, highly accurate feedback, and enable visual evaluation in real space was needed.

[0220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0221] In this invention, the server includes means for acquiring plans generated by the generating means, means for visualizing the generated plans in real space using augmented reality technology, means for a user to evaluate the visualized plans for optimizing the store layout, and means for acquiring user feedback, thereby enabling the user to visually evaluate and modify the store layout in real space and quickly reflect the evaluation results.

[0222] The "generation means" is a means for automatically generating plans and strategy proposals based on the data input by the user.

[0223] The "means for acquiring a plan" is a means for acquiring the plan or strategy plan generated by the generation means within the system.

[0224] "Augmented reality technology" is a technology that displays digital information overlaid on real space.

[0225] The "means for visualizing in real space" refers to a means for visually displaying the generated proposal in actual physical space using augmented reality technology.

[0226] "Means for optimizing store layout" refers to means for effectively planning, arranging, and optimizing the interior and layout of a store.

[0227] "Means for users to evaluate visualized ideas" refers to the means by which users can evaluate and provide feedback on ideas displayed in augmented reality.

[0228] The "means for obtaining user feedback" refers to a means for collecting user evaluations and feedback.

[0229] "Correction methods" are methods for correcting and updating the generated proposals based on user feedback.

[0230] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[0231] The present invention provides a system that can evaluate generated store layouts and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together so that the content generated based on user input can be intuitively evaluated.

[0232] System program processing overview

[0233] 1. A user inputs text data related to the store design concept using a device (e.g., smartphone, tablet, or PC). This input is performed in the chat UI as a prompt sentence.

[0234] 2. The device sends this text data to the server via an HTTP POST request.

[0235] 3. The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals, including specific store layouts and interior designs.

[0236] 4. The server converts the generated design into 3D model data that can be used with augmented reality technology. The conversion process uses a 3D modeling tool such as the Unity engine.

[0237] 5. The server sends the 3D model data to the device, where it is encoded and returned as an HTTP response.

[0238] 6. The device decodes the 3D model data received from the server and inputs it into an augmented reality display platform, for example, using the Unity engine to display the model on a VR headset or AR device.

[0239] 7. The user visually checks the generated 3D model in the actual store using a VR headset or smart glasses and evaluates the store layout. This evaluation is performed, for example, via an interface.

[0240] 8. The user inputs evaluation feedback into the device, providing specific feedback such as "Please move the display to the left wall and add lighting in the center."

[0241] 9. The device sends the user's feedback to the server, also via an HTTP POST request.

[0242] 10. The server analyzes the received feedback and uses the generation AI to revise and generate a new proposal. The revised 3D model data is sent back to the device, and this process is repeated until the user is satisfied.

[0243] Hardware and software used

[0244] Devices: smartphones, tablets, PCs, VR headsets, smart glasses

[0245] Server: Generative AI model, data analysis software

[0246] Augmented reality display platform: Unity Engine

[0247] Specific examples

[0248] For example, consider a case where a user types "Please suggest a new cafe layout" into the chat UI on their device. This input is sent to the server, which uses generative AI to generate the following proposals:

[0249] "There will be a coffee counter in the center, lounge seating on the right, and a take-out counter on the left."

[0250] This generated proposal is converted into 3D model data and sent to the device. The user puts on a VR headset and views the 3D model. If the user provides feedback such as "Please place the lounge seats closer to the windows," the server receives this feedback and regenerates the proposal. The regenerated 3D model is then provided to the user again, allowing them to evaluate and modify the layout in a concrete and intuitive manner.

[0251] Example prompt sentence:

[0252] "Place a coffee counter in the center, lounge seating on the right, and a takeout counter on the left."

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] The user inputs text data related to the store design concept using a terminal. The user's input is displayed as a prompt in the chat UI.

[0256] Step 2:

[0257] The terminal sends this text data to the server. The data is sent via an HTTP POST request. The input is the store design text data entered by the user, and the output is the request sent to the server.

[0258] Step 3:

[0259] The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals. The input is the prompt text extracted from the HTTP request, and the output is the generated store layout proposal data. Specifically, the generative AI model analyzes the prompt text and creates an appropriate layout proposal.

[0260] Step 4:

[0261] The server converts the generated proposal into 3D model data that can be used with augmented reality technology. The input is the generated layout proposal data, and the output is the 3D model data. Specifically, the server converts the data into model data using a 3D modeling tool such as the Unity engine.

[0262] Step 5:

[0263] The server sends the 3D model data to the terminal. The input is the encoded 3D model data, and the output is the data returned as an HTTP response.

[0264] Step 6:

[0265] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform. The input is the decoded 3D model data, and the output is the model displayed on the AR / VR device. Specifically, it uses the Unity engine to load and display the 3D model on the VR headset or AR device.

[0266] Step 7:

[0267] Users use a VR headset or smart glasses to visually check the 3D model generated in the actual store and evaluate the store layout. The input is the 3D model on the AR / VR device, and the output is the user's evaluation data.

[0268] Step 8:

[0269] The user inputs evaluation feedback into the terminal. For example, specific feedback such as "Please move the display to the left wall and add a light in the center" is provided. The input is the user's specific feedback text, and the output is the feedback data.

[0270] Step 9:

[0271] The terminal sends user feedback to the server. The input is the feedback text data entered by the user, and the output is the HTTP POST request sent to the server.

[0272] Step 10:

[0273] The server analyzes the received feedback and uses the generative AI to revise and generate new layout proposals. The input is the feedback data and the initial layout proposal data, and the output is the revised new store layout proposal data. Specifically, this is the process by which the generative AI model incorporates user feedback and generates new layout proposals.

[0274] Step 11:

[0275] The server resends the modified 3D model data to the device. The input is the new encoded 3D model data, and the output is the resent HTTP response. This process is repeated until the user is satisfied with the result.

[0276] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0277] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[0278] System program processing overview

[0279] 1. User Input

[0280] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0281] As a concrete example, the user inputs "Please suggest a design for our new office."

[0282] 2. Sending input data

[0283] The device receives the user's input text and sends it to the server using an HTTP POST request.

[0284] As a specific example, the terminal sends input text to the server via an HTTP POST request.

[0285] 3. Planning and strategy creation

[0286] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[0287] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[0288] 4. Format conversion of generated results

[0289] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0290] As a specific example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[0291] 5. Send the conversion results

[0292] The server sends the converted 3D model data to the terminal.

[0293] As a specific example, the server encodes 3D model data and sends it back to the terminal.

[0294] 6. Receipt and processing of data

[0295] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[0296] As a specific example, the terminal uses the Unity engine to load 3D model data and display it on the VR device.

[0297] 7. User Ratings

[0298] Users use a VR headset to view and evaluate the generated 3D model.

[0299] Users can freely observe and move around the office layout and design within the virtual space.

[0300] 8. Acquiring Emotion Data

[0301] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[0302] As a specific example, the emotional state of the user is analyzed from their facial expressions and voice, and emotional data such as "joy" and "anxiety" is collected.

[0303] 9. Enter your rating

[0304] The user enters feedback into the device's rating UI.

[0305] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[0306] 10. Sending Rating and Emotion Data

[0307] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[0308] The feedback includes the user's rating and specific correction instructions, as well as emotional data.

[0309] 11. Feedback and Emotional Data Processing

[0310] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[0311] As a concrete example, the server generates a new office design that reflects the modification of "adding a window to the right wall" and the emotional data that confirms "joy."

[0312] 12. Sending the regeneration results

[0313] The server sends the modified 3D model data back to the device.

[0314] The modified data is encoded and sent back to the terminal as an HTTP response.

[0315] 13. Final Evaluation

[0316] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[0317] Allow users to see the new model and iterate with feedback as needed.

[0318] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[0319] The processing flow will be explained below.

[0320] Step 1:

[0321] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0322] As a concrete example, the user inputs "Please suggest a design for our new office."

[0323] Step 2:

[0324] The device receives the user's input text and sends it to the server using an HTTP POST request.

[0325] Here, the text data is properly formatted and sent to a pre-configured API endpoint.

[0326] Step 3:

[0327] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[0328] For example, the server generates a suggestion such as "a spacious office space with plenty of natural light." The generative AI inputs the received text data into a natural language processing (NLP) engine and runs an algorithm to generate the appropriate output.

[0329] Step 4:

[0330] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0331] As a specific example, a script or software is executed to convert the generated proposal into a data format for 3D modeling (e.g., an OBJ file or an FBX file).

[0332] Step 5:

[0333] The server sends the converted 3D model data to the terminal.

[0334] Here, the generated 3D model data is encoded as an HTTP response and sent back to the terminal.

[0335] Step 6:

[0336] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[0337] As a specific example, the device will use an xR development environment such as the Unity engine to read 3D model data and make arrangements to display it on the VR headset.

[0338] Step 7:

[0339] Users use a VR headset to view and evaluate the generated 3D model.

[0340] Users can freely observe and move around the office layout and design in the virtual space, where the emotion engine analyzes the user's facial expressions and voice and collects them as feedback.

[0341] Step 8:

[0342] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[0343] As a specific example, facial expressions and voice data are analyzed in real time via the user's camera and microphone, and emotional states such as "joy" and "anxiety" are recorded as digital data.

[0344] Step 9:

[0345] The user enters feedback into the device's rating UI.

[0346] As an example, a user may enter a text rating such as "Add a window to the right wall."

[0347] Step 10:

[0348] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[0349] Here, the feedback content and emotional data are linked, encoded, and transmitted.

[0350] Step 11:

[0351] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[0352] As a concrete example, the server generates a new office design that reflects the instruction "add a window to the right wall" and the sentiment data "user is satisfied," again using a natural language processing (NLP) engine and 3D modeling software.

[0353] Step 12:

[0354] The server sends the modified 3D model data back to the device.

[0355] The modified data is then re-encoded and sent to the device as an HTTP response.

[0356] Step 13:

[0357] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[0358] This allows the user to see the new model.

[0359] Step 14:

[0360] The user evaluates the new 3D model again and provides feedback as needed.

[0361] This process is repeated until the user is completely satisfied with the design, and the emotion engine supports this, enabling a better feedback loop.

[0362] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[0363] Example 2

[0364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] Existing systems for evaluating plans and strategy proposals have the problem of being unable to incorporate users' intuitive evaluations and emotional data. Furthermore, the time required to reflect and revise evaluation results in real time makes it difficult to achieve an efficient feedback cycle.

[0366] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0367] In this invention, the server includes means for acquiring the ideas generated by the generating means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotional data of the user, means for correcting the generated ideas based on the emotional data, and means for revising the corrected ideas, thereby enabling an efficient feedback cycle that incorporates the user's intuitive evaluation and emotional data.

[0368] The "generation means" is a means for generating plans and strategy proposals based on the data input by the user.

[0369] "Augmented reality technology" is a technology that overlays virtual information onto the real world.

[0370] The "visualization means" is a means for visually displaying the generated proposals.

[0371] The "means for user evaluation" is a means for the user to evaluate the visualized proposals.

[0372] The "means for acquiring user emotional data" refers to a means for analyzing the user's emotional state and collecting that data.

[0373] The "means for correcting the proposal generated based on emotion data" is a means for correcting the proposal generated based on the acquired emotion data.

[0374] A "terminal" is an electronic device that a user uses to interact.

[0375] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[0376] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[0377] First, the user uses the device to input text data related to plans and strategies into the chat UI. For example, the user might input, "Please come up with a design proposal for our new office." This prompt is sent from the device to the server, where processing begins.

[0378] The server then uses a generative AI model to analyze the received text data and generate a plan or strategy proposal. For example, based on the input data, it generates a proposal such as "a spacious office space with plenty of natural light." The proposal is then converted into a format that can be used with augmented reality technology. Specifically, it is converted into a data format for 3D modeling (e.g., OBJ or FBX files).

[0379] The converted 3D model data is sent from the server to the device, which then decodes the data and loads it into the augmented reality display platform. Specifically, the Unity engine is used to display the 3D model data on a VR device. The user then wears a VR headset and checks and evaluates the generated 3D model in a virtual space.

[0380] While the evaluation is in progress, the device uses an emotion engine to obtain the user's emotional data. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety," and collects that data. The user enters feedback into the evaluation UI, specifically rating the device by saying, "I'd like a window added to the right wall."

[0381] The device then sends the user's feedback and emotional data to a server, which analyzes the data and uses a generative AI model to regenerate and modify plans and strategies. For example, a new office design could be generated based on the modification of "adding a window to the right wall" and the emotional data of "joy."

[0382] Finally, the revised 3D model data is sent back to the device, which then decodes it again and loads it into the augmented reality display platform. The user then checks the new model and, if necessary, the feedback cycle is repeated. This process allows the user to visually evaluate the generated plans and strategies, and makes accurate decisions based on intuitive feedback based on emotional data.

[0383] As a specific example, this system uses hardware and software such as generative AI models, augmented reality technology, emotion engines, evaluation UIs, and the Unity engine to efficiently generate, evaluate, and revise plans and strategies.

[0384] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0385] Step 1:

[0386] The user enters a prompt statement

[0387] The user uses a terminal to input text data related to plans and strategies into the chat UI. Specifically, the user inputs a prompt such as, "Please come up with a design proposal for our new office." This input data is then sent to the server in the next step.

[0388] Input: Text data "Please propose a design for our new office"

[0389] Output: User input text data

[0390] Step 2:

[0391] The device sends the input data to the server

[0392] The device sends the user's input text to the server as an HTTP POST request. Specifically, the device formats the input data in JSON format and issues a POST request to the API endpoint.

[0393] Input: User-entered text data

[0394] Output: HTTP POST request to send to the server

[0395] Step 3:

[0396] The server generates plans and strategies

[0397] The server analyzes the input data it receives and generates plans and strategy proposals using a generative AI model. For example, based on the input data "Please propose a design for a new office," the server might generate a proposal for a "spacious office space with plenty of natural light."

[0398] Input: User-supplied text data extracted from the HTTP POST request

[0399] Output: Plans and strategies generated by the generative AI model

[0400] Step 4:

[0401] The server converts the generated results

[0402] The server converts the generated proposals into a format that can be used with augmented reality technology. Specifically, it uses a library to convert the generated proposals into data formats for 3D modeling (e.g., OBJ files or FBX files).

[0403] Input: Proposals generated by a generative AI model

[0404] Output: 3D model data converted into a format usable by augmented reality technology

[0405] Step 5:

[0406] The server sends the conversion result to the device.

[0407] The server encodes the converted 3D model data and sends it to the terminal as an HTTP response. Specifically, it encodes the 3D data into an appropriate format and returns it as an API response.

[0408] Input: 3D model data converted into a format usable by augmented reality technology

[0409] Output: 3D model data encoded as an HTTP response

[0410] Step 6:

[0411] The device receives and processes the data

[0412] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform. Specifically, the 3D model data is displayed on the VR device using the Unity engine.

[0413] Input: 3D model data encoded as an HTTP response

[0414] Output: 3D model loaded into Unity engine

[0415] Step 7:

[0416] The user evaluates the generated 3D model

[0417] The user wears a VR headset and checks and evaluates the generated 3D model in a virtual space. Specifically, the user can move freely within the virtual space and observe the layout and design of the office.

[0418] Input: 3D model loaded into the Unity engine

[0419] Output: User ratings and experience information

[0420] Step 8:

[0421] The device acquires the user's emotional data.

[0422] The device uses an emotion engine to collect data on the user's emotions. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety."

[0423] Input: User ratings and experience information

[0424] Output: Emotion data analyzed by the emotion engine

[0425] Step 9:

[0426] User enters rating feedback

[0427] The user enters feedback into the device's evaluation UI. For example, the user might say, "I'd like a window added to the right wall."

[0428] Input: User ratings and feedback

[0429] Output: Feedback content

[0430] Step 10:

[0431] The device sends the evaluation data and emotion data to the server.

[0432] The device sends the user's feedback and emotion data as an HTTP POST request to the server. Specifically, the feedback content and emotion data are compiled in JSON format and sent to the API endpoint.

[0433] Input: Feedback and emotion data

[0434] Output: HTTP POST request to send to the server

[0435] Step 11:

[0436] The server processes the feedback and emotion data

[0437] The server analyzes the received feedback and emotion data and uses a generative AI model to regenerate and modify the proposal. For example, a new office design is generated based on the modification of "adding a window to the right wall" and the emotion data of "joy."

[0438] Input: Feedback content and sentiment data extracted from HTTP POST requests

[0439] Output: Revised plans and strategies

[0440] Step 12:

[0441] The server sends the regeneration result to the terminal.

[0442] The server re-encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[0443] Input: Revised plans and strategies

[0444] Output: Modified 3D model data encoded as an HTTP response

[0445] Step 13:

[0446] The device processes the re-received 3D model data

[0447] The device then decodes the re-received 3D model data and loads it back into the augmented reality display platform. The user can then review the new model and provide feedback as needed. This allows for visual evaluation of the plans and strategies generated, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[0448] Input: Modified 3D model data encoded as an HTTP response

[0449] Output: Modified 3D model reloaded into Unity engine

[0450] (Application example 2)

[0451] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0452] Conventional evaluation systems for planning and strategy proposals obtain feedback without taking user emotions into account, which can lead to reduced accuracy of revisions and reduced user satisfaction. Furthermore, there is a lack of systems that provide intuitive evaluation methods using AR technology. Therefore, there is a need for the development of a highly accurate and user-friendly evaluation system.

[0453] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ideas generated by a generation means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotion data, and means for analyzing the emotion data. This enables highly accurate evaluation and correction based on the user's evaluation and emotion data.

[0454] "Generation means" refers to a device or software that generates a plan or strategy proposal based on user input.

[0455] "Augmented reality technology" is a technology that overlays virtual information onto real visual information.

[0456] "Evaluation means" refers to an interface or tool that allows a user to review and evaluate the generated proposals.

[0457] "Emotion data" refers to information about the user's emotional state obtained from facial expressions, voice, body movements, etc.

[0458] The "modification means" refers to a device or software that has the function of modifying the proposal generated based on the user's evaluation and emotion data.

[0459] "Regeneration" is the process of regenerating something new based on feedback or additional information.

[0460] "Communication means" refers to the communication protocol or technology used for exchanging data between the user's terminal and the generation means.

[0461] The present invention provides a system that generates plans and strategies based on user input, visualizes the plans using augmented reality technology, and allows the user to evaluate them to propose more accurate revisions. In this system, a generation unit, an augmented reality unit, an evaluation unit, an emotion data acquisition unit, and a communication unit work in cooperation with each other.

[0462] 1. User Input

[0463] Users input design requests using a chat UI on their smartphone, for example, by entering specific instructions such as, "Please propose a layout for a new store. Make the entrance more open and create a spacious area around the cash register."

[0464] 2. Sending input data

[0465] The terminal sends the user's input text to the server using an HTTP POST request, and this data is obtained by the generating means.

[0466] 3. Use of generative AI models

[0467] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the received text data and generate a proposed store layout plan, which is used as the initial planning proposal.

[0468] 4. Use of Augmented Reality Technology

[0469] The generated proposal is converted into a 3D modeling data format (e.g., OBJ or FBX file) on the server side, and then this data is sent back to the device, where it is displayed in AR using the Unity engine.

[0470] 5. User Ratings

[0471] Users can use their smartphone's camera and microphone to view the displayed design proposals, and their evaluations and emotional data are collected. For example, users can provide feedback such as, "Please make the entrance door double-doors. Please add LED lights to the cash register."

[0472] 6. Emotion Data Analysis and Correction Suggestion Generation

[0473] The emotional data and feedback acquired by the device are sent to the server and analyzed by the emotional data acquisition means and evaluation means. The generative AI model generates revision suggestions based on this data. The revision suggestions are also visualized using AR technology, and the user evaluates them again.

[0474] 7. Final Evaluation and Regeneration

[0475] The user again provides their rating and feedback, and the server receives the data and generates the final revision proposal, thereby obtaining the optimal proposal based on the user's rating and sentiment data.

[0476] Specific examples

[0477] User input prompt:

[0478] "Please suggest a new store layout, with a more open entrance and more space around the registers."

[0479] Feedback example:

[0480] "Please make the entrance door double-doored. Please add LED lights to the cash register."

[0481] In this way, a highly accurate evaluation system that reflects user feedback and emotional data can be realized.

[0482] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0483] Step 1:

[0484] The user enters a design request using the chat UI on their smartphone and presses the send button. For example, they might enter, "Please propose a layout for our new store. Make the entrance more open and create a spacious area around the cash register." The entered data is sent from the device to the server as an HTTP POST request.

[0485] Step 2:

[0486] The server receives the HTTP POST request and analyzes the received text data using the generation method. The analyzed data is input into a generative AI model (e.g., OpenAI GPT-4), which generates an initial proposal for the store layout. The generated result is then formatted as a 3D model.

[0487] Step 3:

[0488] The server encodes the converted 3D model data (e.g., OBJ or FBX files) and sends it to the device. The device decodes the 3D model data received from the server and displays it in AR using the Unity engine.

[0489] Step 4:

[0490] The user uses the smartphone camera and screen to view the generated AR store layout, and the system captures user feedback and emotional data obtained from facial expressions and voice.

[0491] Step 5:

[0492] The device sends the text feedback entered by the user and the acquired emotion data to the server as an HTTP POST request.

[0493] Step 6:

[0494] The server analyzes the feedback and emotional data received from the user. The emotional data acquisition means analyzes the user's emotional state (e.g., joy or anxiety) and generates correction suggestions using the generative AI model.

[0495] Step 7:

[0496] The server re-encodes the modified 3D model data and sends it to the device, which decodes it again and uses the Unity engine to display the proposed modifications in AR.

[0497] Step 8:

[0498] The user then evaluates the proposed AR revisions and provides final feedback. The captured emotion data is also collected on the device.

[0499] Step 9:

[0500] The device sends the user's final feedback and emotion data to the server, which then reanalyzes the data to generate a final design, which then encodes the 3D model data of the final design and sends it to the device.

[0501] Step 10:

[0502] The device then re-decodes the received 3D model data and uses the Unity engine to display the final design in AR, allowing the user to review the design and make further evaluations and revisions as needed.

[0503] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0505] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0506] [Second embodiment]

[0507] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0508] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0509] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0510] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0511] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0512] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0513] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0514] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0517] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0518] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0519] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[0520] System program processing overview

[0521] 1. User Input

[0522] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0523] Example: A user types, "Come up with a new office design."

[0524] 2. Sending input data

[0525] The terminal sends the user's input text to the server.

[0526] Example: The device sends the input text to the server via an HTTP POST request.

[0527] 3. Planning and strategy creation

[0528] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[0529] Example: The server generates a "spacious office space with lots of natural light."

[0530] 4. Format conversion of generated results

[0531] The server converts the generated proposals into a format that can be used with augmented reality technology (e.g., 3D model data).

[0532] Example: The server converts the generated proposal into a data format for 3D modeling.

[0533] 5. Send the conversion results

[0534] The 3D model data generated by the server is sent to the terminal.

[0535] Example: The server encodes 3D model data and sends it back to the device.

[0536] 6. Receipt and processing of data

[0537] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[0538] Example: The device loads 3D model data into the Unity engine and displays it on the VR device.

[0539] 7. User Ratings

[0540] Users visually check and evaluate the generated 3D models using a VR headset.

[0541] Example: A user puts on a VR headset and walks freely around a 3D model of an office.

[0542] 8. Enter your rating

[0543] The user enters feedback into the device's rating UI.

[0544] Example: A user enters feedback such as "I'd like a window added to the right wall."

[0545] 9. Submission of evaluation data

[0546] The terminal sends the user's feedback to the server.

[0547] Example: The device sends feedback to the server via an HTTP POST request.

[0548] 10. Processing Feedback

[0549] The server analyzes the received feedback, and the generating AI reevaluates and corrects it.

[0550] Example: The server executes the modification "add a window to the right wall" and generates the office design again.

[0551] 11. Sending the regeneration results

[0552] The server resends the corrected 3D model data to the device.

[0553] Example: The server sends the modified 3D data back to the device.

[0554] 12. Final Evaluation

[0555] Users review the new model and provide feedback and ratings until they are satisfied.

[0556] Example: Users review and rate new office designs until they are satisfied.

[0557] Specific examples

[0558] When a user types "Please propose a conference room layout" into the device's chat UI, the device sends this text data to the server. The server uses generative AI to generate a proposal for a "conference room with a large table, multiple chairs, and a large display" and converts it into 3D model data. This 3D model data is sent to the device, which displays the received data on the VR device. The user uses a VR headset to check the conference room model and provides feedback such as "I'd like the table to be moved closer to the window." The server then receives the feedback, regenerates the proposal, and sends it to the device, where the new model is displayed. The user can repeat this process until they are finally satisfied.

[0559] The present invention can provide a system that allows users to visually and intuitively evaluate generated ideas and provide highly accurate feedback.

[0560] The processing flow will be explained below.

[0561] Step 1:

[0562] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0563] As a concrete example, the user inputs "Please suggest a design for our new office."

[0564] Step 2:

[0565] The device receives the user's input text and sends it to the server using an HTTP POST request.

[0566] The request includes the text data and any required metadata.

[0567] Step 3:

[0568] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[0569] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[0570] Step 4:

[0571] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0572] For example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[0573] Step 5:

[0574] The server sends the converted 3D model data to the terminal.

[0575] This data is encoded and sent back to the device as an HTTP response.

[0576] Step 6:

[0577] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[0578] For example, the device uses the Unity engine to load 3D model data and display it on the VR headset.

[0579] Step 7:

[0580] Users use a VR headset to view and evaluate the generated 3D model.

[0581] Users can freely observe and move around the office layout and design within the virtual space.

[0582] Step 8:

[0583] The user enters feedback into the device's rating UI.

[0584] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[0585] Step 9:

[0586] The device receives the user's feedback and sends it to the server via an HTTP POST request.

[0587] The feedback includes the user's rating and specific correction instructions.

[0588] Step 10:

[0589] The server analyzes the received feedback and uses generative AI to regenerate and revise the proposal.

[0590] For example, the server generates a new office design that reflects the modification "add a window to the right wall."

[0591] Step 11:

[0592] The server sends the modified 3D model data back to the device.

[0593] The corrected data is re-encoded and sent back to the terminal as an HTTP response.

[0594] Step 12:

[0595] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[0596] Allow users to see the new model.

[0597] Step 13:

[0598] The user evaluates the new 3D model again and provides feedback as needed.

[0599] This process is repeated until the user is satisfied.

[0600] This allows for visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback.

[0601] Example 1

[0602] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0603] In systems using conventional generation and visualization methods, it was difficult for users to intuitively evaluate the generated proposals. Furthermore, there was a lack of a process for quickly reflecting user evaluations in feedback and regenerating the proposals. This made it difficult to provide highly accurate proposals that met the user's requirements in a short amount of time.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0605] In this invention, the server includes means for receiving a prompt sentence based on a user's input and generating a proposal using a generative AI model, means for converting the generated proposal into a format usable in augmented reality technology, and means for displaying the converted proposal on an augmented reality device, thereby enabling the user to intuitively evaluate the generated proposal and quickly modify and regenerate it based on the evaluation.

[0606] "User input" refers to text data that a user inputs using a terminal through a chat UI, etc.

[0607] A "prompt" refers to an instruction or question that is passed to a generative AI model based on user input.

[0608] A "generative AI model" refers to an artificial intelligence model that receives a prompt sentence as input and generates a plan or strategy proposal in the specified format.

[0609] "Means of generation" refers to the means of using a generative AI model to execute the process of generating plans and strategy proposals based on user input.

[0610] "Augmented reality technology" refers to technology that displays digital information overlaid on the physical environment.

[0611] "Means for converting" refers to means for converting ideas generated by a generative AI model into a data format that can be used in augmented reality technology.

[0612] "Means for displaying" refers to means for displaying the converted data on an augmented reality or VR device.

[0613] The term "means for evaluation" refers to the means used by a user to visually check and evaluate the generated proposals using an augmented reality device.

[0614] The "user evaluation acquisition means" refers to a means for collecting evaluations and feedback given by users on generated proposals.

[0615] "Means for correction" refers to the process of correcting the generated proposal based on user evaluations and feedback and then generating it again.

[0616] "Means of communication" refers to the process of sending and receiving data between a terminal and a server.

[0617] This invention provides a system that intuitively evaluates plans and strategies generated based on user input and quickly modifies them based on user feedback. This system is realized by combining a generation unit, augmented reality technology, and a user evaluation acquisition unit.

[0618] First, a user inputs text data into the chat UI using a terminal. For example, the user inputs "Please come up with a design for our new office." This input is sent to the server by the terminal via an HTTP POST request.

[0619] The server analyzes the received text data and generates plans and strategy proposals using a generative AI model. Specifically, in response to the received prompt "New office design," the server uses the generative AI model to generate a proposal for "a spacious office space with plenty of natural light."

[0620] The server then converts the generated proposal into a format that can be used with augmented reality technology, converting the text data into 3D model data (e.g., OBJ format). The converted data is then sent back to the device as an HTTP response.

[0621] The device decodes the received 3D model data and inputs it into an augmented reality display platform such as the Unity engine, which then displays the 3D model on the VR device.

[0622] The user visually checks the generated 3D model using a VR headset. For example, the user puts on the VR headset and checks the generated 3D model of an office in detail. Next, the user enters feedback in an evaluation form, such as "I would like a window added to the right wall." This feedback is then sent back to the server from the device.

[0623] The server analyzes the received feedback and reevaluates and modifies it using the generative AI model. Specifically, it generates a new office design that reflects the modification instruction to "add a window to the right wall." This modified 3D model data is then sent back from the server to the device.

[0624] Users review new models and provide feedback until they are satisfied. Through this cycle, users can intuitively provide highly accurate feedback.

[0625] As a result, this system combines the use of generative AI models based on user input with augmented reality technology to enable highly accurate evaluation and revision of plans and strategic proposals.

[0626] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0627] Step 1:

[0628] User Input

[0629] The user uses the terminal to input text data related to the plan or strategy into the chat UI.

[0630] Input: Text data entered by the user into the chat UI (e.g., "Please come up with a design for our new office.")

[0631] Specific operation: The user enters text data into the chat input field on the device and presses the send button.

[0632] Step 2:

[0633] Sending input data

[0634] The terminal sends the user's input text to the server as an HTTP POST request.

[0635] Input: The text data entered by the user in step 1

[0636] Output: HTTP POST request sent to the server

[0637] Specific operation: The terminal creates an HTTP request and sends the request including text data to the server.

[0638] Step 3:

[0639] Planning and strategy creation

[0640] The server analyzes the received text data and uses a generative AI model to generate plans and strategy proposals.

[0641] Input: Prompt sent from the terminal (e.g., "Could you come up with a design for our new office?")

[0642] Output: Generated planning and strategy proposals (e.g., "A spacious office space with plenty of natural light")

[0643] Specific operation: The server analyzes the text data and inputs it into a generative AI model to generate suggestions.

[0644] Step 4:

[0645] Format conversion of generated results

[0646] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0647] Input: Text data of plans and strategies output from the generative AI model

[0648] Output: Data format for 3D modeling (e.g. OBJ format)

[0649] Specific operation: The server uses a format conversion engine to convert the generated text data into a 3D model.

[0650] Step 5:

[0651] Sending the conversion results

[0652] The server sends the generated 3D model data to the terminal as an HTTP response.

[0653] Input: Format-converted 3D model data

[0654] Output: HTTP response sent from the server to the device

[0655] Specific operation: The server encodes the 3D model data and sends it to the terminal as an HTTP response.

[0656] Step 6:

[0657] Receiving and processing data

[0658] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[0659] Input: 3D model data sent from the server

[0660] Output: 3D model data loaded into an augmented reality viewing platform

[0661] Specific operation: The device decodes the received data and inputs it into an augmented reality display platform such as the Unity engine.

[0662] Step 7:

[0663] User Rating

[0664] Users can visually check and evaluate the generated 3D models using a VR headset.

[0665] Input: 3D model displayed on an augmented reality viewing platform

[0666] Output: User ratings and feedback

[0667] Specific actions: The user puts on a VR headset and evaluates the 3D model while examining it in detail.

[0668] Step 8:

[0669] Enter your rating

[0670] The user enters feedback into the device's rating UI.

[0671] Input: User rating and feedback (e.g., "I'd like a window added to the right wall")

[0672] Output: Feedback data entered into the terminal

[0673] Specific actions: The user enters feedback into the evaluation form on the device and presses the submit button.

[0674] Step 9:

[0675] Submitting evaluation data

[0676] The device sends the user's feedback to the server as an HTTP POST request.

[0677] Input: Feedback data entered by the user into the rating UI

[0678] Output: HTTP POST request sent to the server

[0679] Specific operation: The device creates an HTTP request containing feedback data and sends it to the server.

[0680] Step 10:

[0681] Processing Feedback

[0682] The server analyzes the received feedback and reevaluates and corrects it using a generative AI model.

[0683] Input: Feedback data sent from the device (e.g., "Please add a window to the right wall.")

[0684] Output: Revised plan / strategy proposal (e.g. "Office space with a window added to the right wall")

[0685] Specific operation: The server analyzes the feedback data and inputs it into the generative AI model to generate revised proposals.

[0686] Step 11:

[0687] Sending regeneration results

[0688] The server retransmits the modified 3D model data to the terminal as an HTTP response.

[0689] Input: 3D model data regenerated from a generative AI model

[0690] Output: HTTP response sent from the server to the device

[0691] Specific operation: The server encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[0692] Step 12:

[0693] Final evaluation

[0694] Users review the new model and provide feedback and ratings until they are satisfied.

[0695] Input: Augmented reality viewing platform displaying the modified 3D model data.

[0696] Output: User's final rating and feedback

[0697] Specific actions: Users make final checks on the new office design and continue rating and providing feedback until they are satisfied.

[0698] (Application example 1)

[0699] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0700] Conventional evaluation systems for store design plans and strategy proposals are limited to simple 2D drawings or still images, making visual evaluation in real space difficult. Furthermore, the process of quickly modifying and reevaluating designs based on user feedback is inefficient, placing a significant burden on store owners and designers. A system that can solve these problems, quickly obtain intuitive, highly accurate feedback, and enable visual evaluation in real space was needed.

[0701] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0702] In this invention, the server includes means for acquiring plans generated by the generating means, means for visualizing the generated plans in real space using augmented reality technology, means for a user to evaluate the visualized plans for optimizing the store layout, and means for acquiring user feedback, thereby enabling the user to visually evaluate and modify the store layout in real space and quickly reflect the evaluation results.

[0703] The "generation means" is a means for automatically generating plans and strategy proposals based on the data input by the user.

[0704] The "means for acquiring a plan" is a means for acquiring the plan or strategy plan generated by the generation means within the system.

[0705] "Augmented reality technology" is a technology that displays digital information overlaid on real space.

[0706] The "means for visualizing in real space" refers to a means for visually displaying the generated proposal in actual physical space using augmented reality technology.

[0707] "Means for optimizing store layout" refers to means for effectively planning, arranging, and optimizing the interior and layout of a store.

[0708] "Means for users to evaluate visualized ideas" refers to the means by which users can evaluate and provide feedback on ideas displayed in augmented reality.

[0709] The "means for obtaining user feedback" refers to a means for collecting user evaluations and feedback.

[0710] "Correction methods" are methods for correcting and updating the generated proposals based on user feedback.

[0711] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[0712] The present invention provides a system that can evaluate generated store layouts and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together so that the content generated based on user input can be intuitively evaluated.

[0713] System program processing overview

[0714] 1. A user inputs text data related to the store design concept using a device (e.g., smartphone, tablet, or PC). This input is performed in the chat UI as a prompt sentence.

[0715] 2. The device sends this text data to the server via an HTTP POST request.

[0716] 3. The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals, including specific store layouts and interior designs.

[0717] 4. The server converts the generated design into 3D model data that can be used with augmented reality technology. The conversion process uses a 3D modeling tool such as the Unity engine.

[0718] 5. The server sends the 3D model data to the device, where it is encoded and returned as an HTTP response.

[0719] 6. The device decodes the 3D model data received from the server and inputs it into an augmented reality display platform, for example, using the Unity engine to display the model on a VR headset or AR device.

[0720] 7. The user visually checks the generated 3D model in the actual store using a VR headset or smart glasses and evaluates the store layout. This evaluation is performed, for example, via an interface.

[0721] 8. The user inputs evaluation feedback into the device, providing specific feedback such as "Please move the display to the left wall and add lighting in the center."

[0722] 9. The device sends the user's feedback to the server, also via an HTTP POST request.

[0723] 10. The server analyzes the received feedback and uses the generation AI to revise and generate a new proposal. The revised 3D model data is sent back to the device, and this process is repeated until the user is satisfied.

[0724] Hardware and software used

[0725] Devices: smartphones, tablets, PCs, VR headsets, smart glasses

[0726] Server: Generative AI model, data analysis software

[0727] Augmented reality display platform: Unity Engine

[0728] Specific examples

[0729] For example, consider a case where a user types "Please suggest a new cafe layout" into the chat UI on their device. This input is sent to the server, which uses generative AI to generate the following proposals:

[0730] "There will be a coffee counter in the center, lounge seating on the right, and a take-out counter on the left."

[0731] This generated proposal is converted into 3D model data and sent to the device. The user puts on a VR headset and views the 3D model. If the user provides feedback such as "Please place the lounge seats closer to the windows," the server receives this feedback and regenerates the proposal. The regenerated 3D model is then provided to the user again, allowing them to evaluate and modify the layout in a concrete and intuitive manner.

[0732] Example prompt sentence:

[0733] "Place a coffee counter in the center, lounge seating on the right, and a takeout counter on the left."

[0734] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0735] Step 1:

[0736] The user inputs text data related to the store design concept using a terminal. The user's input is displayed as a prompt in the chat UI.

[0737] Step 2:

[0738] The terminal sends this text data to the server. The data is sent via an HTTP POST request. The input is the store design text data entered by the user, and the output is the request sent to the server.

[0739] Step 3:

[0740] The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals. The input is the prompt text extracted from the HTTP request, and the output is the generated store layout proposal data. Specifically, the generative AI model analyzes the prompt text and creates an appropriate layout proposal.

[0741] Step 4:

[0742] The server converts the generated proposal into 3D model data that can be used with augmented reality technology. The input is the generated layout proposal data, and the output is the 3D model data. Specifically, the server converts the data into model data using a 3D modeling tool such as the Unity engine.

[0743] Step 5:

[0744] The server sends the 3D model data to the terminal. The input is the encoded 3D model data, and the output is the data returned as an HTTP response.

[0745] Step 6:

[0746] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform. The input is the decoded 3D model data, and the output is the model displayed on the AR / VR device. Specifically, it uses the Unity engine to load and display the 3D model on the VR headset or AR device.

[0747] Step 7:

[0748] Users use a VR headset or smart glasses to visually check the 3D model generated in the actual store and evaluate the store layout. The input is the 3D model on the AR / VR device, and the output is the user's evaluation data.

[0749] Step 8:

[0750] The user inputs evaluation feedback into the terminal. For example, specific feedback such as "Please move the display to the left wall and add a light in the center" is provided. The input is the user's specific feedback text, and the output is the feedback data.

[0751] Step 9:

[0752] The terminal sends user feedback to the server. The input is the feedback text data entered by the user, and the output is the HTTP POST request sent to the server.

[0753] Step 10:

[0754] The server analyzes the received feedback and uses the generative AI to revise and generate new layout proposals. The input is the feedback data and the initial layout proposal data, and the output is the revised new store layout proposal data. Specifically, this is the process by which the generative AI model incorporates user feedback and generates new layout proposals.

[0755] Step 11:

[0756] The server resends the modified 3D model data to the device. The input is the new encoded 3D model data, and the output is the resent HTTP response. This process is repeated until the user is satisfied with the result.

[0757] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0758] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[0759] System program processing overview

[0760] 1. User Input

[0761] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0762] As a concrete example, the user inputs "Please suggest a design for our new office."

[0763] 2. Sending input data

[0764] The device receives the user's input text and sends it to the server using an HTTP POST request.

[0765] As a specific example, the terminal sends input text to the server via an HTTP POST request.

[0766] 3. Planning and strategy creation

[0767] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[0768] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[0769] 4. Format conversion of generated results

[0770] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0771] As a specific example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[0772] 5. Send the conversion results

[0773] The server sends the converted 3D model data to the terminal.

[0774] As a specific example, the server encodes 3D model data and sends it back to the terminal.

[0775] 6. Receipt and processing of data

[0776] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[0777] As a specific example, the terminal uses the Unity engine to load 3D model data and display it on the VR device.

[0778] 7. User Ratings

[0779] Users use a VR headset to view and evaluate the generated 3D model.

[0780] Users can freely observe and move around the office layout and design within the virtual space.

[0781] 8. Acquiring Emotion Data

[0782] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[0783] As a specific example, the emotional state of the user is analyzed from their facial expressions and voice, and emotional data such as "joy" and "anxiety" is collected.

[0784] 9. Enter your rating

[0785] The user enters feedback into the device's rating UI.

[0786] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[0787] 10. Sending Rating and Emotion Data

[0788] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[0789] The feedback includes the user's rating and specific correction instructions, as well as emotional data.

[0790] 11. Feedback and Emotional Data Processing

[0791] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[0792] As a concrete example, the server generates a new office design that reflects the modification of "adding a window to the right wall" and the emotional data that confirms "joy."

[0793] 12. Sending the regeneration results

[0794] The server sends the modified 3D model data back to the device.

[0795] The modified data is encoded and sent back to the terminal as an HTTP response.

[0796] 13. Final Evaluation

[0797] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[0798] Allow users to see the new model and iterate with feedback as needed.

[0799] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[0800] The processing flow will be explained below.

[0801] Step 1:

[0802] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[0803] As a concrete example, the user inputs "Please suggest a design for our new office."

[0804] Step 2:

[0805] The device receives the user's input text and sends it to the server using an HTTP POST request.

[0806] Here, the text data is properly formatted and sent to a pre-configured API endpoint.

[0807] Step 3:

[0808] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[0809] For example, the server generates a suggestion such as "a spacious office space with plenty of natural light." The generative AI inputs the received text data into a natural language processing (NLP) engine and runs an algorithm to generate the appropriate output.

[0810] Step 4:

[0811] The server converts the generated proposals into a format that can be used with augmented reality technology.

[0812] As a specific example, a script or software is executed to convert the generated proposal into a data format for 3D modeling (e.g., an OBJ file or an FBX file).

[0813] Step 5:

[0814] The server sends the converted 3D model data to the terminal.

[0815] Here, the generated 3D model data is encoded as an HTTP response and sent back to the terminal.

[0816] Step 6:

[0817] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[0818] As a specific example, the device will use an xR development environment such as the Unity engine to read 3D model data and make arrangements to display it on the VR headset.

[0819] Step 7:

[0820] Users use a VR headset to view and evaluate the generated 3D model.

[0821] Users can freely observe and move around the office layout and design in the virtual space, where the emotion engine analyzes the user's facial expressions and voice and collects them as feedback.

[0822] Step 8:

[0823] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[0824] As a specific example, facial expressions and voice data are analyzed in real time via the user's camera and microphone, and emotional states such as "joy" and "anxiety" are recorded as digital data.

[0825] Step 9:

[0826] The user enters feedback into the device's rating UI.

[0827] As an example, a user may enter a text rating such as "Add a window to the right wall."

[0828] Step 10:

[0829] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[0830] Here, the feedback content and emotional data are linked, encoded, and transmitted.

[0831] Step 11:

[0832] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[0833] As a concrete example, the server generates a new office design that reflects the instruction "add a window to the right wall" and the sentiment data "user is satisfied," again using a natural language processing (NLP) engine and 3D modeling software.

[0834] Step 12:

[0835] The server sends the modified 3D model data back to the device.

[0836] The modified data is then re-encoded and sent to the device as an HTTP response.

[0837] Step 13:

[0838] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[0839] This allows the user to see the new model.

[0840] Step 14:

[0841] The user evaluates the new 3D model again and provides feedback as needed.

[0842] This process is repeated until the user is completely satisfied with the design, and the emotion engine supports this, enabling a better feedback loop.

[0843] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[0844] Example 2

[0845] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0846] Existing systems for evaluating plans and strategy proposals have the problem of being unable to incorporate users' intuitive evaluations and emotional data. Furthermore, the time required to reflect and revise evaluation results in real time makes it difficult to achieve an efficient feedback cycle.

[0847] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0848] In this invention, the server includes means for acquiring the ideas generated by the generating means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotional data of the user, means for correcting the generated ideas based on the emotional data, and means for revising the corrected ideas, thereby enabling an efficient feedback cycle that incorporates the user's intuitive evaluation and emotional data.

[0849] The "generation means" is a means for generating plans and strategy proposals based on the data input by the user.

[0850] "Augmented reality technology" is a technology that overlays virtual information onto the real world.

[0851] The "visualization means" is a means for visually displaying the generated proposals.

[0852] The "means for user evaluation" is a means for the user to evaluate the visualized proposals.

[0853] The "means for acquiring user emotional data" refers to a means for analyzing the user's emotional state and collecting that data.

[0854] The "means for correcting the proposal generated based on emotion data" is a means for correcting the proposal generated based on the acquired emotion data.

[0855] A "terminal" is an electronic device that a user uses to interact.

[0856] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[0857] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[0858] First, the user uses the device to input text data related to plans and strategies into the chat UI. For example, the user might input, "Please come up with a design proposal for our new office." This prompt is sent from the device to the server, where processing begins.

[0859] The server then uses a generative AI model to analyze the received text data and generate a plan or strategy proposal. For example, based on the input data, it generates a proposal such as "a spacious office space with plenty of natural light." The proposal is then converted into a format that can be used with augmented reality technology. Specifically, it is converted into a data format for 3D modeling (e.g., OBJ or FBX files).

[0860] The converted 3D model data is sent from the server to the device, which then decodes the data and loads it into the augmented reality display platform. Specifically, the Unity engine is used to display the 3D model data on a VR device. The user then wears a VR headset and checks and evaluates the generated 3D model in a virtual space.

[0861] While the evaluation is in progress, the device uses an emotion engine to obtain the user's emotional data. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety," and collects that data. The user enters feedback into the evaluation UI, specifically rating the device by saying, "I'd like a window added to the right wall."

[0862] The device then sends the user's feedback and emotional data to a server, which analyzes the data and uses a generative AI model to regenerate and modify plans and strategies. For example, a new office design could be generated based on the modification of "adding a window to the right wall" and the emotional data of "joy."

[0863] Finally, the revised 3D model data is sent back to the device, which then decodes it again and loads it into the augmented reality display platform. The user then checks the new model and, if necessary, the feedback cycle is repeated. This process allows the user to visually evaluate the generated plans and strategies, and makes accurate decisions based on intuitive feedback based on emotional data.

[0864] As a specific example, this system uses hardware and software such as generative AI models, augmented reality technology, emotion engines, evaluation UIs, and the Unity engine to efficiently generate, evaluate, and revise plans and strategies.

[0865] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0866] Step 1:

[0867] The user enters a prompt statement

[0868] The user uses a terminal to input text data related to plans and strategies into the chat UI. Specifically, the user inputs a prompt such as, "Please come up with a design proposal for our new office." This input data is then sent to the server in the next step.

[0869] Input: Text data "Please propose a design for our new office"

[0870] Output: User input text data

[0871] Step 2:

[0872] The device sends the input data to the server

[0873] The device sends the user's input text to the server as an HTTP POST request. Specifically, the device formats the input data in JSON format and issues a POST request to the API endpoint.

[0874] Input: User-entered text data

[0875] Output: HTTP POST request to send to the server

[0876] Step 3:

[0877] The server generates plans and strategies

[0878] The server analyzes the input data it receives and generates plans and strategy proposals using a generative AI model. For example, based on the input data "Please propose a design for a new office," the server might generate a proposal for a "spacious office space with plenty of natural light."

[0879] Input: User-supplied text data extracted from the HTTP POST request

[0880] Output: Plans and strategies generated by the generative AI model

[0881] Step 4:

[0882] The server converts the generated results

[0883] The server converts the generated proposals into a format that can be used with augmented reality technology. Specifically, it uses a library to convert the generated proposals into data formats for 3D modeling (e.g., OBJ files or FBX files).

[0884] Input: Proposals generated by a generative AI model

[0885] Output: 3D model data converted into a format usable by augmented reality technology

[0886] Step 5:

[0887] The server sends the conversion result to the device.

[0888] The server encodes the converted 3D model data and sends it to the terminal as an HTTP response. Specifically, it encodes the 3D data into an appropriate format and returns it as an API response.

[0889] Input: 3D model data converted into a format usable by augmented reality technology

[0890] Output: 3D model data encoded as an HTTP response

[0891] Step 6:

[0892] The device receives and processes the data

[0893] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform. Specifically, the 3D model data is displayed on the VR device using the Unity engine.

[0894] Input: 3D model data encoded as an HTTP response

[0895] Output: 3D model loaded into Unity engine

[0896] Step 7:

[0897] The user evaluates the generated 3D model

[0898] The user wears a VR headset and checks and evaluates the generated 3D model in a virtual space. Specifically, the user can move freely within the virtual space and observe the layout and design of the office.

[0899] Input: 3D model loaded into the Unity engine

[0900] Output: User ratings and experience information

[0901] Step 8:

[0902] The device acquires the user's emotional data.

[0903] The device uses an emotion engine to collect data on the user's emotions. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety."

[0904] Input: User ratings and experience information

[0905] Output: Emotion data analyzed by the emotion engine

[0906] Step 9:

[0907] User enters rating feedback

[0908] The user enters feedback into the device's evaluation UI. For example, the user might say, "I'd like a window added to the right wall."

[0909] Input: User ratings and feedback

[0910] Output: Feedback content

[0911] Step 10:

[0912] The device sends the evaluation data and emotion data to the server.

[0913] The device sends the user's feedback and emotion data as an HTTP POST request to the server. Specifically, the feedback content and emotion data are compiled in JSON format and sent to the API endpoint.

[0914] Input: Feedback and emotion data

[0915] Output: HTTP POST request to send to the server

[0916] Step 11:

[0917] The server processes the feedback and emotion data

[0918] The server analyzes the received feedback and emotion data and uses a generative AI model to regenerate and modify the proposal. For example, a new office design is generated based on the modification of "adding a window to the right wall" and the emotion data of "joy."

[0919] Input: Feedback content and sentiment data extracted from HTTP POST requests

[0920] Output: Revised plans and strategies

[0921] Step 12:

[0922] The server sends the regeneration result to the terminal.

[0923] The server re-encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[0924] Input: Revised plans and strategies

[0925] Output: Modified 3D model data encoded as an HTTP response

[0926] Step 13:

[0927] The device processes the re-received 3D model data

[0928] The device then decodes the re-received 3D model data and loads it back into the augmented reality display platform. The user can then review the new model and provide feedback as needed. This allows for visual evaluation of the plans and strategies generated, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[0929] Input: Modified 3D model data encoded as an HTTP response

[0930] Output: Modified 3D model reloaded into Unity engine

[0931] (Application example 2)

[0932] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0933] Conventional evaluation systems for planning and strategy proposals obtain feedback without taking user emotions into account, which can lead to reduced accuracy of revisions and reduced user satisfaction. Furthermore, there is a lack of systems that provide intuitive evaluation methods using AR technology. Therefore, there is a need for the development of a highly accurate and user-friendly evaluation system.

[0934] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ideas generated by a generation means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotion data, and means for analyzing the emotion data. This enables highly accurate evaluation and correction based on the user's evaluation and emotion data.

[0935] "Generation means" refers to a device or software that generates a plan or strategy proposal based on user input.

[0936] "Augmented reality technology" is a technology that overlays virtual information onto real visual information.

[0937] "Evaluation means" refers to an interface or tool that allows a user to review and evaluate the generated proposals.

[0938] "Emotion data" refers to information about the user's emotional state obtained from facial expressions, voice, body movements, etc.

[0939] The "modification means" refers to a device or software that has the function of modifying the proposal generated based on the user's evaluation and emotion data.

[0940] "Regeneration" is the process of regenerating something new based on feedback or additional information.

[0941] "Communication means" refers to the communication protocol or technology used for exchanging data between the user's terminal and the generation means.

[0942] The present invention provides a system that generates plans and strategies based on user input, visualizes the plans using augmented reality technology, and allows the user to evaluate them to propose more accurate revisions. In this system, a generation unit, an augmented reality unit, an evaluation unit, an emotion data acquisition unit, and a communication unit work in cooperation with each other.

[0943] 1. User Input

[0944] Users input design requests using a chat UI on their smartphone, for example, by entering specific instructions such as, "Please propose a layout for a new store. Make the entrance more open and create a spacious area around the cash register."

[0945] 2. Sending input data

[0946] The terminal sends the user's input text to the server using an HTTP POST request, and this data is obtained by the generating means.

[0947] 3. Use of generative AI models

[0948] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the received text data and generate a proposed store layout plan, which is used as the initial planning proposal.

[0949] 4. Use of Augmented Reality Technology

[0950] The generated proposal is converted into a 3D modeling data format (e.g., OBJ or FBX file) on the server side, and then this data is sent back to the device, where it is displayed in AR using the Unity engine.

[0951] 5. User Ratings

[0952] Users can use their smartphone's camera and microphone to view the displayed design proposals, and their evaluations and emotional data are collected. For example, users can provide feedback such as, "Please make the entrance door double-doors. Please add LED lights to the cash register."

[0953] 6. Emotion Data Analysis and Correction Suggestion Generation

[0954] The emotional data and feedback acquired by the device are sent to the server and analyzed by the emotional data acquisition means and evaluation means. The generative AI model generates revision suggestions based on this data. The revision suggestions are also visualized using AR technology, and the user evaluates them again.

[0955] 7. Final Evaluation and Regeneration

[0956] The user again provides their rating and feedback, and the server receives the data and generates the final revision proposal, thereby obtaining the optimal proposal based on the user's rating and sentiment data.

[0957] Specific examples

[0958] User input prompt:

[0959] "Please suggest a new store layout, with a more open entrance and more space around the registers."

[0960] Feedback example:

[0961] "Please make the entrance door double-doored. Please add LED lights to the cash register."

[0962] In this way, a highly accurate evaluation system that reflects user feedback and emotional data can be realized.

[0963] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0964] Step 1:

[0965] The user enters a design request using the chat UI on their smartphone and presses the send button. For example, they might enter, "Please propose a layout for our new store. Make the entrance more open and create a spacious area around the cash register." The entered data is sent from the device to the server as an HTTP POST request.

[0966] Step 2:

[0967] The server receives the HTTP POST request and analyzes the received text data using the generation method. The analyzed data is input into a generative AI model (e.g., OpenAI GPT-4), which generates an initial proposal for the store layout. The generated results are then formatted as a 3D model.

[0968] Step 3:

[0969] The server encodes the converted 3D model data (e.g., OBJ or FBX files) and sends it to the device. The device decodes the 3D model data received from the server and displays it in AR using the Unity engine.

[0970] Step 4:

[0971] The user uses the smartphone camera and screen to view the generated AR store layout, and the system captures user feedback and emotional data obtained from facial expressions and voice.

[0972] Step 5:

[0973] The device sends the text feedback entered by the user and the acquired emotion data to the server as an HTTP POST request.

[0974] Step 6:

[0975] The server analyzes the feedback and emotional data received from the user. The emotional data acquisition means analyzes the user's emotional state (e.g., joy or anxiety) and generates correction suggestions using the generative AI model.

[0976] Step 7:

[0977] The server re-encodes the modified 3D model data and sends it to the device, which decodes it again and uses the Unity engine to display the proposed modifications in AR.

[0978] Step 8:

[0979] The user then evaluates the proposed AR revisions and provides final feedback. The captured emotion data is also collected on the device.

[0980] Step 9:

[0981] The device sends the user's final feedback and emotion data to the server, which then reanalyzes the data to generate a final design, which then encodes the 3D model data of the final design and sends it to the device.

[0982] Step 10:

[0983] The device then re-decodes the received 3D model data and uses the Unity engine to display the final design in AR, allowing the user to review the design and make further evaluations and revisions as needed.

[0984] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0985] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0986] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0987] [Third embodiment]

[0988] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0989] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0990] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0991] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0992] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0993] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0994] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0995] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0996] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0997] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0998] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0999] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1000] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[1001] System program processing overview

[1002] 1. User Input

[1003] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1004] Example: A user types, "Come up with a new office design."

[1005] 2. Sending input data

[1006] The terminal sends the user's input text to the server.

[1007] Example: The device sends the input text to the server via an HTTP POST request.

[1008] 3. Planning and strategy creation

[1009] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[1010] Example: The server generates a "spacious office space with lots of natural light."

[1011] 4. Format conversion of generated results

[1012] The server converts the generated proposals into a format that can be used with augmented reality technology (e.g., 3D model data).

[1013] Example: The server converts the generated proposal into a data format for 3D modeling.

[1014] 5. Send the conversion results

[1015] The 3D model data generated by the server is sent to the terminal.

[1016] Example: The server encodes 3D model data and sends it back to the device.

[1017] 6. Receipt and processing of data

[1018] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[1019] Example: The device loads 3D model data into the Unity engine and displays it on the VR device.

[1020] 7. User Ratings

[1021] Users visually check and evaluate the generated 3D models using a VR headset.

[1022] Example: A user puts on a VR headset and walks freely around a 3D model of an office.

[1023] 8. Enter your rating

[1024] The user enters feedback into the device's rating UI.

[1025] Example: A user enters feedback such as "I'd like a window added to the right wall."

[1026] 9. Submission of evaluation data

[1027] The terminal sends the user's feedback to the server.

[1028] Example: The device sends feedback to the server via an HTTP POST request.

[1029] 10. Processing Feedback

[1030] The server analyzes the received feedback, and the generating AI reevaluates and corrects it.

[1031] Example: The server executes the modification "add a window to the right wall" and generates the office design again.

[1032] 11. Sending the regeneration results

[1033] The server resends the corrected 3D model data to the device.

[1034] Example: The server sends the modified 3D data back to the device.

[1035] 12. Final Evaluation

[1036] Users review the new model and provide feedback and ratings until they are satisfied.

[1037] Example: Users review and rate new office designs until they are satisfied.

[1038] Specific examples

[1039] When a user types "Please propose a conference room layout" into the device's chat UI, the device sends this text data to the server. The server uses generative AI to generate a proposal for a "conference room with a large table, multiple chairs, and a large display" and converts it into 3D model data. This 3D model data is sent to the device, which displays the received data on the VR device. The user uses a VR headset to check the conference room model and provides feedback such as "I'd like the table to be moved closer to the window." The server then receives the feedback, regenerates the proposal, and sends it to the device, where the new model is displayed. The user can repeat this process until they are finally satisfied.

[1040] The present invention can provide a system that allows users to visually and intuitively evaluate generated ideas and provide highly accurate feedback.

[1041] The processing flow will be explained below.

[1042] Step 1:

[1043] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1044] As a concrete example, the user inputs "Please suggest a design for our new office."

[1045] Step 2:

[1046] The device receives the user's input text and sends it to the server using an HTTP POST request.

[1047] The request includes the text data and any required metadata.

[1048] Step 3:

[1049] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[1050] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[1051] Step 4:

[1052] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1053] For example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[1054] Step 5:

[1055] The server sends the converted 3D model data to the terminal.

[1056] This data is encoded and sent back to the device as an HTTP response.

[1057] Step 6:

[1058] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[1059] For example, the device uses the Unity engine to load 3D model data and display it on the VR headset.

[1060] Step 7:

[1061] Users use a VR headset to view and evaluate the generated 3D model.

[1062] Users can freely observe and move around the office layout and design within the virtual space.

[1063] Step 8:

[1064] The user enters feedback into the device's rating UI.

[1065] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[1066] Step 9:

[1067] The device receives the user's feedback and sends it to the server via an HTTP POST request.

[1068] The feedback includes the user's rating and specific correction instructions.

[1069] Step 10:

[1070] The server analyzes the received feedback and uses generative AI to regenerate and revise the proposal.

[1071] For example, the server generates a new office design that reflects the modification "add a window to the right wall."

[1072] Step 11:

[1073] The server sends the modified 3D model data back to the device.

[1074] The corrected data is re-encoded and sent back to the terminal as an HTTP response.

[1075] Step 12:

[1076] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[1077] Allow users to see the new model.

[1078] Step 13:

[1079] The user evaluates the new 3D model again and provides feedback as needed.

[1080] This process is repeated until the user is satisfied.

[1081] This allows for visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback.

[1082] Example 1

[1083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1084] In systems using conventional generation and visualization methods, it was difficult for users to intuitively evaluate the generated proposals. Furthermore, there was a lack of a process for quickly reflecting user evaluations in feedback and regenerating the proposals. This made it difficult to provide highly accurate proposals that met the user's requirements in a short amount of time.

[1085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1086] In this invention, the server includes means for receiving a prompt sentence based on a user's input and generating a proposal using a generative AI model, means for converting the generated proposal into a format usable in augmented reality technology, and means for displaying the converted proposal on an augmented reality device, thereby enabling the user to intuitively evaluate the generated proposal and quickly modify and regenerate it based on the evaluation.

[1087] "User input" refers to text data that a user inputs using a terminal through a chat UI, etc.

[1088] A "prompt" refers to an instruction or question that is passed to a generative AI model based on user input.

[1089] A "generative AI model" refers to an artificial intelligence model that receives a prompt sentence as input and generates a plan or strategy proposal in the specified format.

[1090] "Means of generation" refers to the means of using a generative AI model to execute the process of generating plans and strategy proposals based on user input.

[1091] "Augmented reality technology" refers to technology that displays digital information overlaid on the physical environment.

[1092] "Means for converting" refers to means for converting ideas generated by a generative AI model into a data format that can be used in augmented reality technology.

[1093] "Means for displaying" refers to means for displaying the converted data on an augmented reality or VR device.

[1094] The term "means for evaluation" refers to the means used by a user to visually check and evaluate the generated proposals using an augmented reality device.

[1095] The "user evaluation acquisition means" refers to a means for collecting evaluations and feedback given by users on generated proposals.

[1096] "Means for correction" refers to the process of correcting the generated proposal based on user evaluations and feedback and then generating it again.

[1097] "Means of communication" refers to the process of sending and receiving data between a terminal and a server.

[1098] This invention provides a system for intuitively evaluating plans and strategies generated based on user input and quickly correcting them based on user feedback. This system is realized by combining a generation unit, augmented reality technology, and a user evaluation acquisition unit.

[1099] First, a user inputs text data into the chat UI using a terminal. For example, the user inputs "Please come up with a design for our new office." This input is sent to the server by the terminal via an HTTP POST request.

[1100] The server analyzes the received text data and generates plans and strategy proposals using a generative AI model. Specifically, in response to the received prompt, "New office design," the server uses the generative AI model to generate a proposal: "A spacious office space with plenty of natural light."

[1101] The server then converts the generated proposal into a format that can be used with augmented reality technology, converting the text data into 3D model data (e.g., OBJ format). The converted data is then sent back to the device as an HTTP response.

[1102] The device decodes the received 3D model data and inputs it into an augmented reality display platform such as the Unity engine, which then displays the 3D model on the VR device.

[1103] The user visually checks the generated 3D model using a VR headset. For example, the user puts on the VR headset and checks the generated 3D model of an office in detail. Next, the user enters feedback in an evaluation form, such as "I would like a window added to the right wall." This feedback is then sent back to the server from the device.

[1104] The server analyzes the received feedback and reevaluates and modifies it using the generative AI model. Specifically, it generates a new office design that reflects the modification instruction to "add a window to the right wall." This modified 3D model data is then sent back from the server to the device.

[1105] Users review new models and provide feedback until they are satisfied. Through this cycle, users can intuitively provide highly accurate feedback.

[1106] As a result, this system combines the use of generative AI models based on user input with augmented reality technology to enable highly accurate evaluation and revision of plans and strategic proposals.

[1107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1108] Step 1:

[1109] User Input

[1110] The user uses the terminal to input text data related to the plan or strategy into the chat UI.

[1111] Input: Text data entered by the user into the chat UI (e.g., "Please come up with a design for our new office.")

[1112] Specific operation: The user enters text data into the chat input field on the device and presses the send button.

[1113] Step 2:

[1114] Sending input data

[1115] The terminal sends the user's input text to the server as an HTTP POST request.

[1116] Input: The text data entered by the user in step 1

[1117] Output: HTTP POST request sent to the server

[1118] Specific operation: The terminal creates an HTTP request and sends the request including text data to the server.

[1119] Step 3:

[1120] Planning and strategy creation

[1121] The server analyzes the received text data and uses a generative AI model to generate plans and strategy proposals.

[1122] Input: Prompt sent from the terminal (e.g., "Could you come up with a design for our new office?")

[1123] Output: Generated planning and strategy proposals (e.g., "A spacious office space with plenty of natural light")

[1124] Specific operation: The server analyzes the text data and inputs it into a generative AI model to generate suggestions.

[1125] Step 4:

[1126] Format conversion of generated results

[1127] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1128] Input: Text data of plans and strategies output from the generative AI model

[1129] Output: Data format for 3D modeling (e.g. OBJ format)

[1130] Specific operation: The server uses a format conversion engine to convert the generated text data into a 3D model.

[1131] Step 5:

[1132] Sending the conversion results

[1133] The server sends the generated 3D model data to the terminal as an HTTP response.

[1134] Input: Format-converted 3D model data

[1135] Output: HTTP response sent from the server to the device

[1136] Specific operation: The server encodes the 3D model data and sends it to the terminal as an HTTP response.

[1137] Step 6:

[1138] Receiving and processing data

[1139] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[1140] Input: 3D model data sent from the server

[1141] Output: 3D model data loaded into an augmented reality viewing platform

[1142] Specific operation: The device decodes the received data and inputs it into an augmented reality display platform such as the Unity engine.

[1143] Step 7:

[1144] User Rating

[1145] Users can visually check and evaluate the generated 3D models using a VR headset.

[1146] Input: 3D model displayed on an augmented reality viewing platform

[1147] Output: User ratings and feedback

[1148] Specific actions: The user puts on a VR headset and evaluates the 3D model while examining it in detail.

[1149] Step 8:

[1150] Enter your rating

[1151] The user enters feedback into the device's rating UI.

[1152] Input: User rating and feedback (e.g., "I'd like a window added to the right wall")

[1153] Output: Feedback data entered into the terminal

[1154] Specific actions: The user enters feedback into the evaluation form on the device and presses the submit button.

[1155] Step 9:

[1156] Submitting evaluation data

[1157] The device sends the user's feedback to the server as an HTTP POST request.

[1158] Input: Feedback data entered by the user into the rating UI

[1159] Output: HTTP POST request sent to the server

[1160] Specific operation: The device creates an HTTP request containing feedback data and sends it to the server.

[1161] Step 10:

[1162] Processing Feedback

[1163] The server analyzes the received feedback and reevaluates and corrects it using a generative AI model.

[1164] Input: Feedback data sent from the device (e.g., "Please add a window to the right wall.")

[1165] Output: Revised plan / strategy proposal (e.g. "Office space with a window added to the right wall")

[1166] Specific operation: The server analyzes the feedback data and inputs it into the generative AI model to generate revised proposals.

[1167] Step 11:

[1168] Sending regeneration results

[1169] The server retransmits the modified 3D model data to the terminal as an HTTP response.

[1170] Input: 3D model data regenerated from a generative AI model

[1171] Output: HTTP response sent from the server to the device

[1172] Specific operation: The server encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[1173] Step 12:

[1174] Final evaluation

[1175] Users review the new model and provide feedback and ratings until they are satisfied.

[1176] Input: Augmented reality viewing platform displaying the modified 3D model data.

[1177] Output: User's final rating and feedback

[1178] Specific actions: Users make final checks on the new office design and continue rating and providing feedback until they are satisfied.

[1179] (Application example 1)

[1180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Conventional evaluation systems for store design plans and strategy proposals are limited to simple 2D drawings or still images, making visual evaluation in real space difficult. Furthermore, the process of quickly modifying and reevaluating designs based on user feedback is inefficient, placing a significant burden on store owners and designers. A system that can solve these problems, quickly obtain intuitive, highly accurate feedback, and enable visual evaluation in real space was needed.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1183] In this invention, the server includes means for acquiring plans generated by the generating means, means for visualizing the generated plans in real space using augmented reality technology, means for a user to evaluate the visualized plans for optimizing the store layout, and means for acquiring user feedback, thereby enabling the user to visually evaluate and modify the store layout in real space and quickly reflect the evaluation results.

[1184] The "generation means" is a means for automatically generating plans and strategy proposals based on the data input by the user.

[1185] The "means for acquiring a plan" is a means for acquiring the plan or strategy plan generated by the generation means within the system.

[1186] "Augmented reality technology" is a technology that displays digital information overlaid on real space.

[1187] The "means for visualizing in real space" refers to a means for visually displaying the generated proposal in actual physical space using augmented reality technology.

[1188] "Means for optimizing store layout" refers to means for effectively planning, arranging, and optimizing the interior and layout of a store.

[1189] "Means for users to evaluate visualized ideas" refers to the means by which users can evaluate and provide feedback on ideas displayed in augmented reality.

[1190] The "means for obtaining user feedback" refers to a means for collecting user evaluations and feedback.

[1191] "Correction methods" are methods for correcting and updating the generated proposals based on user feedback.

[1192] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[1193] The present invention provides a system that can evaluate generated store layouts and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together so that the content generated based on user input can be intuitively evaluated.

[1194] System program processing overview

[1195] 1. A user inputs text data related to the store design concept using a device (e.g., smartphone, tablet, or PC). This input is performed in the chat UI as a prompt sentence.

[1196] 2. The device sends this text data to the server via an HTTP POST request.

[1197] 3. The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals, including specific store layouts and interior designs.

[1198] 4. The server converts the generated design into 3D model data that can be used with augmented reality technology. The conversion process uses a 3D modeling tool such as the Unity engine.

[1199] 5. The server sends the 3D model data to the device, where it is encoded and returned as an HTTP response.

[1200] 6. The device decodes the 3D model data received from the server and inputs it into an augmented reality display platform, for example, using the Unity engine to display the model on a VR headset or AR device.

[1201] 7. The user visually checks the generated 3D model in the actual store using a VR headset or smart glasses and evaluates the store layout. This evaluation is performed, for example, via an interface.

[1202] 8. The user inputs evaluation feedback into the device, providing specific feedback such as "Please move the display to the left wall and add lighting in the center."

[1203] 9. The device sends the user's feedback to the server, also via an HTTP POST request.

[1204] 10. The server analyzes the received feedback and uses the generation AI to revise and generate a new proposal. The revised 3D model data is sent back to the device, and this process is repeated until the user is satisfied.

[1205] Hardware and software used

[1206] Devices: smartphones, tablets, PCs, VR headsets, smart glasses

[1207] Server: Generative AI model, data analysis software

[1208] Augmented reality display platform: Unity Engine

[1209] Specific examples

[1210] For example, consider a case where a user types "Please suggest a new cafe layout" into the chat UI on their device. This input is sent to the server, which uses generative AI to generate the following proposals:

[1211] "There will be a coffee counter in the center, lounge seating on the right, and a take-out counter on the left."

[1212] This generated proposal is converted into 3D model data and sent to the device. The user puts on a VR headset and views the 3D model. If the user provides feedback such as "Please place the lounge seats closer to the windows," the server receives this feedback and regenerates the proposal. The regenerated 3D model is then provided to the user again, allowing them to evaluate and modify the layout in a concrete and intuitive manner.

[1213] Example prompt sentence:

[1214] "Place a coffee counter in the center, lounge seating on the right, and a takeout counter on the left."

[1215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1216] Step 1:

[1217] The user inputs text data related to the store design concept using a terminal. The user's input is displayed as a prompt in the chat UI.

[1218] Step 2:

[1219] The terminal sends this text data to the server. The data is sent via an HTTP POST request. The input is the store design text data entered by the user, and the output is the request sent to the server.

[1220] Step 3:

[1221] The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals. The input is the prompt text extracted from the HTTP request, and the output is the generated store layout proposal data. Specifically, the generative AI model analyzes the prompt text and creates an appropriate layout proposal.

[1222] Step 4:

[1223] The server converts the generated proposal into 3D model data that can be used with augmented reality technology. The input is the generated layout proposal data, and the output is the 3D model data. Specifically, the server converts the data into model data using a 3D modeling tool such as the Unity engine.

[1224] Step 5:

[1225] The server sends the 3D model data to the terminal. The input is the encoded 3D model data, and the output is the data returned as an HTTP response.

[1226] Step 6:

[1227] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform. The input is the decoded 3D model data, and the output is the model displayed on the AR / VR device. Specifically, it uses the Unity engine to load and display the 3D model on the VR headset or AR device.

[1228] Step 7:

[1229] Users use a VR headset or smart glasses to visually check the 3D model generated in the actual store and evaluate the store layout. The input is the 3D model on the AR / VR device, and the output is the user's evaluation data.

[1230] Step 8:

[1231] The user inputs evaluation feedback into the terminal. For example, specific feedback such as "Please move the display to the left wall and add a light in the center" is provided. The input is the user's specific feedback text, and the output is the feedback data.

[1232] Step 9:

[1233] The terminal sends user feedback to the server. The input is the feedback text data entered by the user, and the output is the HTTP POST request sent to the server.

[1234] Step 10:

[1235] The server analyzes the received feedback and uses the generative AI to revise and generate new layout proposals. The input is the feedback data and the initial layout proposal data, and the output is the revised new store layout proposal data. Specifically, this is the process by which the generative AI model incorporates user feedback and generates new layout proposals.

[1236] Step 11:

[1237] The server resends the modified 3D model data to the device. The input is the new encoded 3D model data, and the output is the resent HTTP response. This process is repeated until the user is satisfied with the result.

[1238] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1239] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[1240] System program processing overview

[1241] 1. User Input

[1242] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1243] As a concrete example, the user inputs "Please suggest a design for our new office."

[1244] 2. Sending input data

[1245] The device receives the user's input text and sends it to the server using an HTTP POST request.

[1246] As a specific example, the terminal sends input text to the server via an HTTP POST request.

[1247] 3. Planning and strategy creation

[1248] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[1249] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[1250] 4. Format conversion of generated results

[1251] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1252] As a specific example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[1253] 5. Send the conversion results

[1254] The server sends the converted 3D model data to the terminal.

[1255] As a specific example, the server encodes 3D model data and sends it back to the terminal.

[1256] 6. Receipt and processing of data

[1257] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[1258] As a specific example, the terminal uses the Unity engine to load 3D model data and display it on the VR device.

[1259] 7. User Ratings

[1260] Users use a VR headset to view and evaluate the generated 3D model.

[1261] Users can freely observe and move around the office layout and design within the virtual space.

[1262] 8. Acquiring Emotion Data

[1263] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[1264] As a specific example, the emotional state of the user is analyzed from their facial expressions and voice, and emotional data such as "joy" and "anxiety" is collected.

[1265] 9. Enter your rating

[1266] The user enters feedback into the device's rating UI.

[1267] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[1268] 10. Sending Rating and Emotion Data

[1269] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[1270] The feedback includes the user's rating and specific correction instructions, as well as emotional data.

[1271] 11. Feedback and Emotional Data Processing

[1272] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[1273] As a concrete example, the server generates a new office design that reflects the modification of "adding a window to the right wall" and the emotional data that confirms "joy."

[1274] 12. Sending the regeneration results

[1275] The server sends the modified 3D model data back to the device.

[1276] The modified data is encoded and sent back to the terminal as an HTTP response.

[1277] 13. Final Evaluation

[1278] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[1279] Allow users to see the new model and iterate with feedback as needed.

[1280] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[1281] The processing flow will be explained below.

[1282] Step 1:

[1283] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1284] As a concrete example, the user inputs "Please suggest a design for our new office."

[1285] Step 2:

[1286] The device receives the user's input text and sends it to the server using an HTTP POST request.

[1287] Here, the text data is properly formatted and sent to a pre-configured API endpoint.

[1288] Step 3:

[1289] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[1290] For example, the server generates a suggestion such as "a spacious office space with plenty of natural light." The generative AI inputs the received text data into a natural language processing (NLP) engine and runs an algorithm to generate the appropriate output.

[1291] Step 4:

[1292] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1293] As a specific example, a script or software is executed to convert the generated proposal into a data format for 3D modeling (e.g., an OBJ file or an FBX file).

[1294] Step 5:

[1295] The server sends the converted 3D model data to the terminal.

[1296] Here, the generated 3D model data is encoded as an HTTP response and sent back to the terminal.

[1297] Step 6:

[1298] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[1299] As a specific example, the device will use an xR development environment such as the Unity engine to read 3D model data and make arrangements to display it on the VR headset.

[1300] Step 7:

[1301] Users use a VR headset to view and evaluate the generated 3D model.

[1302] Users can freely observe and move around the office layout and design in the virtual space, where the emotion engine analyzes the user's facial expressions and voice and collects them as feedback.

[1303] Step 8:

[1304] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[1305] As a specific example, facial expressions and voice data are analyzed in real time via the user's camera and microphone, and emotional states such as "joy" and "anxiety" are recorded as digital data.

[1306] Step 9:

[1307] The user enters feedback into the device's rating UI.

[1308] As an example, a user may enter a text rating such as "Add a window to the right wall."

[1309] Step 10:

[1310] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[1311] Here, the feedback content and emotional data are linked, encoded, and transmitted.

[1312] Step 11:

[1313] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[1314] As a concrete example, the server generates a new office design that reflects the instruction "add a window to the right wall" and the sentiment data "user is satisfied," again using a natural language processing (NLP) engine and 3D modeling software.

[1315] Step 12:

[1316] The server sends the modified 3D model data back to the device.

[1317] The modified data is then re-encoded and sent to the device as an HTTP response.

[1318] Step 13:

[1319] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[1320] This allows the user to see the new model.

[1321] Step 14:

[1322] The user evaluates the new 3D model again and provides feedback as needed.

[1323] This process is repeated until the user is completely satisfied with the design, and the emotion engine supports this, enabling a better feedback loop.

[1324] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[1325] Example 2

[1326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1327] Existing systems for evaluating plans and strategy proposals have the problem of being unable to incorporate users' intuitive evaluations and emotional data. Furthermore, the time required to reflect and revise evaluation results in real time makes it difficult to achieve an efficient feedback cycle.

[1328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1329] In this invention, the server includes means for acquiring the ideas generated by the generating means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotional data of the user, means for correcting the generated ideas based on the emotional data, and means for revising the corrected ideas, thereby enabling an efficient feedback cycle that incorporates the user's intuitive evaluation and emotional data.

[1330] The "generation means" is a means for generating plans and strategy proposals based on the data input by the user.

[1331] "Augmented reality technology" is a technology that overlays virtual information onto the real world.

[1332] The "visualization means" is a means for visually displaying the generated proposals.

[1333] The "means for user evaluation" is a means for the user to evaluate the visualized proposals.

[1334] The "means for acquiring user emotional data" refers to a means for analyzing the user's emotional state and collecting that data.

[1335] The "means for correcting the proposal generated based on emotion data" is a means for correcting the proposal generated based on the acquired emotion data.

[1336] A "terminal" is an electronic device that a user uses to interact.

[1337] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[1338] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[1339] First, the user uses the device to input text data related to plans and strategies into the chat UI. For example, the user might input, "Please come up with a design proposal for our new office." This prompt is sent from the device to the server, where processing begins.

[1340] The server then uses a generative AI model to analyze the received text data and generate a plan or strategy proposal. For example, based on the input data, it generates a proposal such as "a spacious office space with plenty of natural light." The proposal is then converted into a format that can be used with augmented reality technology. Specifically, it is converted into a data format for 3D modeling (e.g., OBJ or FBX files).

[1341] The converted 3D model data is sent from the server to the device, which then decodes the data and loads it into the augmented reality display platform. Specifically, the Unity engine is used to display the 3D model data on a VR device. The user then wears a VR headset and checks and evaluates the generated 3D model in a virtual space.

[1342] While the evaluation is in progress, the device uses an emotion engine to obtain the user's emotional data. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety," and collects that data. The user enters feedback into the evaluation UI, specifically rating the device by saying, "I'd like a window added to the right wall."

[1343] The device then sends the user's feedback and emotional data to a server, which analyzes the data and uses a generative AI model to regenerate and modify plans and strategies. For example, a new office design could be generated based on the modification of "adding a window to the right wall" and the emotional data of "joy."

[1344] Finally, the revised 3D model data is sent back to the device, which then decodes it again and loads it into the augmented reality display platform. The user then checks the new model and, if necessary, the feedback cycle is repeated. This process allows the user to visually evaluate the generated plans and strategies, and makes accurate decisions based on intuitive feedback based on emotional data.

[1345] As a specific example, this system uses hardware and software such as generative AI models, augmented reality technology, emotion engines, evaluation UIs, and the Unity engine to efficiently generate, evaluate, and revise plans and strategies.

[1346] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1347] Step 1:

[1348] The user enters a prompt statement

[1349] The user uses a terminal to input text data related to plans and strategies into the chat UI. Specifically, the user inputs a prompt such as, "Please come up with a design proposal for our new office." This input data is then sent to the server in the next step.

[1350] Input: Text data "Please propose a design for our new office"

[1351] Output: User input text data

[1352] Step 2:

[1353] The device sends the input data to the server

[1354] The device sends the user's input text to the server as an HTTP POST request. Specifically, the device formats the input data in JSON format and issues a POST request to the API endpoint.

[1355] Input: User-entered text data

[1356] Output: HTTP POST request to send to the server

[1357] Step 3:

[1358] The server generates plans and strategies

[1359] The server analyzes the input data it receives and generates plans and strategy proposals using a generative AI model. For example, based on the input data "Please propose a design for a new office," the server might generate a proposal for a "spacious office space with plenty of natural light."

[1360] Input: User-supplied text data extracted from the HTTP POST request

[1361] Output: Plans and strategies generated by the generative AI model

[1362] Step 4:

[1363] The server converts the generated results

[1364] The server converts the generated proposals into a format that can be used with augmented reality technology. Specifically, it uses a library to convert the generated proposals into data formats for 3D modeling (e.g., OBJ files or FBX files).

[1365] Input: Proposals generated by a generative AI model

[1366] Output: 3D model data converted into a format usable by augmented reality technology

[1367] Step 5:

[1368] The server sends the conversion result to the device.

[1369] The server encodes the converted 3D model data and sends it to the terminal as an HTTP response. Specifically, it encodes the 3D data into an appropriate format and returns it as an API response.

[1370] Input: 3D model data converted into a format usable by augmented reality technology

[1371] Output: 3D model data encoded as an HTTP response

[1372] Step 6:

[1373] The device receives and processes the data

[1374] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform. Specifically, the 3D model data is displayed on the VR device using the Unity engine.

[1375] Input: 3D model data encoded as an HTTP response

[1376] Output: 3D model loaded into Unity engine

[1377] Step 7:

[1378] The user evaluates the generated 3D model

[1379] The user wears a VR headset and checks and evaluates the generated 3D model in a virtual space. Specifically, the user can move freely within the virtual space and observe the layout and design of the office.

[1380] Input: 3D model loaded into the Unity engine

[1381] Output: User ratings and experience information

[1382] Step 8:

[1383] The device acquires the user's emotional data.

[1384] The device uses an emotion engine to collect data on the user's emotions. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety."

[1385] Input: User ratings and experience information

[1386] Output: Emotion data analyzed by the emotion engine

[1387] Step 9:

[1388] User enters rating feedback

[1389] The user enters feedback into the device's evaluation UI. For example, the user might say, "I'd like a window added to the right wall."

[1390] Input: User ratings and feedback

[1391] Output: Feedback content

[1392] Step 10:

[1393] The device sends the evaluation data and emotion data to the server.

[1394] The device sends the user's feedback and emotion data as an HTTP POST request to the server. Specifically, the feedback content and emotion data are compiled in JSON format and sent to the API endpoint.

[1395] Input: Feedback and emotion data

[1396] Output: HTTP POST request to send to the server

[1397] Step 11:

[1398] The server processes the feedback and emotion data

[1399] The server analyzes the received feedback and emotion data and uses a generative AI model to regenerate and modify the proposal. For example, a new office design is generated based on the modification of "adding a window to the right wall" and the emotion data of "joy."

[1400] Input: Feedback content and sentiment data extracted from HTTP POST requests

[1401] Output: Revised plans and strategies

[1402] Step 12:

[1403] The server sends the regeneration result to the terminal.

[1404] The server re-encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[1405] Input: Revised plans and strategies

[1406] Output: Modified 3D model data encoded as an HTTP response

[1407] Step 13:

[1408] The device processes the re-received 3D model data

[1409] The device then decodes the re-received 3D model data and loads it back into the augmented reality display platform. The user can then review the new model and provide feedback as needed. This allows for visual evaluation of the plans and strategies generated, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[1410] Input: Modified 3D model data encoded as an HTTP response

[1411] Output: Modified 3D model reloaded into Unity engine

[1412] (Application example 2)

[1413] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1414] Conventional evaluation systems for planning and strategy proposals obtain feedback without taking user emotions into account, which can lead to reduced accuracy of revisions and reduced user satisfaction. Furthermore, there is a lack of systems that provide intuitive evaluation methods using AR technology. Therefore, there is a need for the development of a highly accurate and user-friendly evaluation system.

[1415] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ideas generated by a generation means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotion data, and means for analyzing the emotion data. This enables highly accurate evaluation and correction based on the user's evaluation and emotion data.

[1416] "Generation means" refers to a device or software that generates a plan or strategy proposal based on user input.

[1417] "Augmented reality technology" is a technology that overlays virtual information onto real visual information.

[1418] "Evaluation means" refers to an interface or tool that allows a user to review and evaluate the generated proposals.

[1419] "Emotion data" refers to information about the user's emotional state obtained from facial expressions, voice, body movements, etc.

[1420] The "modification means" refers to a device or software that has the function of modifying the proposal generated based on the user's evaluation and emotion data.

[1421] "Regeneration" is the process of regenerating something new based on feedback or additional information.

[1422] "Communication means" refers to the communication protocol or technology used for exchanging data between the user's terminal and the generation means.

[1423] The present invention provides a system that generates plans and strategies based on user input, visualizes the plans using augmented reality technology, and allows the user to evaluate them to propose more accurate revisions. In this system, a generation unit, an augmented reality unit, an evaluation unit, an emotion data acquisition unit, and a communication unit work in cooperation with each other.

[1424] 1. User Input

[1425] Users input design requests using a chat UI on their smartphone, for example, by entering specific instructions such as, "Please propose a layout for a new store. Make the entrance more open and create a spacious area around the cash register."

[1426] 2. Sending input data

[1427] The terminal sends the user's input text to the server using an HTTP POST request, and this data is obtained by the generating means.

[1428] 3. Use of generative AI models

[1429] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the received text data and generate a proposed store layout plan, which is used as the initial planning proposal.

[1430] 4. Use of Augmented Reality Technology

[1431] The generated proposal is converted into a 3D modeling data format (e.g., OBJ or FBX file) on the server side, and then this data is sent back to the device, where it is displayed in AR using the Unity engine.

[1432] 5. User Ratings

[1433] Users can use their smartphone's camera and microphone to view the displayed design proposals, and their evaluations and emotional data are collected. For example, users can provide feedback such as, "Please make the entrance door double-doors. Please add LED lights to the cash register."

[1434] 6. Emotion Data Analysis and Correction Suggestion Generation

[1435] The emotional data and feedback acquired by the device are sent to the server and analyzed by the emotional data acquisition means and evaluation means. The generative AI model generates revision suggestions based on this data. The revision suggestions are also visualized using AR technology, and the user evaluates them again.

[1436] 7. Final Evaluation and Regeneration

[1437] The user again provides their rating and feedback, and the server receives the data and generates the final revision proposal, thereby obtaining the optimal proposal based on the user's rating and sentiment data.

[1438] Specific examples

[1439] User input prompt:

[1440] "Please suggest a new store layout, with a more open entrance and more space around the registers."

[1441] Feedback example:

[1442] "Please make the entrance door double-doored. Please add LED lights to the cash register."

[1443] In this way, a highly accurate evaluation system that reflects user feedback and emotional data can be realized.

[1444] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1445] Step 1:

[1446] The user enters a design request using the chat UI on their smartphone and presses the send button. For example, they might enter, "Please propose a layout for our new store. Make the entrance more open and create a spacious area around the cash register." The entered data is sent from the device to the server as an HTTP POST request.

[1447] Step 2:

[1448] The server receives the HTTP POST request and analyzes the received text data using the generation method. The analyzed data is input into a generative AI model (e.g., OpenAI GPT-4), which generates an initial proposal for the store layout. The generated result is then formatted as a 3D model.

[1449] Step 3:

[1450] The server encodes the converted 3D model data (e.g., OBJ or FBX files) and sends it to the device. The device decodes the 3D model data received from the server and displays it in AR using the Unity engine.

[1451] Step 4:

[1452] The user uses the smartphone camera and screen to view the generated AR store layout, and the system captures user feedback and emotional data obtained from facial expressions and voice.

[1453] Step 5:

[1454] The device sends the text feedback entered by the user and the acquired emotion data to the server as an HTTP POST request.

[1455] Step 6:

[1456] The server analyzes the feedback and emotional data received from the user. The emotional data acquisition means analyzes the user's emotional state (e.g., joy or anxiety) and generates correction suggestions using the generative AI model.

[1457] Step 7:

[1458] The server re-encodes the modified 3D model data and sends it to the device, which decodes it again and uses the Unity engine to display the proposed modifications in AR.

[1459] Step 8:

[1460] The user then evaluates the proposed AR revisions and provides final feedback. The captured emotion data is also collected on the device.

[1461] Step 9:

[1462] The device sends the user's final feedback and emotion data to the server, which then reanalyzes the data to generate a final design, which then encodes the 3D model data of the final design and sends it to the device.

[1463] Step 10:

[1464] The device then re-decodes the received 3D model data and uses the Unity engine to display the final design in AR, allowing the user to review the design and make further evaluations and revisions as needed.

[1465] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1466] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1467] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1468] [Fourth embodiment]

[1469] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1470] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1471] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1472] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1473] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1474] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1475] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1476] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1477] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1480] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1481] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1482] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[1483] System program processing overview

[1484] 1. User Input

[1485] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1486] Example: A user types, "Come up with a new office design."

[1487] 2. Sending input data

[1488] The terminal sends the user's input text to the server.

[1489] Example: The device sends the input text to the server via an HTTP POST request.

[1490] 3. Planning and strategy creation

[1491] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[1492] Example: The server generates a "spacious office space with lots of natural light."

[1493] 4. Format conversion of generated results

[1494] The server converts the generated proposals into a format that can be used with augmented reality technology (e.g., 3D model data).

[1495] Example: The server converts the generated proposal into a data format for 3D modeling.

[1496] 5. Send the conversion results

[1497] The 3D model data generated by the server is sent to the terminal.

[1498] Example: The server encodes 3D model data and sends it back to the device.

[1499] 6. Receipt and processing of data

[1500] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[1501] Example: The device loads 3D model data into the Unity engine and displays it on the VR device.

[1502] 7. User Ratings

[1503] Users visually check and evaluate the generated 3D models using a VR headset.

[1504] Example: A user puts on a VR headset and walks freely around a 3D model of an office.

[1505] 8. Enter your rating

[1506] The user enters feedback into the device's rating UI.

[1507] Example: A user enters feedback such as "I'd like a window added to the right wall."

[1508] 9. Submission of evaluation data

[1509] The terminal sends the user's feedback to the server.

[1510] Example: The device sends feedback to the server via an HTTP POST request.

[1511] 10. Processing Feedback

[1512] The server analyzes the received feedback, and the generating AI reevaluates and corrects it.

[1513] Example: The server executes the modification "add a window to the right wall" and generates the office design again.

[1514] 11. Sending the regeneration results

[1515] The server resends the corrected 3D model data to the device.

[1516] Example: The server sends the modified 3D data back to the device.

[1517] 12. Final Evaluation

[1518] Users review the new model and provide feedback and ratings until they are satisfied.

[1519] Example: Users review and rate new office designs until they are satisfied.

[1520] Specific examples

[1521] When a user types "Please propose a conference room layout" into the device's chat UI, the device sends this text data to the server. The server uses generative AI to generate a proposal for a "conference room with a large table, multiple chairs, and a large display" and converts it into 3D model data. This 3D model data is sent to the device, which displays the received data on the VR device. The user uses a VR headset to check the conference room model and provides feedback such as "I'd like the table to be moved closer to the window." The server then receives the feedback, regenerates the proposal, and sends it to the device, where the new model is displayed. The user can repeat this process until they are finally satisfied.

[1522] The present invention can provide a system that allows users to visually and intuitively evaluate generated ideas and provide highly accurate feedback.

[1523] The processing flow will be explained below.

[1524] Step 1:

[1525] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1526] As a concrete example, the user inputs "Please suggest a design for our new office."

[1527] Step 2:

[1528] The device receives the user's input text and sends it to the server using an HTTP POST request.

[1529] The request includes the text data and any required metadata.

[1530] Step 3:

[1531] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[1532] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[1533] Step 4:

[1534] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1535] For example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[1536] Step 5:

[1537] The server sends the converted 3D model data to the terminal.

[1538] This data is encoded and sent back to the device as an HTTP response.

[1539] Step 6:

[1540] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[1541] For example, the device uses the Unity engine to load 3D model data and display it on the VR headset.

[1542] Step 7:

[1543] Users use a VR headset to view and evaluate the generated 3D model.

[1544] Users can freely observe and move around the office layout and design within the virtual space.

[1545] Step 8:

[1546] The user enters feedback into the device's rating UI.

[1547] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[1548] Step 9:

[1549] The device receives the user's feedback and sends it to the server via an HTTP POST request.

[1550] The feedback includes the user's rating and specific correction instructions.

[1551] Step 10:

[1552] The server analyzes the received feedback and uses generative AI to regenerate and revise the proposal.

[1553] For example, the server generates a new office design that reflects the modification "add a window to the right wall."

[1554] Step 11:

[1555] The server sends the modified 3D model data back to the device.

[1556] The corrected data is re-encoded and sent back to the terminal as an HTTP response.

[1557] Step 12:

[1558] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[1559] Allow users to see the new model.

[1560] Step 13:

[1561] The user evaluates the new 3D model again and provides feedback as needed.

[1562] This process is repeated until the user is satisfied.

[1563] This allows for visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback.

[1564] Example 1

[1565] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1566] In systems using conventional generation and visualization methods, it was difficult for users to intuitively evaluate the generated proposals. Furthermore, there was a lack of a process for quickly reflecting user evaluations in feedback and regenerating the proposals. This made it difficult to provide highly accurate proposals that met the user's requirements in a short amount of time.

[1567] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1568] In this invention, the server includes means for receiving a prompt sentence based on a user's input and generating a proposal using a generative AI model, means for converting the generated proposal into a format usable in augmented reality technology, and means for displaying the converted proposal on an augmented reality device, thereby enabling the user to intuitively evaluate the generated proposal and quickly modify and regenerate it based on the evaluation.

[1569] "User input" refers to text data that a user inputs using a terminal through a chat UI, etc.

[1570] A "prompt" refers to an instruction or question that is passed to a generative AI model based on user input.

[1571] A "generative AI model" refers to an artificial intelligence model that receives a prompt sentence as input and generates a plan or strategy proposal in the specified format.

[1572] "Means of generation" refers to the means of using a generative AI model to execute the process of generating plans and strategy proposals based on user input.

[1573] "Augmented reality technology" refers to technology that displays digital information overlaid on the physical environment.

[1574] "Means for converting" refers to means for converting ideas generated by a generative AI model into a data format that can be used in augmented reality technology.

[1575] "Means for displaying" refers to means for displaying the converted data on an augmented reality or VR device.

[1576] The term "means for evaluation" refers to the means used by a user to visually check and evaluate the generated proposals using an augmented reality device.

[1577] The "user evaluation acquisition means" refers to a means for collecting evaluations and feedback given by users on generated proposals.

[1578] "Means for correction" refers to the process of correcting the generated proposal based on user evaluations and feedback and then generating it again.

[1579] "Means of communication" refers to the process of sending and receiving data between a terminal and a server.

[1580] This invention provides a system that intuitively evaluates plans and strategies generated based on user input and quickly modifies them based on user feedback. This system is realized by combining a generation unit, augmented reality technology, and a user evaluation acquisition unit.

[1581] First, a user inputs text data into the chat UI using a terminal. For example, the user inputs "Please come up with a design for our new office." This input is sent to the server by the terminal via an HTTP POST request.

[1582] The server analyzes the received text data and generates plans and strategy proposals using a generative AI model. Specifically, in response to the received prompt "New office design," the server uses the generative AI model to generate a proposal for "a spacious office space with plenty of natural light."

[1583] The server then converts the generated proposal into a format that can be used with augmented reality technology, converting the text data into 3D model data (e.g., OBJ format). The converted data is then sent back to the device as an HTTP response.

[1584] The device decodes the received 3D model data and inputs it into an augmented reality display platform such as the Unity engine, which then displays the 3D model on the VR device.

[1585] The user visually checks the generated 3D model using a VR headset. For example, the user puts on the VR headset and checks the generated 3D model of an office in detail. Next, the user enters feedback in an evaluation form, such as "I would like a window added to the right wall." This feedback is then sent back to the server from the device.

[1586] The server analyzes the received feedback and reevaluates and modifies it using the generative AI model. Specifically, it generates a new office design that reflects the modification instruction to "add a window to the right wall." This modified 3D model data is then sent back from the server to the device.

[1587] Users review new models and provide feedback until they are satisfied. Through this cycle, users can intuitively provide highly accurate feedback.

[1588] As a result, this system combines the use of generative AI models based on user input with augmented reality technology to enable highly accurate evaluation and revision of plans and strategic proposals.

[1589] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1590] Step 1:

[1591] User Input

[1592] The user uses the terminal to input text data related to the plan or strategy into the chat UI.

[1593] Input: Text data entered by the user into the chat UI (e.g., "Please come up with a design for our new office.")

[1594] Specific operation: The user enters text data into the chat input field on the device and presses the send button.

[1595] Step 2:

[1596] Sending input data

[1597] The terminal sends the user's input text to the server as an HTTP POST request.

[1598] Input: The text data entered by the user in step 1

[1599] Output: HTTP POST request sent to the server

[1600] Specific operation: The terminal creates an HTTP request and sends the request including text data to the server.

[1601] Step 3:

[1602] Planning and strategy creation

[1603] The server analyzes the received text data and uses a generative AI model to generate plans and strategy proposals.

[1604] Input: Prompt sent from the terminal (e.g., "Could you come up with a design for our new office?")

[1605] Output: Generated planning and strategy proposals (e.g., "A spacious office space with plenty of natural light")

[1606] Specific operation: The server analyzes the text data and inputs it into a generative AI model to generate suggestions.

[1607] Step 4:

[1608] Format conversion of generated results

[1609] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1610] Input: Text data of plans and strategies output from the generative AI model

[1611] Output: Data format for 3D modeling (e.g. OBJ format)

[1612] Specific operation: The server uses a format conversion engine to convert the generated text data into a 3D model.

[1613] Step 5:

[1614] Sending the conversion results

[1615] The server sends the generated 3D model data to the terminal as an HTTP response.

[1616] Input: Format-converted 3D model data

[1617] Output: HTTP response sent from the server to the device

[1618] Specific operation: The server encodes the 3D model data and sends it to the terminal as an HTTP response.

[1619] Step 6:

[1620] Receiving and processing data

[1621] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform.

[1622] Input: 3D model data sent from the server

[1623] Output: 3D model data loaded into an augmented reality viewing platform

[1624] Specific operation: The device decodes the received data and inputs it into an augmented reality display platform such as the Unity engine.

[1625] Step 7:

[1626] User Rating

[1627] Users can visually check and evaluate the generated 3D models using a VR headset.

[1628] Input: 3D model displayed on an augmented reality viewing platform

[1629] Output: User ratings and feedback

[1630] Specific actions: The user puts on a VR headset and evaluates the 3D model while examining it in detail.

[1631] Step 8:

[1632] Enter your rating

[1633] The user enters feedback into the device's rating UI.

[1634] Input: User rating and feedback (e.g., "I'd like a window added to the right wall")

[1635] Output: Feedback data entered into the terminal

[1636] Specific actions: The user enters feedback into the evaluation form on the device and presses the submit button.

[1637] Step 9:

[1638] Submitting evaluation data

[1639] The device sends the user's feedback to the server as an HTTP POST request.

[1640] Input: Feedback data entered by the user into the rating UI

[1641] Output: HTTP POST request sent to the server

[1642] Specific operation: The device creates an HTTP request containing feedback data and sends it to the server.

[1643] Step 10:

[1644] Processing Feedback

[1645] The server analyzes the received feedback and reevaluates and corrects it using a generative AI model.

[1646] Input: Feedback data sent from the device (e.g., "Please add a window to the right wall.")

[1647] Output: Revised plan / strategy proposal (e.g. "Office space with a window added to the right wall")

[1648] Specific operation: The server analyzes the feedback data and inputs it into the generative AI model to generate revised proposals.

[1649] Step 11:

[1650] Sending regeneration results

[1651] The server retransmits the modified 3D model data to the terminal as an HTTP response.

[1652] Input: 3D model data regenerated from a generative AI model

[1653] Output: HTTP response sent from the server to the device

[1654] Specific operation: The server encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[1655] Step 12:

[1656] Final evaluation

[1657] Users review the new model and provide feedback and ratings until they are satisfied.

[1658] Input: Augmented reality viewing platform displaying the modified 3D model data.

[1659] Output: User's final rating and feedback

[1660] Specific actions: Users make final checks on the new office design and continue rating and providing feedback until they are satisfied.

[1661] (Application example 1)

[1662] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1663] Conventional evaluation systems for store design plans and strategy proposals are limited to simple 2D drawings or still images, making visual evaluation in real space difficult. Furthermore, the process of quickly modifying and reevaluating designs based on user feedback is inefficient, placing a significant burden on store owners and designers. A system that can solve these problems, quickly obtain intuitive, highly accurate feedback, and enable visual evaluation in real space was needed.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1665] In this invention, the server includes means for acquiring plans generated by the generating means, means for visualizing the generated plans in real space using augmented reality technology, means for a user to evaluate the visualized plans for optimizing the store layout, and means for acquiring user feedback, thereby enabling the user to visually evaluate and modify the store layout in real space and quickly reflect the evaluation results.

[1666] The "generation means" is a means for automatically generating plans and strategy proposals based on the data input by the user.

[1667] The "means for acquiring a plan" is a means for acquiring the plan or strategy plan generated by the generation means within the system.

[1668] "Augmented reality technology" is a technology that displays digital information overlaid on real space.

[1669] The "means for visualizing in real space" refers to a means for visually displaying the generated proposal in actual physical space using augmented reality technology.

[1670] "Means for optimizing store layout" refers to means for effectively planning, arranging, and optimizing the interior and layout of a store.

[1671] "Means for users to evaluate visualized ideas" refers to the means by which users can evaluate and provide feedback on ideas displayed in augmented reality.

[1672] The "means for obtaining user feedback" refers to a means for collecting user evaluations and feedback.

[1673] "Correction methods" are methods for correcting and updating the generated proposals based on user feedback.

[1674] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[1675] The present invention provides a system that can evaluate generated store layouts and strategy proposals with high accuracy by combining a generation means, augmented reality technology, and a user evaluation acquisition means. This system is implemented in a form in which these means work together so that the content generated based on user input can be intuitively evaluated.

[1676] System program processing overview

[1677] 1. A user inputs text data related to the store design concept using a device (e.g., smartphone, tablet, or PC). This input is performed in the chat UI as a prompt sentence.

[1678] 2. The device sends this text data to the server via an HTTP POST request.

[1679] 3. The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals, including specific store layouts and interior designs.

[1680] 4. The server converts the generated design into 3D model data that can be used with augmented reality technology. The conversion process uses a 3D modeling tool such as the Unity engine.

[1681] 5. The server sends the 3D model data to the device, where it is encoded and returned as an HTTP response.

[1682] 6. The device decodes the 3D model data received from the server and inputs it into an augmented reality display platform, for example, using the Unity engine to display the model on a VR headset or AR device.

[1683] 7. The user visually checks the generated 3D model in the actual store using a VR headset or smart glasses and evaluates the store layout. This evaluation is performed, for example, via an interface.

[1684] 8. The user inputs evaluation feedback into the device, providing specific feedback such as "Please move the display to the left wall and add lighting in the center."

[1685] 9. The device sends the user's feedback to the server, also via an HTTP POST request.

[1686] 10. The server analyzes the received feedback and uses the generation AI to revise and generate a new proposal. The revised 3D model data is sent back to the device, and this process is repeated until the user is satisfied.

[1687] Hardware and software used

[1688] Devices: smartphones, tablets, PCs, VR headsets, smart glasses

[1689] Server: Generative AI model, data analysis software

[1690] Augmented reality display platform: Unity Engine

[1691] Specific examples

[1692] For example, consider a case where a user types "Please suggest a new cafe layout" into the chat UI on their device. This input is sent to the server, which uses generative AI to generate the following proposals:

[1693] "There will be a coffee counter in the center, lounge seating on the right, and a take-out counter on the left."

[1694] This generated proposal is converted into 3D model data and sent to the device. The user puts on a VR headset and views the 3D model. If the user provides feedback such as "Please place the lounge seats closer to the windows," the server receives this feedback and regenerates the proposal. The regenerated 3D model is then provided to the user again, allowing them to evaluate and modify the layout in a concrete and intuitive manner.

[1695] Example prompt sentence:

[1696] "Place a coffee counter in the center, lounge seating on the right, and a takeout counter on the left."

[1697] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1698] Step 1:

[1699] The user inputs text data related to the store design concept using a terminal. The user's input is displayed as a prompt in the chat UI.

[1700] Step 2:

[1701] The terminal sends this text data to the server. The data is sent via an HTTP POST request. The input is the text data of the store design entered by the user, and the output is the request sent to the server.

[1702] Step 3:

[1703] The server analyzes the received text data and uses generative AI to automatically generate store layout and strategy proposals. The input is the prompt text extracted from the HTTP request, and the output is the generated store layout proposal data. Specifically, the generative AI model analyzes the prompt text and creates an appropriate layout proposal.

[1704] Step 4:

[1705] The server converts the generated proposal into 3D model data that can be used with augmented reality technology. The input is the generated layout proposal data, and the output is the 3D model data. Specifically, the server converts the data into model data using a 3D modeling tool such as the Unity engine.

[1706] Step 5:

[1707] The server sends the 3D model data to the terminal. The input is the encoded 3D model data, and the output is the data returned as an HTTP response.

[1708] Step 6:

[1709] The device decodes the 3D model data received from the server and inputs it into the augmented reality display platform. The input is the decoded 3D model data, and the output is the model displayed on the AR / VR device. Specifically, it uses the Unity engine to load and display the 3D model on the VR headset or AR device.

[1710] Step 7:

[1711] Users use a VR headset or smart glasses to visually check the 3D model generated in the actual store and evaluate the store layout. The input is the 3D model on the AR / VR device, and the output is the user's evaluation data.

[1712] Step 8:

[1713] The user inputs evaluation feedback into the terminal. For example, specific feedback such as "Please move the display to the left wall and add a light in the center" is provided. The input is the user's specific feedback text, and the output is the feedback data.

[1714] Step 9:

[1715] The terminal sends user feedback to the server. The input is the feedback text data entered by the user, and the output is the HTTP POST request sent to the server.

[1716] Step 10:

[1717] The server analyzes the received feedback and uses the generative AI to revise and generate new layout proposals. The input is the feedback data and the initial layout proposal data, and the output is the revised new store layout proposal data. Specifically, this is the process by which the generative AI model incorporates user feedback and generates new layout proposals.

[1718] Step 11:

[1719] The server resends the modified 3D model data to the device. The input is the new encoded 3D model data, and the output is the resent HTTP response. This process is repeated until the user is satisfied with the result.

[1720] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1721] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[1722] System program processing overview

[1723] 1. User Input

[1724] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1725] As a concrete example, the user inputs "Please suggest a design for our new office."

[1726] 2. Sending input data

[1727] The device receives the user's input text and sends it to the server using an HTTP POST request.

[1728] As a specific example, the terminal sends input text to the server via an HTTP POST request.

[1729] 3. Planning and strategy creation

[1730] Based on the text data received by the server, a generative AI is used to generate plans and strategy proposals.

[1731] As a specific example, the server analyzes the received text data and generates a suggestion such as "a spacious office space with plenty of natural light."

[1732] 4. Format conversion of generated results

[1733] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1734] As a specific example, the generated proposal is converted into a data format for 3D modeling (for example, an OBJ file or an FBX file).

[1735] 5. Send the conversion results

[1736] The server sends the converted 3D model data to the terminal.

[1737] As a specific example, the server encodes 3D model data and sends it back to the terminal.

[1738] 6. Receipt and processing of data

[1739] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[1740] As a specific example, the terminal uses the Unity engine to load 3D model data and display it on the VR device.

[1741] 7. User Ratings

[1742] Users use a VR headset to view and evaluate the generated 3D model.

[1743] Users can freely observe and move around the office layout and design within the virtual space.

[1744] 8. Acquiring Emotion Data

[1745] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[1746] As a specific example, the emotional state of the user is analyzed from their facial expressions and voice, and emotional data such as "joy" and "anxiety" is collected.

[1747] 9. Enter your rating

[1748] The user enters feedback into the device's rating UI.

[1749] As a specific example, a user may input a rating such as "I would like a window added to the right wall."

[1750] 10. Sending Rating and Emotion Data

[1751] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[1752] The feedback includes the user's rating and specific correction instructions, as well as emotional data.

[1753] 11. Feedback and Emotional Data Processing

[1754] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[1755] As a concrete example, the server generates a new office design that reflects the modification of "adding a window to the right wall" and the emotional data that confirms "joy."

[1756] 12. Sending the regeneration results

[1757] The server sends the modified 3D model data back to the device.

[1758] The modified data is encoded and sent back to the terminal as an HTTP response.

[1759] 13. Final Evaluation

[1760] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[1761] Allow users to see the new model and iterate with feedback as needed.

[1762] This enables the visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[1763] The processing flow will be explained below.

[1764] Step 1:

[1765] A user uses a terminal to input text data related to a plan or strategy into the chat UI.

[1766] As a concrete example, the user inputs "Please suggest a design for our new office."

[1767] Step 2:

[1768] The device receives the user's input text and sends it to the server using an HTTP POST request.

[1769] Here, the text data is properly formatted and sent to a pre-configured API endpoint.

[1770] Step 3:

[1771] The server analyzes the received text data and uses generative AI to generate plans and strategy proposals.

[1772] For example, the server generates a suggestion such as "a spacious office space with plenty of natural light." The generative AI inputs the received text data into a natural language processing (NLP) engine and runs an algorithm to generate the appropriate output.

[1773] Step 4:

[1774] The server converts the generated proposals into a format that can be used with augmented reality technology.

[1775] As a specific example, a script or software is executed to convert the generated proposal into a data format for 3D modeling (e.g., an OBJ file or an FBX file).

[1776] Step 5:

[1777] The server sends the converted 3D model data to the terminal.

[1778] Here, the generated 3D model data is encoded as an HTTP response and sent back to the terminal.

[1779] Step 6:

[1780] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform.

[1781] As a specific example, the device will use an xR development environment such as the Unity engine to read 3D model data and make arrangements to display it on the VR headset.

[1782] Step 7:

[1783] Users use a VR headset to view and evaluate the generated 3D model.

[1784] Users can freely observe and move around the office layout and design in the virtual space, where the emotion engine analyzes the user's facial expressions and voice and collects them as feedback.

[1785] Step 8:

[1786] The terminal acquires the user's emotion data through the emotion engine during the user's evaluation.

[1787] As a specific example, facial expressions and voice data are analyzed in real time via the user's camera and microphone, and emotional states such as "joy" and "anxiety" are recorded as digital data.

[1788] Step 9:

[1789] The user enters feedback into the device's rating UI.

[1790] As an example, a user may enter a text rating such as "Add a window to the right wall."

[1791] Step 10:

[1792] The device receives user feedback and emotion data and sends it to the server via an HTTP POST request.

[1793] Here, the feedback content and emotional data are linked, encoded, and transmitted.

[1794] Step 11:

[1795] The server analyzes the received feedback and emotional data and uses generative AI to regenerate and revise the proposals.

[1796] As a concrete example, the server generates a new office design that reflects the instruction "add a window to the right wall" and the sentiment data "user is satisfied," again using a natural language processing (NLP) engine and 3D modeling software.

[1797] Step 12:

[1798] The server sends the modified 3D model data back to the device.

[1799] The modified data is then re-encoded and sent to the device as an HTTP response.

[1800] Step 13:

[1801] The device decodes the re-received 3D model data and loads it back into the augmented reality display platform.

[1802] This allows the user to see the new model.

[1803] Step 14:

[1804] The user evaluates the new 3D model again and provides feedback as needed.

[1805] This process is repeated until the user is completely satisfied with the design, and the emotion engine supports this, enabling a better feedback loop.

[1806] This enables visual evaluation of generated plans and strategy proposals, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[1807] Example 2

[1808] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1809] Existing systems for evaluating plans and strategy proposals have the problem of being unable to incorporate users' intuitive evaluations and emotional data. Furthermore, the time required to reflect and revise evaluation results in real time makes it difficult to achieve an efficient feedback cycle.

[1810] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1811] In this invention, the server includes means for acquiring the ideas generated by the generating means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotional data of the user, means for correcting the generated ideas based on the emotional data, and means for revising the corrected ideas, thereby enabling an efficient feedback cycle that incorporates the user's intuitive evaluation and emotional data.

[1812] The "generation means" is a means for generating plans and strategy proposals based on the data input by the user.

[1813] "Augmented reality technology" is a technology that overlays virtual information onto the real world.

[1814] The "visualization means" is a means for visually displaying the generated proposals.

[1815] The "means for user evaluation" is a means for the user to evaluate the visualized proposals.

[1816] The "means for acquiring user emotional data" refers to a means for analyzing the user's emotional state and collecting that data.

[1817] The "means for correcting the proposal generated based on emotion data" is a means for correcting the proposal generated based on the acquired emotion data.

[1818] A "terminal" is an electronic device that a user uses to interact.

[1819] "Communication means" refers to a means for transmitting and receiving data between a terminal and a server.

[1820] The present invention provides a system that can evaluate generated plans and strategy proposals with high accuracy by combining a generation means, augmented reality technology, a user evaluation acquisition means, and an emotion engine. This system is implemented in a form in which these means work together to enable intuitive evaluation of content generated based on user input.

[1821] First, the user uses the device to input text data related to plans and strategies into the chat UI. For example, the user might input, "Please come up with a design proposal for our new office." This prompt is sent from the device to the server, where processing begins.

[1822] The server then uses a generative AI model to analyze the received text data and generate a plan or strategy proposal. For example, based on the input data, it generates a proposal such as "a spacious office space with plenty of natural light." The proposal is then converted into a format that can be used with augmented reality technology. Specifically, it is converted into a data format for 3D modeling (e.g., OBJ or FBX files).

[1823] The converted 3D model data is sent from the server to the device, which then decodes the data and loads it into the augmented reality display platform. Specifically, the Unity engine is used to display the 3D model data on a VR device. The user then wears a VR headset and checks and evaluates the generated 3D model in a virtual space.

[1824] While the evaluation is in progress, the device uses an emotion engine to obtain the user's emotional data. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety," and collects that data. The user enters feedback into the evaluation UI, specifically rating the device by saying, "I'd like a window added to the right wall."

[1825] The device then sends the user's feedback and emotional data to a server, which analyzes the data and uses a generative AI model to regenerate and modify plans and strategies. For example, a new office design could be generated based on the modification of "adding a window to the right wall" and the emotional data of "joy."

[1826] Finally, the revised 3D model data is sent back to the device, which then decodes it again and loads it into the augmented reality display platform. The user then checks the new model and, if necessary, the feedback cycle is repeated. This process allows the user to visually evaluate the generated plans and strategies, and makes accurate decisions based on intuitive feedback based on emotional data.

[1827] As a specific example, this system uses hardware and software such as generative AI models, augmented reality technology, emotion engines, evaluation UIs, and the Unity engine to efficiently generate, evaluate, and revise plans and strategies.

[1828] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1829] Step 1:

[1830] The user enters a prompt statement

[1831] The user uses a terminal to input text data related to plans and strategies into the chat UI. Specifically, the user inputs a prompt such as, "Please come up with a design proposal for our new office." This input data is then sent to the server in the next step.

[1832] Input: Text data "Please propose a design for our new office"

[1833] Output: User input text data

[1834] Step 2:

[1835] The device sends the input data to the server

[1836] The device sends the user's input text to the server as an HTTP POST request. Specifically, the device formats the input data in JSON format and issues a POST request to the API endpoint.

[1837] Input: User-entered text data

[1838] Output: HTTP POST request to send to the server

[1839] Step 3:

[1840] The server generates plans and strategies

[1841] The server analyzes the input data it receives and generates plans and strategy proposals using a generative AI model. For example, based on the input data "Please propose a design for a new office," the server might generate a proposal for a "spacious office space with plenty of natural light."

[1842] Input: User-supplied text data extracted from the HTTP POST request

[1843] Output: Plans and strategies generated by the generative AI model

[1844] Step 4:

[1845] The server converts the generated results

[1846] The server converts the generated proposals into a format that can be used with augmented reality technology. Specifically, it uses a library to convert the generated proposals into data formats for 3D modeling (e.g., OBJ files or FBX files).

[1847] Input: Proposals generated by a generative AI model

[1848] Output: 3D model data converted into a format usable by augmented reality technology

[1849] Step 5:

[1850] The server sends the conversion result to the device.

[1851] The server encodes the converted 3D model data and sends it to the terminal as an HTTP response. Specifically, it encodes the 3D data into an appropriate format and returns it as an API response.

[1852] Input: 3D model data converted into a format usable by augmented reality technology

[1853] Output: 3D model data encoded as an HTTP response

[1854] Step 6:

[1855] The device receives and processes the data

[1856] The device decodes the 3D model data received from the server and loads it into the augmented reality display platform. Specifically, the 3D model data is displayed on the VR device using the Unity engine.

[1857] Input: 3D model data encoded as an HTTP response

[1858] Output: 3D model loaded into Unity engine

[1859] Step 7:

[1860] The user evaluates the generated 3D model

[1861] The user wears a VR headset and checks and evaluates the generated 3D model in a virtual space. Specifically, the user can move freely within the virtual space and observe the layout and design of the office.

[1862] Input: 3D model loaded into the Unity engine

[1863] Output: User ratings and experience information

[1864] Step 8:

[1865] The device acquires the user's emotional data.

[1866] The device uses an emotion engine to collect data on the user's emotions. For example, it analyzes the user's facial expressions and voice to determine emotional states such as "happiness" or "anxiety."

[1867] Input: User ratings and experience information

[1868] Output: Emotion data analyzed by the emotion engine

[1869] Step 9:

[1870] User enters rating feedback

[1871] The user enters feedback into the device's evaluation UI. For example, the user might say, "I'd like a window added to the right wall."

[1872] Input: User ratings and feedback

[1873] Output: Feedback content

[1874] Step 10:

[1875] The device sends the evaluation data and emotion data to the server.

[1876] The device sends the user's feedback and emotion data as an HTTP POST request to the server. Specifically, the feedback content and emotion data are compiled in JSON format and sent to the API endpoint.

[1877] Input: Feedback and emotion data

[1878] Output: HTTP POST request to send to the server

[1879] Step 11:

[1880] The server processes the feedback and emotion data

[1881] The server analyzes the received feedback and emotion data and uses a generative AI model to regenerate and modify the proposal. For example, a new office design is generated based on the modification of "adding a window to the right wall" and the emotion data of "joy."

[1882] Input: Feedback content and sentiment data extracted from HTTP POST requests

[1883] Output: Revised plans and strategies

[1884] Step 12:

[1885] The server sends the regeneration result to the terminal.

[1886] The server re-encodes the modified 3D model data and sends it to the terminal as an HTTP response.

[1887] Input: Revised plans and strategies

[1888] Output: Modified 3D model data encoded as an HTTP response

[1889] Step 13:

[1890] The device processes the re-received 3D model data

[1891] The device then decodes the re-received 3D model data and loads it back into the augmented reality display platform. The user can then review the new model and provide feedback as needed. This allows for visual evaluation of the plans and strategies generated, enabling highly accurate decisions that reflect intuitive feedback based on emotional data.

[1892] Input: Modified 3D model data encoded as an HTTP response

[1893] Output: Modified 3D model reloaded into Unity engine

[1894] (Application example 2)

[1895] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1896] Conventional evaluation systems for planning and strategy proposals obtain feedback without taking user emotions into account, which can lead to reduced accuracy of revisions and reduced user satisfaction. Furthermore, there is a lack of systems that provide intuitive evaluation methods using AR technology. Therefore, there is a need for the development of a highly accurate and user-friendly evaluation system.

[1897] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ideas generated by a generation means, means for visualizing the generated ideas using augmented reality technology, means for a user to evaluate the visualized ideas, means for acquiring emotion data, and means for analyzing the emotion data. This enables highly accurate evaluation and correction based on the user's evaluation and emotion data.

[1898] "Generation means" refers to a device or software that generates a plan or strategy proposal based on user input.

[1899] "Augmented reality technology" is a technology that overlays virtual information onto real visual information.

[1900] "Evaluation means" refers to an interface or tool that allows a user to review and evaluate the generated proposals.

[1901] "Emotion data" refers to information about the user's emotional state obtained from facial expressions, voice, body movements, etc.

[1902] The "modification means" refers to a device or software that has the function of modifying the proposal generated based on the user's evaluation and emotion data.

[1903] "Regeneration" is the process of regenerating something new based on feedback or additional information.

[1904] "Communication means" refers to the communication protocol or technology used for exchanging data between the user's terminal and the generation means.

[1905] The present invention provides a system that generates plans and strategies based on user input, visualizes the plans using augmented reality technology, and allows the user to evaluate them to propose more accurate revisions. In this system, a generation unit, an augmented reality unit, an evaluation unit, an emotion data acquisition unit, and a communication unit work in cooperation with each other.

[1906] 1. User Input

[1907] Users input design requests using a chat UI on their smartphone, for example, by entering specific instructions such as, "Please propose a layout for a new store. Make the entrance more open and create a spacious area around the cash register."

[1908] 2. Sending input data

[1909] The terminal sends the user's input text to the server using an HTTP POST request, and this data is obtained by the generating means.

[1910] 3. Use of generative AI models

[1911] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the received text data and generate a proposed store layout plan, which is used as the initial planning proposal.

[1912] 4. Use of Augmented Reality Technology

[1913] The generated proposal is converted into a 3D modeling data format (e.g., OBJ or FBX file) on the server side, and then this data is sent back to the device, where it is displayed in AR using the Unity engine.

[1914] 5. User Ratings

[1915] Users can use their smartphone's camera and microphone to view the displayed design proposals, and their evaluations and emotional data are collected. For example, users can provide feedback such as, "Please make the entrance door double-doors. Please add LED lights to the cash register."

[1916] 6. Emotion Data Analysis and Correction Suggestion Generation

[1917] The emotional data and feedback acquired by the device are sent to the server and analyzed by the emotional data acquisition means and evaluation means. The generative AI model generates revision suggestions based on this data. The revision suggestions are also visualized using AR technology, and the user evaluates them again.

[1918] 7. Final Evaluation and Regeneration

[1919] The user again provides their rating and feedback, and the server receives the data and generates the final revision proposal, thereby obtaining the optimal proposal based on the user's rating and sentiment data.

[1920] Specific examples

[1921] User input prompt:

[1922] "Please suggest a new store layout, with a more open entrance and more space around the registers."

[1923] Feedback example:

[1924] "Please make the entrance door double-doored. Please add LED lights to the cash register."

[1925] In this way, a highly accurate evaluation system that reflects user feedback and emotional data can be realized.

[1926] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1927] Step 1:

[1928] The user enters a design request using the chat UI on their smartphone and presses the send button. For example, they might enter, "Please propose a layout for our new store. Make the entrance more open and create a spacious area around the cash register." The entered data is sent from the device to the server as an HTTP POST request.

[1929] Step 2:

[1930] The server receives the HTTP POST request and analyzes the received text data using the generation method. The analyzed data is input into a generative AI model (e.g., OpenAI GPT-4), which generates an initial proposal for the store layout. The generated result is then formatted as a 3D model.

[1931] Step 3:

[1932] The server encodes the converted 3D model data (e.g., OBJ or FBX files) and sends it to the device. The device decodes the 3D model data received from the server and displays it in AR using the Unity engine.

[1933] Step 4:

[1934] The user uses the smartphone camera and screen to view the generated AR store layout, and the system captures user feedback and emotional data obtained from facial expressions and voice.

[1935] Step 5:

[1936] The device sends the text feedback entered by the user and the acquired emotion data to the server as an HTTP POST request.

[1937] Step 6:

[1938] The server analyzes the feedback and emotional data received from the user. The emotional data acquisition means analyzes the user's emotional state (e.g., joy or anxiety) and generates correction suggestions using the generative AI model.

[1939] Step 7:

[1940] The server re-encodes the modified 3D model data and sends it to the device, which decodes it again and uses the Unity engine to display the proposed modifications in AR.

[1941] Step 8:

[1942] The user then evaluates the proposed AR revisions and provides final feedback. The captured emotion data is also collected on the device.

[1943] Step 9:

[1944] The device sends the user's final feedback and emotion data to the server, which then reanalyzes the data to generate a final design, which then encodes the 3D model data of the final design and sends it to the device.

[1945] Step 10:

[1946] The device then re-decodes the received 3D model data and uses the Unity engine to display the final design in AR, allowing the user to review the design and make further evaluations and revisions as needed.

[1947] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1948] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1949] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1950] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1951] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1952] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1953] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1954] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1955] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1956] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1957] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1958] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1959] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1960] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1961] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1962] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1963] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1964] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1965] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1966] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1967] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1968] The following is further disclosed regarding the above embodiment.

[1969] (Claim 1)

[1970] A means for acquiring the proposal generated by the generating means;

[1971] means for visualizing the generated proposal using augmented reality technology;

[1972] The system includes a means for a user to evaluate the visualized proposals.

[1973] (Claim 2)

[1974] A user evaluation acquisition means for acquiring user evaluations;

[1975] a correction means for correcting the generated proposal based on the user's evaluation;

[1976] 10. The system of claim 1, further comprising means for revisualizing the revised proposal.

[1977] (Claim 3)

[1978] 10. The system of claim 1, further comprising a communication means for receiving user input from a terminal and communicating with said generating means.

[1979] (Claim 4)

[1980] 10. The system of claim 1, further comprising a format conversion means for converting data of the generated proposal into a format usable in augmented reality technology.

[1981] (Claim 5)

[1982] 10. The system of claim 1, further comprising means for using a virtual reality or augmented reality device to evaluate the generated proposals.

[1983] "Example 1"

[1984] (Claim 1)

[1985] means for receiving prompts based on user input and generating suggestions using a generative AI model;

[1986] means for converting the generated proposal into a format usable in augmented reality technology;

[1987] means for displaying the converted proposal on an augmented reality device;

[1988] a means for users to evaluate the visualized proposals;

[1989] means for sending a feedback of the proposal to a server based on said evaluation;

[1990] A system including:

[1991] (Claim 2)

[1992] A user evaluation acquisition means for acquiring user evaluations;

[1993] a correction means for correcting the generated proposal based on the user's evaluation;

[1994] 10. The system of claim 1, further comprising means for revisualizing the revised proposal.

[1995] (Claim 3)

[1996] 10. The system of claim 1, further comprising a communication means for receiving user input from a terminal and communicating with said generating means.

[1997] "Application Example 1"

[1998] (Claim 1)

[1999] A means for acquiring the proposal generated by the generating means;

[2000] a means for visualizing the generated proposal in real space using augmented reality technology;

[2001] A means for a user to evaluate the visualized proposals for optimizing the store layout;

[2002] The system includes a means for obtaining user feedback.

[2003] (Claim 2)

[2004] A user evaluation acquisition means for acquiring user evaluations;

[2005] a correction means for correcting the generated proposal based on the user's evaluation;

[2006] 10. The system of claim 1, further comprising means for visualizing the revised proposal again in real space.

[2007] (Claim 3)

[2008] 10. The system of claim 1, further comprising a communication means for receiving a user's store layout input from a terminal and communicating with said generating means.

[2009] "Example 2: Combining Emotion Engines"

[2010] (Claim 1)

[2011] A means for acquiring the proposal generated by the generating means;

[2012] means for visualizing the generated proposal using augmented reality technology;

[2013] a means for users to evaluate the visualized proposals;

[2014] A means for acquiring user emotion data;

[2015] a means for modifying the generated proposal based on the emotion data;

[2016] The system includes a means for revisualizing the revised proposal.

[2017] (Claim 2)

[2018] A user evaluation acquisition means for acquiring user evaluations;

[2019] a correction means for correcting the generated proposal based on the user's evaluation;

[2020] 10. The system of claim 1, further comprising means for revisualizing the revised proposal.

[2021] (Claim 3)

[2022] 10. The system of claim 1, further comprising a communication means for receiving user input from a terminal and communicating with said generating means.

[2023] "Application example 2 when combining emotion engines"

[2024] (Claim 1)

[2025] A means for acquiring the proposal generated by the generating means;

[2026] means for visualizing the generated proposal using augmented reality technology;

[2027] a means for users to evaluate the visualized proposals;

[2028] A means for acquiring emotion data;

[2029] means for analyzing the emotion data;

[2030] A system including:

[2031] (Claim 2)

[2032] A user evaluation acquisition means for acquiring user evaluations;

[2033] a correction means for correcting the proposal generated based on the user's evaluation and emotion data;

[2034] A means to revisit the revised proposal;

[2035] a means for the user to make a final assessment;

[2036] means for further regeneration based on said final evaluation;

[2037] 10. The system of claim 1, comprising:

[2038] (Claim 3)

[2039] 10. The system of claim 1, further comprising a communication means for receiving user input from a terminal and communicating with said generating means. [Explanation of symbols]

[2040] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring the proposal generated by the generating means; means for visualizing the generated proposal using augmented reality technology; The system includes a means for a user to evaluate the visualized proposals.

2. A user evaluation acquisition means for acquiring user evaluations; a correction means for correcting the generated proposal based on the user's evaluation; The system of claim 1 , further comprising means for revisualizing the revised proposal.

3. The system of claim 1 further comprising a communication means for receiving user input from a terminal and communicating with said generating means.

4. The system of claim 1 , further comprising a format conversion means for converting data of the generated proposal into a format usable in augmented reality technology.

5. The system of claim 1 , further comprising means for using a virtual reality or augmented reality device to evaluate the generated proposals.

Images