system

Abstract

To provide a system allowing a user to concretely image a detailed design and to realize an ideal house.SOLUTION: An input means for inputting a location desired by a user, building conditions, and a preference of a design; a modeling means for generating a 3D model of a house based on the inputted information; and a conversion means for converting the 3D model into a VR format; A system includes a VR experience means for allowing a user to experience a virtual experience, an interior input means for inputting desired information on an arrangement of interior design, furniture, and home appliances, an interior design generation means for proposing the interior design, an interior VR experience means for allowing the interior design to be virtually experienced in a VR format, a site input means for inputting actual site information, an MR projection means for virtually projecting a house on the actual site based on the input information, and a prediction means for predicting deterioration of the house and a change in lifestyle and proposing appropriate maintenance and renovation timing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, home building has been difficult for users to visualize their desired location and design. Furthermore, considering the interior and furniture layout after the home is completed is time-consuming and costly. It is also difficult to determine the appropriate timing for maintenance and renovations in response to deterioration of the home or changes in lifestyle. These challenges leave users with an unsatisfactory home building experience and prevent them from realizing their ideal home. [Means for solving the problem]

[0005] This invention provides a system that uses generative AI to generate a 3D model of a home based on the user's desired location, architectural conditions, and design preferences. The system then converts this 3D model into VR format, allowing the user to virtually experience it. The system also uses generative AI to propose interior designs based on the user's desired interior design and furniture and appliance placement, allowing the user to virtually experience the design in VR. The system combines generative AI and mixed reality technology to virtually project a home onto the actual site based on real-world site information, predicting deterioration of the home and changes in lifestyle, and recommending appropriate maintenance and renovation timing. This system allows users to visualize detailed designs and realize their ideal home.

[0006] The "input means" is an interface that allows users to input information such as their desired location, construction conditions, and design preferences into the system.

[0007] "Modeling means" is a system component for generating a 3D model of a house using a generating AI based on information input through an input means.

[0008] The "VR experience means" is an interface that converts the 3D model generated by the modeling means into a VR format, allowing the user to virtually experience the house.

[0009] The "interior input means" is an interface that allows the user to input desired information regarding the interior design and the placement of furniture and home appliances.

[0010] The "design generation means" is a system component that uses generation AI to propose interior designs based on information input through the interior input means.

[0011] The "interior VR experience means" is an interface that allows the interior design generated by the design generation means to be virtually experienced in VR format.

[0012] The "site input means" is an interface that allows the user to input actual site information (site size, location, sunlight conditions, etc.).

[0013] The "MR projection means" is a system component that combines generation AI and MR technology to virtually project a house onto a real site based on information input through the site input means.

[0014] The "prediction means" is a system component that predicts deterioration of the home and changes in the user's lifestyle, and suggests appropriate timing for maintenance and renovation. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that allows users to design and virtually experience the home they desire in the most specific way. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, and prediction means.

[0037] 1. User Information Collection Practices

[0038] First, the terminal provides an input method for the user to input their desired location, architectural conditions, and design preferences. For example, suppose a user requests a "modern three-story house in central Tokyo."

[0039] 2. Generate 3D modeling

[0040] The device sends the collected information to a server, which uses generative AI to generate a 3D model of a home based on the user's specifications. For example, a 3D model could be generated based on the user's criteria: "A three-story home with a modern design."

[0041] 3. Creating and experiencing a VR prototype

[0042] The server converts the generated 3D model into a VR format and sends it to the terminal, which then allows the user to experience the virtual house through a VR device. For example, the user can wear the VR device and virtually walk around inside and outside the building.

[0043] 4. Interior design proposals

[0044] Next, the device provides an interior design input means for users to input their desired conditions regarding interior design, furniture, and the placement of home appliances. For example, if a user requests a "natural interior," "large TV," or "open kitchen," that information is collected.

[0045] The server uses generative AI to propose detailed interior designs based on the collected information and sends the design data to the device, where the user can then use the VR device to view the virtual interior design.

[0046] 5. Projecting the site using MR technology

[0047] Users input real-world site information and send it to the server via their device. For example, if a user specifies, "I want to build on a 500-square-meter site in the suburbs," that information is used.

[0048] The server combines generative AI and MR technology to generate data for projecting a virtual house onto a real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site, allowing the user to see how the light hits it, how the shadows are, and the color scheme in a realistic way.

[0049] 6. Maintenance and renovation forecasts

[0050] Finally, the server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need to be renovated in five years.

[0051] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them to live comfortably in the home for the long term.

[0052] In this way, users can visualize the specific design in detail and virtually experience it before actually building it.The system also provides the ability to determine the appropriate timing for maintenance and renovations, allowing users to live comfortably in their home for many years to come.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[0056] Step 2:

[0057] Users input information about the home they desire (e.g., location "Minato Ward, Tokyo," architectural requirements "a three-story home with a modern design," design preferences "simple and modern," etc.) into the terminal.

[0058] Step 3:

[0059] The terminal transmits the information entered by the user to the server.

[0060] Step 4:

[0061] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[0062] Step 5:

[0063] The server converts the generated 3D model into VR format and sends the data to the terminal.

[0064] Step 6:

[0065] The terminal sends the received VR data to a VR device, allowing the user to experience the virtual home in VR.

[0066] Step 7:

[0067] The device displays an interior input form to collect information about the user's desired interior design and the placement of furniture and appliances.

[0068] Step 8:

[0069] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[0070] Step 9:

[0071] The terminal transmits the interior information input by the user to the server.

[0072] Step 10:

[0073] The server uses generative AI to generate a specific interior design based on the received information.

[0074] Step 11:

[0075] The server converts the generated interior design into VR format and sends the data to the terminal.

[0076] Step 12:

[0077] The terminal again uses a VR device to allow the user to experience a virtual interior.

[0078] Step 13:

[0079] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[0080] Step 14:

[0081] The user inputs real-world site information into the terminal.

[0082] Step 15:

[0083] The terminal transmits the site information entered by the user to the server.

[0084] Step 16:

[0085] Based on the received site information, the server combines generation AI and MR technology to generate data for virtually projecting the house onto the real site.

[0086] Step 17:

[0087] The server transmits the generated MR projection data to the terminal.

[0088] Step 18:

[0089] The terminal uses an MR device to allow the user to project a virtual home onto a real property.

[0090] Step 19:

[0091] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[0092] Step 20:

[0093] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[0094] Step 21:

[0095] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[0096] Step 22:

[0097] The server transmits the generated proposal data to the terminal.

[0098] Step 23:

[0099] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] Conventional home design systems lacked the functionality to allow users to specifically simulate and virtually experience their desired location, architectural conditions, and detailed design preferences. They also lacked the means to properly predict interior design, site projection, and the timing of long-term maintenance and renovations. This could prevent users from fully realizing their ideal home design, leading to lower satisfaction with the completed home.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a three-dimensional model of the house using a generating AI based on the information input through the input means; virtual reality experience means for converting the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generating AI based on the information input through the interior input means; interior virtual reality experience means for allowing the interior design generated by the design generation means to be virtually experienced in a virtual reality format; site input means for inputting real site information; mixed reality projection means for virtually projecting the house onto the real site by combining generating AI and mixed reality technology based on the information input through the site input means; and prediction means for predicting deterioration of the house and changes in lifestyle, and suggesting appropriate timing for maintenance and renovation.

[0105] This allows users to simulate and virtually experience the detailed design of their desired home. It also allows them to get a detailed understanding of the interior design, projection onto the site, and the timing of long-term maintenance and renovations, which increases satisfaction after the home is completed.

[0106] "Input means" refers to a device or interface that allows a user to input information such as desired location, architectural conditions, and design preferences.

[0107] "Modeling means" refers to a device or software that uses generation AI to generate a three-dimensional model of a house based on information input through the input means.

[0108] The "virtual reality experience means" refers to a device or software that converts the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience.

[0109] "Interior input means" refers to a device or interface for inputting desired information regarding the user's interior design and the placement of furniture and home appliances.

[0110] The "design generation means" refers to a device or software that uses generation AI to propose interior designs based on information input through the interior input means.

[0111] The "interior virtual reality experience means" refers to a device or software that allows the interior design generated by the design generation means to be virtually experienced in virtual reality format.

[0112] The "site input means" is a device or interface for inputting actual site information.

[0113] A "mixed reality projection means" is a device or software that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[0114] "Predictive tools" are devices or software that predict deterioration of homes and changes in lifestyles, and suggest appropriate timing for maintenance and renovations.

[0115] MODE FOR CARRYING OUT THE INVENTION

[0116] This invention is a system for allowing users to design and virtually experience the home they desire. This system generates a three-dimensional model of the home based on user input information and provides the user with a virtual experience using virtual reality and mixed reality technologies.

[0117] Hardware and software used

[0118] Hardware:

[0119] Device: The device (e.g., PC, tablet, smartphone) that collects user input and provides the virtual experience.

[0120] Server: A high-performance computer that runs generative AI models and processes data

[0121] VR device: A device that allows users to experience a virtual home (e.g., Oculus Rift, HTC Vive)

[0122] MR device: A device for providing a mixed reality experience (e.g., Microsoft® HoloLens®)

[0123] software:

[0124] Generative AI model: An AI system that generates 3D models and interior designs based on user-entered information

[0125] Modeling software: Tools for creating 3D models (e.g. Blender, 3ds Max)

[0126] VR software: Tools for converting 3D models into virtual reality formats (e.g. Unity, Unreal Engine)

[0127] MR Software: Tools for converting virtual homes into mixed reality formats

[0128] Database: A system for managing user input information and generated model data.

[0129] Program processing

[0130] 1. User Information Collection

[0131] Users use the terminal to input their desired location, architectural requirements, and design preferences. The terminal then sends this information to the server. For example, if a user wants a "modern, three-story house in central Tokyo," they would enter those detailed requirements.

[0132] 2. Generation of 3D modeling

[0133] The server uses the generative AI model to generate a 3D model of the house based on the received information. For example, a specific 3D model can be generated based on the criteria of "a three-story house with a modern design."

[0134] 3. Creating and experiencing a VR prototype

[0135] The server converts the generated 3D model into a VR format and sends it to the terminal. The terminal then allows the user to experience the virtual home through a VR device. The user puts on the VR device and can walk around inside and outside the building in the virtual space.

[0136] 4. Interior design proposals

[0137] Next, the user inputs their desired information about the interior design, furniture, and appliance placement into the device. For example, if a user requests a natural interior, a large TV, and an open kitchen, this information is sent to the server. The server then uses a generative AI model to generate a detailed interior design and sends the design data to the device. The user can then view the virtual interior design using the VR device.

[0138] 5. Projecting the site using MR technology

[0139] The user inputs real-world site information into the device and sends it to the server. For example, if a user specifies "I want to build on a 500-square-meter site in the suburbs," that information is processed by the server. The server combines a generative AI model with MR technology to generate data for projecting a virtual home onto the real site, which is then sent to the device. The device then uses an MR device to virtually project the home onto the real site, allowing the user to check the lighting, shadows, color scheme, and other aspects in real time.

[0140] 6. Maintenance and renovation forecasts

[0141] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it generates information such as "a simulation of deterioration after 10 years of construction" or "renovation of the children's room is necessary in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovation, and sends it to the device. The device notifies the user of this information, supporting a comfortable home over the long term.

[0142] Examples of concrete examples and prompts

[0143] As a concrete example, if a user requests a "two-story house with a natural design in the suburbs of Tokyo," the processing steps are as follows:

[0144] 1. The user enters desired conditions such as "suburbs of Tokyo," "natural design," and "two-story building" on the terminal.

[0145] 2. The device sends this information to a server, which uses a generative AI model to create a 3D model based on the conditions.

[0146] 3. The server converts the 3D model into a VR format and sends it to the terminal, where the user begins the virtual experience using the VR device.

[0147] 4. If the user requests "wood-style furniture," "eco-friendly appliances," or "a living room with an open ceiling space" as interior design preferences, this information will be collected from the device.

[0148] 5. The server generates a detailed interior design based on the collected information and sends it to the device, where the user can again view the virtual interior in their VR device.

[0149] 6. When the user selects "build on a 200-square-meter lot," the server uses MR technology to generate data for projecting a virtual house onto the real lot and sends it to the device. The user can then check how the light and shadows are reflected on the MR device.

[0150] 7. Finally, the server performs a "10-year deterioration simulation" and a "5-year renovation prediction" and sends the results to the terminal. The terminal then notifies the user of the results.

[0151] Example prompt sentence:

[0152] "As a user, I would like you to design a two-story house with a natural design in the suburbs of Tokyo. Specifically, I would like to use wood-style furniture and incorporate eco-friendly appliances. I would also like a living room with an open ceiling. I plan to build it on a 200-square-meter lot, and I would like to be able to use mixed reality technology to create a virtual experience. I would also like you to propose maintenance and renovations for 10 years from now."

[0153] In this way, the system allows users to imagine the design of their home in detail, experience it in a virtual space, and even create long-term maintenance plans.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1:

[0156] The user enters information

[0157] Users use the terminal to input their desired location, architectural requirements, and design preferences. This information includes the desired location, detailed architectural requirements, interior style, etc. If a user wants a "modern three-story home in central Tokyo," they input specific details.

[0158] Input: location, architectural conditions, design preferences

[0159] Output: Detailed data about the user's preferences

[0160] Step 2:

[0161] Send the collected information to a server

[0162] The device transmits the desired information collected from the user to the server, which stores this information and uses it for subsequent data processing.

[0163] Input: User's desired information

[0164] Output: User preference data stored on the server

[0165] Step 3:

[0166] The server generates the 3D model

[0167] The server uses the generative AI model to generate a 3D model of the house based on the information requested by the user. In this process, the AI ​​model generates a design based on the input information and creates 3D model data.

[0168] Input: User preference data

[0169] Data processing: Generating 3D models using generative AI models

[0170] Output: 3D model data

[0171] Step 4:

[0172] Converting 3D models into VR format

[0173] The server converts the generated 3D model data into VR format, allowing users to experience the home in a virtual space.

[0174] Input: 3D model data

[0175] Data processing: Converting 3D models into VR format

[0176] Output: VR format housing data

[0177] Step 5:

[0178] The user experiences VR

[0179] The terminal transmits the VR format house data received from the server to the VR device, providing the user with a virtual experience. The user wears the VR device and visually experiences the inside and outside of the virtual house.

[0180] Input: VR format housing data

[0181] Output: User's virtual experience information

[0182] Step 6:

[0183] Enter your interior design preferences

[0184] Users input their preferences for interior design, furniture, and appliance placement into the terminal, such as "natural interior," "large TV," and "open kitchen."

[0185] Input: Interior design, furniture, and home appliance preferences

[0186] Output: User's interior design preferences data

[0187] Step 7:

[0188] The collected interior preference information is sent to the server.

[0189] The device sends the interior design preferences collected from the user to the server, which receives the information and uses it for subsequent data processing.

[0190] Input: User's interior design preferences

[0191] Output: Interior preference data saved on the server

[0192] Step 8:

[0193] The server generates the interior design

[0194] The server generates a detailed interior design using a generative AI model based on the received interior design preferences.

[0195] Input: Interior design preferences data

[0196] Data processing: Generating interior designs using generative AI models

[0197] Output: Interior design data

[0198] Step 9:

[0199] Convert interior designs into VR format and provide them to users

[0200] The server converts the generated interior design data into VR format and sends it to the terminal, which then uses a VR device to provide the user with a virtual interior experience.

[0201] Input: Interior design data

[0202] Data processing: Converting interior design into VR format

[0203] Output: VR format interior data

[0204] Step 10:

[0205] Users input real site information

[0206] The user enters information about the lot on which they want to build a house into the terminal, such as "a 500-square-meter lot in the suburbs."

[0207] Input: Site information

[0208] Output: User's site information data

[0209] Step 11:

[0210] Send the collected site information to the server

[0211] The terminal sends the site information collected from the user to the server, which uses this information to perform the following processes:

[0212] Input: User's site information

[0213] Output: Site information data stored on the server

[0214] Step 12:

[0215] The server generates data to project the virtual house onto the site.

[0216] The server combines site information with a generative AI model and MR technology to generate data that projects a virtual home onto the real site.

[0217] Input: Site information data

[0218] Data processing: Generating residential projection data using generative AI and MR technology

[0219] Output: Virtual house projection data in MR format

[0220] Step 13:

[0221] Projecting a virtual house onto a real site through an MR device

[0222] The server then sends the generated MR-formatted virtual house projection data to the terminal, which then uses the MR device to virtually project the house onto the user's screen.The user can then use the MR device to check the location of the house, how the light hits it, and the shadows.

[0223] Input: MR format virtual house projection data

[0224] Output: User's projection confirmation information

[0225] Step 14:

[0226] The server predicts the deterioration and renovation of the home.

[0227] The server processes data to predict deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction and predicting renovation needs after five years.

[0228] Input: Housing information, user lifestyle information

[0229] Data processing: Deterioration and renovation prediction using generative AI

[0230] Output: Deterioration simulation data, renovation prediction data

[0231] Step 15:

[0232] Notifying users when maintenance or renovation is required

[0233] The server sends the generated simulation data and prediction data to the terminal, which then notifies the user of appropriate maintenance and renovation timing.

[0234] Input: Deterioration simulation data, renovation prediction data

[0235] Output: Notification information for the user

[0236] In this way, the present invention provides a system that allows users to design a home in the most specific way, virtually experience it, and even plan maintenance and renovations.

[0237] (Application example 1)

[0238] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] In home and interior design, users often have difficulty experiencing and confirming their preferences in a concrete and intuitive way. In particular, in physical stores, there are limited ways to check the actual layout and design of interior products before purchasing them, which can lead to lower post-purchase satisfaction. Furthermore, when users want to make revisions to detailed interior designs while experiencing them virtually, the process is cumbersome and they are unable to receive timely feedback.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0241] In this invention, the server includes: an input means for inputting a user's desired location, architectural conditions, and design preferences; a modeling means for generating a 3D model of a house using a generative AI based on the information input through the input means; a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience; an interior input means for inputting the user's desired information regarding the interior design and layout of furniture and home appliances; a design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; an interior VR experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; a site input means for inputting information about a real site; an MR projection means for virtually projecting a house onto the real site based on the information input through the site input means by combining the generative AI and MR technology; and an experience means for allowing the user to wear smart glasses or a head-mounted display at an interior shop and experience the interior in both real and virtual spaces. This allows the user to virtually experience interior products at the interior shop and intuitively and concretely check the layout and design.

[0242] "Input means" refers to the means by which a user inputs desired location, architectural conditions, and design preferences into a terminal.

[0243] The "modeling means" is a means for generating a 3D model of a house using a generation AI based on information input through the input means.

[0244] The "VR experience means" is a means for converting the 3D model generated by the modeling means into a VR format, allowing the user to have a virtual experience.

[0245] The "interior input means" is a means for inputting desired information regarding the user's interior design and the placement of furniture and home appliances into the terminal.

[0246] The "design generation means" is a means for proposing an interior design using a generation AI based on the information input through the interior input means.

[0247] The "interior VR experience means" is a means that allows the interior design generated by the design generation means to be virtually experienced in VR format.

[0248] The "site input means" is a means for inputting actual site information into the terminal.

[0249] The "MR projection means" is a means for virtually projecting a house onto a real site by combining generation AI and MR technology based on information input through the site input means.

[0250] The "experience means" refers to a means by which users can wear smart glasses or a head-mounted display in an interior design shop and experience the interior design in both real and virtual spaces.

[0251] A "prediction method" is a means of predicting when a home will deteriorate or require maintenance based on the user's lifestyle and environmental conditions.

[0252] The system for carrying out the present invention is composed of multiple components that allow a user to virtually experience the home design and interior design they desire. The system includes the following elements:

[0253] First, the user terminal provides an input means for the user to input their desired location, architectural conditions, and design preferences. The input means includes an interface through which the user can input their preferences, such as "a three-story house with a modern design in central Tokyo."

[0254] The server then uses the collected information to generate a 3D model of the house using a generative AI model. During the generation process, for example, the server sends the following prompt to the generative AI:

[0255] "Create a 3D model of a modern, three-story house in central Tokyo."

[0256] The server converts the generated 3D model into VR format and sends it to the user's device, which then allows the user to experience the virtual home through a VR device. This process uses software such as OpenVR and PyOpenGL.

[0257] Next, an interior design input interface is provided for users to input their preferences regarding interior design, furniture, and appliance placement. Users can use this interface to input preferences such as "natural interior," "large TV," and "open kitchen."

[0258] The server then uses the collected interior information to propose a detailed interior design using generative AI again. This time, prompts are used as well. For example:

[0259] "Create a 3D model of a living room with natural decor, a large TV, and an open kitchen."

[0260] The generated interior design is then converted back into VR format and sent to the user's device, where the user can view the virtual interior design through the interior VR experience.

[0261] Additionally, users can use the site entry tool to input real-world site information, for example, a "500 square meter site in the suburbs."

[0262] Based on this information, the server combines generative AI and MR technology to generate data for projecting a virtual house onto the real site. This MR projection method allows users to use an MR device to virtually project the house onto the real site, allowing them to realistically check how the light hits it, the shadows, the color scheme, and more.

[0263] The server also uses predictive tools to forecast deterioration of the home and changes in lifestyle. For example, predictions include simulations of deterioration after 10 years of construction and the need to renovate the children's room in five years.

[0264] Finally, an experience tool is provided to users in the interior design shop's physical store. This tool allows users to wear smart glasses or a head-mounted display and experience interior design products in both real and virtual spaces. This allows users to check the actual layout and design of the products before purchasing them.

[0265] As described above, this system connects multiple components to enable users to have a concrete and intuitive virtual experience of home and interior design, ultimately helping them make the best choice.

[0266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0267] Step 1:

[0268] An input method is used to input the user's desired location, architectural conditions, and design preferences. In this step, a user terminal is used, and the user inputs information such as "a three-story house with a modern design in central Tokyo" through the interface. The input information is converted into a data format within the terminal and sent to the server.

[0269] Step 2:

[0270] The server receives the information entered by the user. It then uses the generative AI model to generate a 3D model of the house. In this step, it sends the prompt "Please create a 3D model of a modern, three-story house located in central Tokyo" to the generative AI. It then receives the 3D model data returned by the generative AI and converts it into a VR format.

[0271] Step 3:

[0272] The server sends the generated 3D model data in VR format to the user's terminal. The user's terminal uses a VR device (e.g., a head-mounted display) to provide the user with a virtual home experience. The user can wear the VR device and virtually walk around inside and outside the building.

[0273] Step 4:

[0274] The user uses the interior input means to input desired information regarding the interior design, furniture, and placement of home appliances. In this step, the user inputs the desired interior style and specific furniture items, which are then sent to the server by the terminal. For example, information such as "natural interior," "large TV," and "open kitchen" can be input.

[0275] Step 5:

[0276] The server uses the generative AI model to propose a detailed interior design based on the information input through the interior input means. At this time, it sends a prompt to the generative AI: "Please create a 3D model of a living room with natural interior, a large TV, and an open kitchen." The interior design data returned by the generative AI is converted into VR format.

[0277] Step 6:

[0278] The server transmits the generated VR data of the interior design to the user terminal. The user terminal provides the user with a virtual interior experience using the interior VR experience means. The user can then use the VR device again to virtually check the interior design and walk around inside the house.

[0279] Step 7:

[0280] The user uses a site input means to input information about a real site. For example, the user inputs information about a "500 square meter site in the suburbs" and transmits it to the server via the terminal.

[0281] Step 8:

[0282] Based on the information entered through the site input means, the server combines AI generation and MR technology to generate data for virtually projecting the house onto the real site. In doing so, it generates detailed data such as lighting, shadows, and color schemes so that the virtual house is properly projected onto the real site.

[0283] Step 9:

[0284] The server transmits the generated MR data to the user's device, which then projects the virtual house onto the real site, allowing the user to see how the house will look in the real environment.

[0285] Step 10:

[0286] The server predicts deterioration of the home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation. Based on this prediction method, it generates simulation results, such as "simulation of deterioration after 10 years of construction" or "children's room renovations are necessary in 5 years," and sends these to the user's device.

[0287] Step 11:

[0288] Users can wear smart glasses or a head-mounted display in an interior design shop to experience the interior design in both real and virtual space. This experience allows users to check the actual layout and design of the product before purchasing it, allowing them to make the best choice.

[0289] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0290] This invention is a system for designing the home desired by the user in the most specific way and providing a virtual experience, and by combining it with an emotion engine, the system's suggestions and display content can be adjusted according to the user's emotional state. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, prediction means, and the emotion engine.

[0291] 1. User Information Collection Practices

[0292] The terminal displays a data entry form for users to enter their desired location, construction conditions, and design preferences. Users enter information such as location (Central Tokyo), construction conditions (Modern design, three-story house), and design preference (Simple and modern).

[0293] 2. Generate 3D modeling

[0294] The device sends the information entered by the user to a server, which then uses generative AI to generate a 3D model of the home that meets the user's needs.

[0295] 3. Creating and experiencing a VR prototype

[0296] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[0297] 4. Interior design proposals

[0298] The terminal displays an interior input means for users to input their desired conditions regarding interior design, furniture, and appliance placement, and users can enter information such as "natural-style interior," "large TV," and "open kitchen."

[0299] The server uses generative AI to generate a detailed interior design based on the collected information and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[0300] 5. Projecting the site using MR technology

[0301] The user inputs real-world site information and sends it to the server via their device. For example, the user might specify, "I want to build on a 500-square-meter site in the suburbs."

[0302] Based on the received site information, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the terminal. The terminal then uses an MR device to virtually project the house onto the real site, allowing the user to realistically check how the light hits it, the shadows, color scheme, and more.

[0303] 6. Maintenance and renovation forecasts

[0304] The server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need renovation in five years.

[0305] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them maintain a comfortable home over the long term.

[0306] 7. Adjusting recommendations with an emotion engine

[0307] While the user is viewing a virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied.

[0308] The server adjusts its suggestions and displays based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server can quickly generate alternatives and suggest new interior designs or architectural conditions to the user.

[0309] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[0310] The processing flow will be explained below.

[0311] Step 1:

[0312] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[0313] Step 2:

[0314] Users input information about the home they desire (e.g., location "central Tokyo," construction requirements "a three-story home with a modern design," design preferences "simple and modern") into the terminal.

[0315] Step 3:

[0316] The terminal transmits the information entered by the user to the server.

[0317] Step 4:

[0318] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[0319] Step 5:

[0320] The server converts the generated 3D model into VR format and sends the data to the terminal.

[0321] Step 6:

[0322] The terminal prepares the VR device and guides the user to put it on.

[0323] Step 7:

[0324] Users put on a VR device and experience a virtual home.

[0325] Step 8:

[0326] The terminal displays an interior input form that allows the user to enter information about the desired interior design and the placement of furniture and appliances.

[0327] Step 9:

[0328] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[0329] Step 10:

[0330] The terminal transmits the interior information input by the user to the server.

[0331] Step 11:

[0332] The server uses generative AI to generate a specific interior design based on the received information.

[0333] Step 12:

[0334] The server converts the generated interior design into VR format and sends the data to the terminal.

[0335] Step 13:

[0336] The terminal again uses a VR device to allow the user to experience a virtual interior.

[0337] Step 14:

[0338] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[0339] Step 15:

[0340] The user inputs real-world site information into the terminal.

[0341] Step 16:

[0342] The terminal transmits the site information entered by the user to the server.

[0343] Step 17:

[0344] Based on the site information it receives, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site.

[0345] Step 18:

[0346] The server transmits the generated MR projection data to the terminal.

[0347] Step 19:

[0348] The terminal uses an MR device to allow the user to virtually project a house onto a real-world site.

[0349] Step 20:

[0350] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[0351] Step 21:

[0352] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[0353] Step 22:

[0354] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[0355] Step 23:

[0356] The server transmits the generated proposal data to the terminal.

[0357] Step 24:

[0358] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[0359] Step 25:

[0360] The device uses an emotion engine to analyze the user's emotional state in real time while they are experiencing the virtual world.

[0361] Step 26:

[0362] The emotion engine determines the user's emotional state based on biometric information such as facial expressions, voice, and gestures.

[0363] Step 27:

[0364] The server adjusts the suggestions and display content based on feedback from the emotion engine.

[0365] Step 28:

[0366] If the emotion engine detects a user's dissatisfaction, the server immediately generates an alternative and sends it to the device.

[0367] Step 29:

[0368] The device presents alternatives to the user and suggests new interior designs and architectural requirements.

[0369] Example 2

[0370] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] Conventional home design systems were unable to fully reflect the user's wishes and emotions, and virtual and interior experiences were limited. It was also difficult for users to predict when future renovations or maintenance would be required. There was also no real-time proposal adjustment that took the user's emotions into account, which led to problems with the quality of the user experience.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0373] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a 3D model of the house using a generative AI based on the information input through the input means; virtual experience means for converting the 3D model generated by the modeling means into a VR format to allow the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; interior virtual experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; site input means for inputting information about the actual site; mixed reality projection means for virtually projecting the house onto the actual site based on the information input through the site input means by combining generative AI and mixed reality technology; prediction means for predicting deterioration of the house and changes in lifestyle and proposing appropriate timing for maintenance and improvements; and an emotion engine for analyzing the user's emotional state in real time and adjusting the content of the proposals. This will enable specific home designs that reflect the user's wishes and emotions, improving the quality of virtual and interior experiences, and even making it possible to accurately predict the timing of future renovations and maintenance.

[0374] "User" refers to an individual who uses this system to receive home design and interior design proposals and have a virtual experience.

[0375] "Input means" refers to the interface that allows users to input their desired location, architectural conditions, and design preferences.

[0376] "Modeling means" refers to the function of generating a 3D model of a house using generative AI based on information entered by the user.

[0377] "Virtual experience means" refers to the function of converting the generated 3D model into VR format, allowing users to experience the house in a virtual space.

[0378] "Interior input means" refers to an interface that allows users to input desired information regarding interior design and the placement of furniture and home appliances.

[0379] "Design generation means" refers to a function that uses generation AI to propose interior designs based on information input through the interior input means.

[0380] "Interior virtual experience means" refers to a function that provides users with the generated interior design in VR format, allowing them to experience it in a virtual space.

[0381] "Site input means" refers to an interface for a user to input real site information.

[0382] "Mixed reality projection means" refers to a function that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[0383] "Predictive tools" refers to functions that predict deterioration of the home and changes in lifestyle, and suggest appropriate timing for maintenance and renovations.

[0384] The "emotion engine" refers to a function that analyzes the user's emotional state in real time and adjusts the content of suggestions.

[0385] MODE FOR CARRYING OUT THE INVENTION

[0386] An embodiment of this invention is a system that specifically designs a home desired by a user, provides a virtual experience, and further combines an emotion engine to make suggestions based on the user's emotions. This system is composed of the following main components: input means, modeling means, virtual experience means, interior input means, design generation means, interior virtual experience means, site input means, mixed reality projection means, prediction means, and emotion engine.

[0387] Collection of User Information

[0388] The terminal displays a data entry form for users to enter their desired location, architectural conditions, and design preferences. For example, users might enter information such as "Central Tokyo," "Modern design three-story home," and "Simple and modern."

[0389] 3D modeling generation

[0390] The device sends the information entered by the user to the server. The server generates a 3D model of the house using a generative AI model based on the received information. For example, GPT-4 (registered trademark) can be used as the generative AI model. The following prompt sentence is used as input:

[0391] User Information:

[0392] Location: Central Tokyo

[0393] Building requirements: Modern design, three-storey house

[0394] Design preference: Simple and modern

[0395] Generate a 3D model of the house according to the user's wishes.

[0396] Creating and experiencing VR prototypes

[0397] The server converts the generated 3D model into a VR format and sends the data to the terminal, which then provides the user with a virtual experience through a VR device such as Oculus Rift. The user can then use the VR device to virtually explore the inside and outside of the building.

[0398] Interior design proposals

[0399] The device displays an interior input tool for the user to input their desired information regarding interior design, furniture, and appliance placement, prompting them to enter information such as "natural-style interior," "large TV," and "open kitchen." The server uses generative AI based on the collected information to generate a detailed interior design and sends the design data to the device. The user then uses the VR device again to view the virtual interior design.

[0400] Projection of the site using MR technology

[0401] The user inputs real-world site information and sends it to the server via their device. For example, they might specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines generative AI and mixed reality technology to generate data for virtually projecting the house onto the real site, and sends that data to the device. The device then uses an MR device such as Microsoft HoloLens to virtually project the house onto the real site.

[0402] Maintenance and renovation forecasts

[0403] The server predicts deterioration of the home and changes in the user's lifestyle. Specifically, it makes predictions such as "simulating deterioration after 10 years of construction" and "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term.

[0404] Tailoring suggestions with an emotion engine

[0405] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions and voice to determine whether the user is satisfied or dissatisfied. The server then adjusts the proposal based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server will quickly generate alternative proposals and suggest new interior designs or architectural conditions to the user.

[0406] In this way, the system combined with the emotion engine provides users with a more personalized experience, enabling optimal home design based on their emotions, greatly improving user satisfaction and further facilitating the realization of their ideal home.

[0407] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0408] Step 1:

[0409] Users use a data input form on their device to input their desired location, architectural requirements, and design preferences. The input is in the form of "Central Tokyo," "Modern design three-story house," "Simple and modern," etc. The input data is sent to the server in JSON format.

[0410] Step 2:

[0411] The server parses the received JSON formatted user information and generates a prompt to be input to the generative AI model. The generated prompt is in the following text format:

[0412] User Information:

[0413] Location: Central Tokyo

[0414] Building requirements: Modern design, three-storey house

[0415] Design preference: Simple and modern

[0416] Generate a 3D model of the house according to the user's wishes.

[0417] Based on this prompt, the server sends the data to a generative AI model (e.g., GPT-4) to generate a 3D model.

[0418] Step 3:

[0419] The server receives the 3D model data output by the generative AI model and converts it into a VR format (e.g., Unity format), which allows users to experience the data in a virtual space.

[0420] Step 4:

[0421] The server sends the converted 3D model data to the device via an HTTP POST request, and the device then provides the user with a virtual experience through a VR device such as an Oculus Rift. The user can explore the building's interior and exterior in the virtual space.

[0422] Step 5:

[0423] The user uses the device interface to input desired interior design information, such as "natural-style interior," "large TV," and "open kitchen." The input data is sent to the server in JSON format.

[0424] Step 6:

[0425] The server generates a detailed interior design using a generative AI model based on the received interior information. The generated interior design data is then converted back to JSON format and sent to the device.

[0426] Step 7:

[0427] The terminal then provides the received interior design data to the user via a VR device, enabling them to check the interior in a virtual space. The user puts on the VR device and visually checks the interior layout and design in the virtual space.

[0428] Step 8:

[0429] The user inputs real-world site information using a terminal and sends it to the server. For example, a "500 square meter site in the suburbs" is specified. The input data is sent in JSON format.

[0430] Step 9:

[0431] Based on the received site information, the server combines generative AI and mixed reality technology to generate data that virtually projects the house onto the real site. The generated MR data is then sent to the device.

[0432] Step 10:

[0433] The device then provides the received MR data to the user via a mixed reality device such as Microsoft HoloLens, projecting the virtual home onto the real site. By wearing the MR device, the user can visually confirm the layout and appearance of the home on the real site.

[0434] Step 11:

[0435] The server runs an algorithm that predicts deterioration of the home and changes in lifestyle, generating predictive data such as "simulation of deterioration after 10 years of construction" or "need to renovate the children's room in five years." The generated predictive data is sent to the device.

[0436] Step 12:

[0437] The device then notifies the user of the received forecast data and suggests appropriate maintenance and renovation timing, allowing the user to create a maintenance plan to maintain a comfortable home for the long term.

[0438] Step 13:

[0439] While the user is viewing the virtual experience or interior design, the device uses a camera and microphone to capture the user's facial expressions and voice, inputting the data into an emotion engine, which analyzes the user's emotional state in real time and sends feedback to the server.

[0440] Step 14:

[0441] The server adjusts the proposals based on the user's emotions based on feedback from the emotion engine. If the user expresses dissatisfaction, the server quickly generates alternative proposals and suggests new interior designs or architectural conditions to the user, thereby improving user satisfaction.

[0442] (Application example 2)

[0443] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0444] Conventional home design systems have difficulty providing personalized proposals based on the user's emotions and preferences. Furthermore, they lack the functionality to adjust proposals based on the user's emotional responses when viewing homes and interiors in real time through virtual experiences. This has hindered user satisfaction and purchase intent.

[0445] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for inputting the user's desired location, architectural conditions, and design preferences, a modeling means for generating a 3D model of a house using a generation AI based on the information input through the input means, a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience, an interior input means for inputting the user's desired information regarding the interior design and the layout of furniture and home appliances, a design generation means for proposing an interior design using a generation AI based on the information input through the interior input means, and a virtual object in VR format for the interior design generated by the design generation means. The system includes an interior VR experience means for allowing users to experience a virtual home on a real site, a site input means for inputting real site information, an MR projection means for virtually projecting a home onto the real site by combining generative AI and MR technology based on the information input through the site input means, a prediction means for predicting deterioration of the home and changes in lifestyle and proposing appropriate timing for maintenance and renovation, an emotion engine means for analyzing the user's emotional state in real time and adjusting the content of proposals and displays, a biometric information analysis means for photographing the user's face during the virtual experience and analyzing emotions in real time, and a proposal adjustment means for customizing interior design and product proposals according to the user's emotional state. This makes it possible to provide optimal home design and interior proposals taking into account the user's emotional state.

[0446] "Input means" refers to a device or system that allows a user to input information such as desired location, architectural conditions, and design preferences.

[0447] A "modeling means" is a system or algorithm that uses generative AI to generate a 3D model of a house based on input information.

[0448] The "VR experience means" is a system that converts the generated 3D model into VR format, allowing users to view the house through a virtual experience.

[0449] "Interior input means" refers to a device or system for inputting information regarding the user's desired interior design and the layout of furniture and home appliances.

[0450] The "design generation means" is a system or algorithm that uses generative AI to propose interior designs based on information input through the interior input means.

[0451] The "Interior VR Experience Method" is a system that allows users to virtually experience the generated interior design in VR format.

[0452] "Site input means" refers to a device or system that allows a user to input actual site information.

[0453] The "MR projection means" is a system or algorithm that combines generative AI and MR technology to virtually project a house onto a real site based on information input through the site input means.

[0454] The "prediction method" is a system that predicts deterioration of a home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation.

[0455] An "emotion engine means" is a system or algorithm that analyzes a user's emotional state in real time and adjusts the content of suggestions and displays.

[0456] "Biometric information analysis means" refers to a system or device that photographs the user's face during a virtual experience and analyzes emotions in real time.

[0457] A "proposal tailoring means" is a system or algorithm that customizes interior design and product proposals based on the user's emotional state.

[0458] This invention is a system for specifically designing a home desired by a user and providing a virtual experience of the design. This system has the function of analyzing the user's emotional state and adjusting the proposed content. The detailed configuration and processing method of the system are explained below.

[0459] System configuration and hardware / software usage

[0460] This system mainly uses the following hardware and software:

[0461] Hardware:

[0462] Smartphone

[0463] VR devices (e.g., Oculus Rift, HTC Vive, etc.)

[0464] Mixed reality devices (e.g. Microsoft HoloLens)

[0465] Camera (captures the user's facial expression)

[0466] software:

[0467] Python

[0468] OpenCV (real-time image processing and facial expression analysis)

[0469] EmotionRecognizer (emotion recognition library)

[0470] AI model generation library

[0471] VRHandler (VR processing)

[0472] MRHandler (MR processing)

[0473] System processing overview

[0474] 1. Collection of User Information

[0475] The device (smartphone) displays a data entry form for the user to enter their desired location, architectural requirements, and design preferences. For example, the user might enter "Central Tokyo" as the location information, "Modern three-story house" as the architectural requirement, and "Simple and modern" as the design preference.

[0476] 2. Information transmission and 3D modeling

[0477] The device sends the information entered by the user to a server, which uses the received information to generate a 3D model of the home that meets the user's requirements using a generative AI model.

[0478] 3. Creating and experiencing a VR prototype

[0479] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[0480] 4. Interior design proposals

[0481] The terminal displays an interior design input tool that allows users to input their desired conditions regarding interior design, furniture, and appliance placement, such as "natural-style interior," "large TV," and "open kitchen."

[0482] The server uses the collected information to generate a detailed interior design using a generative AI model, and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[0483] 5. Input of site information and projection of the site using MR technology

[0484] The user inputs real-world site information and sends it to the server via their device. For example, the user may specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines a generative AI model and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site.

[0485] 6. Maintenance and renovation forecasts

[0486] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it makes predictions such as "simulating deterioration after 10 years of construction" or "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term.

[0487] 7. Adjusting recommendations with an emotion engine

[0488] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied. The server adjusts the suggestions and display content based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server immediately generates alternatives and suggests new interior designs or architectural conditions to the user.

[0489] Specific examples

[0490] 1. The user enters their preferences into the app (e.g., modern, minimalist style, gray sofa).

[0491] 2. The app identifies the user's facial expression (e.g., the user is smiling).

[0492] 3. The app uses generative AI to suggest furniture that suits the user's preferences and generate a 3D model of it.

[0493] 4. Check the furniture layout in a virtual store using VR or MR devices.

[0494] Prompt Sentence Examples

[0495] An example of a prompt to be fed to a generative AI model:

[0496] Generate interior designs based on user preferences, such as:

[0497] Style: Modern, Simple

[0498] Furniture: Gray sofa, white table

[0499] If the user is smiling, suggest light colors and soft materials.

[0500] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[0501] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0502] Step 1:

[0503] The terminal displays a data input form for the user to input their desired location, architectural conditions, and design preferences. The user enters information such as "Central Tokyo," "Modern three-story house," and "Simple and modern" into the input form. This becomes the input data.

[0504] Step 2:

[0505] The device sends the information entered by the user to the server, which uses the generative AI model to generate a 3D model of the home that meets the user's requirements. This 3D model becomes the output data.

[0506] Step 3:

[0507] The server converts the generated 3D model into VR format and sends the data to the terminal. The terminal allows the user to experience the virtual house through a VR device. The user puts on the VR device and virtually walks around inside and outside the building. The 3D model in VR format becomes the output data.

[0508] Step 4:

[0509] The terminal displays an interior design input tool for inputting desired information regarding interior design, furniture, and appliance placement. Users input information such as "natural-style interior," "large-screen TV," and "open kitchen." This becomes the input data for the interior information.

[0510] Step 5:

[0511] The device sends the interior information entered by the user to the server, which then uses a generative AI model based on the collected interior information to generate a detailed interior design. This interior design becomes the output data.

[0512] Step 6:

[0513] The server converts the generated interior design into VR format and sends the data to the terminal. The terminal then uses a VR device to allow the user to view the virtual interior design. The VR-format interior design becomes the output data.

[0514] Step 7:

[0515] The user inputs information about the actual site and sends it to the server via their terminal. For example, they might input information like, "I want to build on a 500 square meter site in the suburbs." This becomes the input data for the site information.

[0516] Step 8:

[0517] Based on the received site information, the server combines the generative AI model and MR technology to generate data for virtually projecting the house onto the real site, and sends that data to the terminal. The terminal then uses an MR device to allow the user to virtually project the house onto the real site. The MR format data is the output data.

[0518] Step 9:

[0519] The server predicts the deterioration of the home and changes in the user's lifestyle, and generates data suggesting the appropriate timing and content of maintenance and renovations, which it then sends to the device. For example, suggestions include "simulation of deterioration after 10 years of construction" and "renovation of the children's room is necessary in 5 years." This is the output data.

[0520] Step 10:

[0521] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied. The result is output data.

[0522] Step 11:

[0523] Based on the analysis of the user's emotional state, the server adjusts the suggestions and display content. For example, if the user shows a dissatisfied expression, the server will immediately generate alternative suggestions and suggest new interior designs or architectural conditions to the user. This is the output data of the adjusted suggestions.

[0524] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0525] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0526] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0527] [Second embodiment]

[0528] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0529] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0530] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0531] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0532] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0533] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0534] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0535] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0536] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0537] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0538] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0539] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0540] This invention is a system that allows users to design and virtually experience the home they desire in the most specific way. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, and prediction means.

[0541] 1. User Information Collection Practices

[0542] First, the terminal provides an input method for the user to input their desired location, architectural conditions, and design preferences. For example, suppose a user requests a "modern three-story house in central Tokyo."

[0543] 2. Generate 3D modeling

[0544] The device sends the collected information to a server, which uses generative AI to generate a 3D model of a home based on the user's specifications. For example, a 3D model could be generated based on the user's criteria: "A three-story home with a modern design."

[0545] 3. Creating and experiencing a VR prototype

[0546] The server converts the generated 3D model into a VR format and sends it to the terminal, which then allows the user to experience the virtual house through a VR device. For example, the user can wear the VR device and virtually walk around inside and outside the building.

[0547] 4. Interior design proposals

[0548] Next, the device provides an interior design input means for users to input their desired conditions regarding interior design, furniture, and the placement of home appliances. For example, if a user requests a "natural interior," "large TV," or "open kitchen," that information is collected.

[0549] The server uses generative AI to propose detailed interior designs based on the collected information and sends the design data to the device, where the user can then use the VR device to view the virtual interior design.

[0550] 5. Projecting the site using MR technology

[0551] Users input real-world site information and send it to the server via their device. For example, if a user specifies, "I want to build on a 500-square-meter site in the suburbs," that information is used.

[0552] The server combines generative AI and MR technology to generate data for projecting a virtual house onto a real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site, allowing the user to see how the light hits it, how the shadows are, and the color scheme in a realistic way.

[0553] 6. Maintenance and renovation forecasts

[0554] Finally, the server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need to be renovated in five years.

[0555] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them to live comfortably in the home for the long term.

[0556] In this way, users can visualize the specific design in detail and virtually experience it before actually building it.The system also provides the ability to determine the appropriate timing for maintenance and renovations, allowing users to live comfortably in their home for many years to come.

[0557] The processing flow will be explained below.

[0558] Step 1:

[0559] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[0560] Step 2:

[0561] Users input information about the home they desire (e.g., location "Minato Ward, Tokyo," architectural requirements "a three-story home with a modern design," design preferences "simple and modern," etc.) into the terminal.

[0562] Step 3:

[0563] The terminal transmits the information entered by the user to the server.

[0564] Step 4:

[0565] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[0566] Step 5:

[0567] The server converts the generated 3D model into VR format and sends the data to the terminal.

[0568] Step 6:

[0569] The terminal sends the received VR data to a VR device, allowing the user to experience the virtual home in VR.

[0570] Step 7:

[0571] The device displays an interior input form to collect information about the user's desired interior design and the placement of furniture and appliances.

[0572] Step 8:

[0573] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[0574] Step 9:

[0575] The terminal transmits the interior information input by the user to the server.

[0576] Step 10:

[0577] The server uses generative AI to generate a specific interior design based on the received information.

[0578] Step 11:

[0579] The server converts the generated interior design into VR format and sends the data to the terminal.

[0580] Step 12:

[0581] The terminal again uses a VR device to allow the user to experience a virtual interior.

[0582] Step 13:

[0583] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[0584] Step 14:

[0585] The user inputs real-world site information into the terminal.

[0586] Step 15:

[0587] The terminal transmits the site information entered by the user to the server.

[0588] Step 16:

[0589] Based on the received site information, the server combines generation AI and MR technology to generate data for virtually projecting the house onto the real site.

[0590] Step 17:

[0591] The server transmits the generated MR projection data to the terminal.

[0592] Step 18:

[0593] The terminal uses an MR device to allow the user to project a virtual home onto a real property.

[0594] Step 19:

[0595] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[0596] Step 20:

[0597] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[0598] Step 21:

[0599] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[0600] Step 22:

[0601] The server transmits the generated proposal data to the terminal.

[0602] Step 23:

[0603] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[0604] Example 1

[0605] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0606] Conventional home design systems lacked the functionality to allow users to specifically simulate and virtually experience their desired location, architectural conditions, and detailed design preferences. They also lacked the means to properly predict interior design, site projection, and the timing of long-term maintenance and renovations. This could prevent users from fully realizing their ideal home design, leading to lower satisfaction with the completed home.

[0607] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0608] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a three-dimensional model of the house using a generating AI based on the information input through the input means; virtual reality experience means for converting the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generating AI based on the information input through the interior input means; interior virtual reality experience means for allowing the interior design generated by the design generation means to be virtually experienced in a virtual reality format; site input means for inputting real site information; mixed reality projection means for virtually projecting the house onto the real site by combining generating AI and mixed reality technology based on the information input through the site input means; and prediction means for predicting deterioration of the house and changes in lifestyle, and suggesting appropriate timing for maintenance and renovation.

[0609] This allows users to simulate and virtually experience the detailed design of their desired home. It also allows them to get a detailed understanding of the interior design, projection onto the site, and the timing of long-term maintenance and renovations, which increases satisfaction after the home is completed.

[0610] "Input means" refers to a device or interface that allows a user to input information such as desired location, architectural conditions, and design preferences.

[0611] "Modeling means" refers to a device or software that uses generation AI to generate a three-dimensional model of a house based on information input through the input means.

[0612] The "virtual reality experience means" refers to a device or software that converts the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience.

[0613] "Interior input means" refers to a device or interface for inputting desired information regarding the user's interior design and the placement of furniture and home appliances.

[0614] The "design generation means" refers to a device or software that uses generation AI to propose interior designs based on information input through the interior input means.

[0615] The "interior virtual reality experience means" refers to a device or software that allows the interior design generated by the design generation means to be virtually experienced in virtual reality format.

[0616] The "site input means" is a device or interface for inputting actual site information.

[0617] A "mixed reality projection means" is a device or software that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[0618] "Predictive tools" are devices or software that predict deterioration of homes and changes in lifestyles, and suggest appropriate timing for maintenance and renovations.

[0619] MODE FOR CARRYING OUT THE INVENTION

[0620] This invention is a system for allowing users to design and virtually experience the home they desire. This system generates a three-dimensional model of the home based on user input information and provides the user with a virtual experience using virtual reality and mixed reality technologies.

[0621] Hardware and software used

[0622] Hardware:

[0623] Device: The device (e.g., PC, tablet, smartphone) that collects user input and provides the virtual experience.

[0624] Server: A high-performance computer that runs generative AI models and processes data

[0625] VR device: A device that allows users to experience a virtual home (e.g., Oculus Rift, HTC Vive)

[0626] MR device: A device that provides a mixed reality experience (e.g., Microsoft HoloLens)

[0627] software:

[0628] Generative AI model: An AI system that generates 3D models and interior designs based on user-entered information

[0629] Modeling software: Tools for creating 3D models (e.g. Blender, 3ds Max)

[0630] VR software: Tools for converting 3D models into virtual reality formats (e.g. Unity, Unreal Engine)

[0631] MR Software: Tools for converting virtual homes into mixed reality formats

[0632] Database: A system for managing user input information and generated model data.

[0633] Program processing

[0634] 1. User Information Collection

[0635] Users use the terminal to input their desired location, architectural requirements, and design preferences. The terminal then sends this information to the server. For example, if a user wants a "modern, three-story house in central Tokyo," they would enter those detailed requirements.

[0636] 2. Generation of 3D modeling

[0637] The server uses the generative AI model to generate a 3D model of the house based on the received information. For example, a specific 3D model can be generated based on the criteria of "a three-story house with a modern design."

[0638] 3. Creating and experiencing a VR prototype

[0639] The server converts the generated 3D model into a VR format and sends it to the terminal. The terminal then allows the user to experience the virtual home through a VR device. The user puts on the VR device and can walk around inside and outside the building in the virtual space.

[0640] 4. Interior design proposals

[0641] Next, the user inputs their desired information about the interior design, furniture, and appliance placement into the device. For example, if a user requests a natural interior, a large TV, and an open kitchen, this information is sent to the server. The server then uses a generative AI model to generate a detailed interior design and sends the design data to the device. The user can then view the virtual interior design using the VR device.

[0642] 5. Projecting the site using MR technology

[0643] The user inputs real-world site information into the device and sends it to the server. For example, if a user specifies "I want to build on a 500-square-meter site in the suburbs," that information is processed by the server. The server combines a generative AI model with MR technology to generate data for projecting a virtual home onto the real site, which is then sent to the device. The device then uses an MR device to virtually project the home onto the real site, allowing the user to check the lighting, shadows, color scheme, and other aspects in real time.

[0644] 6. Maintenance and renovation forecasts

[0645] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it generates information such as "a simulation of deterioration after 10 years of construction" or "renovation of the children's room is necessary in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovation, and sends it to the device. The device notifies the user of this information, supporting a comfortable home over the long term.

[0646] Examples of concrete examples and prompts

[0647] As a concrete example, if a user requests a "two-story house with a natural design in the suburbs of Tokyo," the processing steps are as follows:

[0648] 1. The user enters desired conditions such as "suburbs of Tokyo," "natural design," and "two-story building" on the terminal.

[0649] 2. The device sends this information to a server, which uses a generative AI model to create a 3D model based on the conditions.

[0650] 3. The server converts the 3D model into a VR format and sends it to the terminal, where the user begins the virtual experience using the VR device.

[0651] 4. If the user requests "wood-style furniture," "eco-friendly appliances," or "a living room with an open ceiling space" as interior design preferences, this information will be collected from the device.

[0652] 5. The server generates a detailed interior design based on the collected information and sends it to the device, where the user can again view the virtual interior in their VR device.

[0653] 6. When the user selects "build on a 200-square-meter lot," the server uses MR technology to generate data for projecting a virtual house onto the real lot and sends it to the device. The user can then check how the light and shadows are reflected on the MR device.

[0654] 7. Finally, the server performs a "10-year deterioration simulation" and a "5-year renovation prediction" and sends the results to the terminal. The terminal then notifies the user of the results.

[0655] Example prompt sentence:

[0656] "As a user, I would like you to design a two-story house with a natural design in the suburbs of Tokyo. Specifically, I would like to use wood-style furniture and incorporate eco-friendly appliances. I would also like a living room with an open ceiling. I plan to build it on a 200-square-meter lot, and I would like to be able to use mixed reality technology to create a virtual experience. I would also like you to propose maintenance and renovations for 10 years from now."

[0657] In this way, the system allows users to imagine the design of their home in detail, experience it in a virtual space, and even create long-term maintenance plans.

[0658] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0659] Step 1:

[0660] The user enters information

[0661] Users use the terminal to input their desired location, architectural requirements, and design preferences. This information includes the desired location, detailed architectural requirements, interior style, etc. If a user wants a "modern three-story home in central Tokyo," they input specific details.

[0662] Input: location, architectural conditions, design preferences

[0663] Output: Detailed data about the user's preferences

[0664] Step 2:

[0665] Send the collected information to a server

[0666] The device transmits the desired information collected from the user to the server, which stores this information and uses it for subsequent data processing.

[0667] Input: User's desired information

[0668] Output: User preference data stored on the server

[0669] Step 3:

[0670] The server generates the 3D model

[0671] The server uses the generative AI model to generate a 3D model of the house based on the information requested by the user. In this process, the AI ​​model generates a design based on the input information and creates 3D model data.

[0672] Input: User preference data

[0673] Data processing: Generating 3D models using generative AI models

[0674] Output: 3D model data

[0675] Step 4:

[0676] Converting 3D models into VR format

[0677] The server converts the generated 3D model data into VR format, allowing users to experience the home in a virtual space.

[0678] Input: 3D model data

[0679] Data processing: Converting 3D models into VR format

[0680] Output: VR format housing data

[0681] Step 5:

[0682] The user experiences VR

[0683] The terminal transmits the VR format house data received from the server to the VR device, providing the user with a virtual experience. The user wears the VR device and visually experiences the inside and outside of the virtual house.

[0684] Input: VR format housing data

[0685] Output: User's virtual experience information

[0686] Step 6:

[0687] Enter your interior design preferences

[0688] Users input their preferences for interior design, furniture, and appliance placement into the terminal, such as "natural interior," "large TV," and "open kitchen."

[0689] Input: Interior design, furniture, and home appliance preferences

[0690] Output: User's interior design preferences data

[0691] Step 7:

[0692] The collected interior preference information is sent to the server.

[0693] The device sends the interior design preferences collected from the user to the server, which receives the information and uses it for subsequent data processing.

[0694] Input: User's interior design preferences

[0695] Output: Interior preference data saved on the server

[0696] Step 8:

[0697] The server generates the interior design

[0698] The server generates a detailed interior design using a generative AI model based on the received interior design preferences.

[0699] Input: Interior design preferences data

[0700] Data processing: Generating interior designs using generative AI models

[0701] Output: Interior design data

[0702] Step 9:

[0703] Convert interior designs into VR format and provide them to users

[0704] The server converts the generated interior design data into VR format and sends it to the terminal, which then uses a VR device to provide the user with a virtual interior experience.

[0705] Input: Interior design data

[0706] Data processing: Converting interior design into VR format

[0707] Output: VR format interior data

[0708] Step 10:

[0709] Users input real site information

[0710] The user enters information about the lot on which they want to build a house into the terminal, such as "a 500-square-meter lot in the suburbs."

[0711] Input: Site information

[0712] Output: User's site information data

[0713] Step 11:

[0714] Send the collected site information to the server

[0715] The terminal sends the site information collected from the user to the server, which uses this information to perform the following processes:

[0716] Input: User's site information

[0717] Output: Site information data stored on the server

[0718] Step 12:

[0719] The server generates data to project the virtual house onto the site.

[0720] The server combines site information with a generative AI model and MR technology to generate data that projects a virtual home onto the real site.

[0721] Input: Site information data

[0722] Data processing: Generating residential projection data using generative AI and MR technology

[0723] Output: Virtual house projection data in MR format

[0724] Step 13:

[0725] Projecting a virtual house onto a real site through an MR device

[0726] The server then sends the generated MR-formatted virtual house projection data to the terminal, which then uses the MR device to virtually project the house onto the user's screen.The user can then use the MR device to check the location of the house, how the light hits it, and the shadows.

[0727] Input: MR format virtual house projection data

[0728] Output: User's projection confirmation information

[0729] Step 14:

[0730] The server predicts the deterioration and renovation of the home.

[0731] The server processes data to predict deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction and predicting renovation needs after five years.

[0732] Input: Housing information, user lifestyle information

[0733] Data processing: Deterioration and renovation prediction using generative AI

[0734] Output: Deterioration simulation data, renovation prediction data

[0735] Step 15:

[0736] Notifying users when maintenance or renovation is required

[0737] The server sends the generated simulation data and prediction data to the terminal, which then notifies the user of appropriate maintenance and renovation timing.

[0738] Input: Deterioration simulation data, renovation prediction data

[0739] Output: Notification information for the user

[0740] In this way, the present invention provides a system that allows users to design a home in the most specific way, virtually experience it, and even plan maintenance and renovations.

[0741] (Application example 1)

[0742] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0743] In home and interior design, users often have difficulty experiencing and confirming their preferences in a concrete and intuitive way. In particular, in physical stores, there are limited ways to check the actual layout and design of interior products before purchasing them, which can lead to lower post-purchase satisfaction. Furthermore, when users want to make revisions to detailed interior designs while experiencing them virtually, the process is cumbersome and they are unable to receive timely feedback.

[0744] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0745] In this invention, the server includes: an input means for inputting a user's desired location, architectural conditions, and design preferences; a modeling means for generating a 3D model of a house using a generative AI based on the information input through the input means; a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience; an interior input means for inputting the user's desired information regarding the interior design and layout of furniture and home appliances; a design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; an interior VR experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; a site input means for inputting information about a real site; an MR projection means for virtually projecting a house onto the real site based on the information input through the site input means by combining the generative AI and MR technology; and an experience means for allowing the user to wear smart glasses or a head-mounted display at an interior shop and experience the interior in both real and virtual spaces. This allows the user to virtually experience interior products at the interior shop and intuitively and concretely check the layout and design.

[0746] "Input means" refers to the means by which a user inputs desired location, architectural conditions, and design preferences into a terminal.

[0747] The "modeling means" is a means for generating a 3D model of a house using a generation AI based on information input through the input means.

[0748] The "VR experience means" is a means for converting the 3D model generated by the modeling means into a VR format, allowing the user to have a virtual experience.

[0749] The "interior input means" is a means for inputting desired information regarding the user's interior design and the placement of furniture and home appliances into the terminal.

[0750] The "design generation means" is a means for proposing an interior design using a generation AI based on the information input through the interior input means.

[0751] The "interior VR experience means" is a means that allows the interior design generated by the design generation means to be virtually experienced in VR format.

[0752] The "site input means" is a means for inputting actual site information into the terminal.

[0753] The "MR projection means" is a means for virtually projecting a house onto a real site by combining generation AI and MR technology based on information input through the site input means.

[0754] The "experience means" refers to a means by which users can wear smart glasses or a head-mounted display in an interior design shop and experience the interior design in both real and virtual spaces.

[0755] A "prediction method" is a means of predicting when a home will deteriorate or require maintenance based on the user's lifestyle and environmental conditions.

[0756] The system for carrying out the present invention is composed of multiple components that allow a user to virtually experience the home design and interior design they desire. The system includes the following elements:

[0757] First, the user terminal provides an input means for the user to input their desired location, architectural conditions, and design preferences. The input means includes an interface through which the user can input their preferences, such as "a three-story house with a modern design in central Tokyo."

[0758] The server then uses the collected information to generate a 3D model of the house using a generative AI model. During the generation process, for example, the server sends the following prompt to the generative AI:

[0759] "Create a 3D model of a modern, three-story house in central Tokyo."

[0760] The server converts the generated 3D model into VR format and sends it to the user's device, which then allows the user to experience the virtual home through a VR device. This process uses software such as OpenVR and PyOpenGL.

[0761] Next, an interior design input interface is provided for users to input their preferences regarding interior design, furniture, and appliance placement. Users can use this interface to input preferences such as "natural interior," "large TV," and "open kitchen."

[0762] The server then uses the collected interior information to propose a detailed interior design using generative AI again. This time, prompts are used as well. For example:

[0763] "Create a 3D model of a living room with natural decor, a large TV, and an open kitchen."

[0764] The generated interior design is then converted back into VR format and sent to the user's device, where the user can view the virtual interior design through the interior VR experience.

[0765] Additionally, users can use the site entry tool to input real-world site information, for example, a "500 square meter site in the suburbs."

[0766] Based on this information, the server combines generative AI and MR technology to generate data for projecting a virtual house onto the real site. This MR projection method allows users to use an MR device to virtually project the house onto the real site, allowing them to realistically check how the light hits it, the shadows, the color scheme, and more.

[0767] The server also uses predictive tools to forecast deterioration of the home and changes in lifestyle. For example, predictions include simulations of deterioration after 10 years of construction and the need to renovate the children's room in five years.

[0768] Finally, an experience tool is provided to users in the interior design shop's physical store. This tool allows users to wear smart glasses or a head-mounted display and experience interior design products in both real and virtual spaces. This allows users to check the actual layout and design of the products before purchasing them.

[0769] As described above, this system connects multiple components to enable users to have a concrete and intuitive virtual experience of home and interior design, ultimately helping them make the best choice.

[0770] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0771] Step 1:

[0772] An input method is used to input the user's desired location, architectural conditions, and design preferences. In this step, a user terminal is used, and the user inputs information such as "a three-story house with a modern design in central Tokyo" through the interface. The input information is converted into a data format within the terminal and sent to the server.

[0773] Step 2:

[0774] The server receives the information entered by the user. It then uses the generative AI model to generate a 3D model of the house. In this step, it sends the prompt "Please create a 3D model of a modern, three-story house located in central Tokyo" to the generative AI. It then receives the 3D model data returned by the generative AI and converts it into a VR format.

[0775] Step 3:

[0776] The server sends the generated 3D model data in VR format to the user's terminal. The user's terminal uses a VR device (e.g., a head-mounted display) to provide the user with a virtual home experience. The user can wear the VR device and virtually walk around inside and outside the building.

[0777] Step 4:

[0778] The user uses the interior input means to input desired information regarding the interior design, furniture, and placement of home appliances. In this step, the user inputs the desired interior style and specific furniture items, which are then sent to the server by the terminal. For example, information such as "natural interior," "large TV," and "open kitchen" can be input.

[0779] Step 5:

[0780] The server uses the generative AI model to propose a detailed interior design based on the information input through the interior input means. At this time, it sends a prompt to the generative AI: "Please create a 3D model of a living room with natural interior, a large TV, and an open kitchen." The interior design data returned by the generative AI is converted into VR format.

[0781] Step 6:

[0782] The server transmits the generated VR data of the interior design to the user terminal. The user terminal provides the user with a virtual interior experience using the interior VR experience means. The user can then use the VR device again to virtually check the interior design and walk around inside the house.

[0783] Step 7:

[0784] The user uses a site input means to input information about a real site. For example, the user inputs information about a "500 square meter site in the suburbs" and transmits it to the server via the terminal.

[0785] Step 8:

[0786] Based on the information entered through the site input means, the server combines AI generation and MR technology to generate data for virtually projecting the house onto the real site. In doing so, it generates detailed data such as lighting, shadows, and color schemes so that the virtual house is properly projected onto the real site.

[0787] Step 9:

[0788] The server transmits the generated MR data to the user's device, which then projects the virtual house onto the real site, allowing the user to see how the house will look in the real environment.

[0789] Step 10:

[0790] The server predicts deterioration of the home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation. Based on this prediction method, it generates simulation results, such as "simulation of deterioration after 10 years of construction" or "children's room renovations are necessary in 5 years," and sends these to the user's device.

[0791] Step 11:

[0792] Users can wear smart glasses or a head-mounted display in an interior design shop to experience the interior design in both real and virtual space. This experience allows users to check the actual layout and design of the product before purchasing it, allowing them to make the best choice.

[0793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0794] This invention is a system for designing the home desired by the user in the most specific way and providing a virtual experience, and by combining it with an emotion engine, the system's suggestions and display content can be adjusted according to the user's emotional state. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, prediction means, and the emotion engine.

[0795] 1. User Information Collection Practices

[0796] The terminal displays a data entry form for users to enter their desired location, construction conditions, and design preferences. Users enter information such as location (Central Tokyo), construction conditions (Modern design, three-story house), and design preference (Simple and modern).

[0797] 2. Generate 3D modeling

[0798] The device sends the information entered by the user to a server, which then uses generative AI to generate a 3D model of the home that meets the user's needs.

[0799] 3. Creating and experiencing a VR prototype

[0800] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[0801] 4. Interior design proposals

[0802] The terminal displays an interior input means for users to input their desired conditions regarding interior design, furniture, and appliance placement, and users can enter information such as "natural-style interior," "large TV," and "open kitchen."

[0803] The server uses generative AI to generate a detailed interior design based on the collected information and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[0804] 5. Projecting the site using MR technology

[0805] The user inputs real-world site information and sends it to the server via their device. For example, the user might specify, "I want to build on a 500-square-meter site in the suburbs."

[0806] Based on the received site information, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the terminal. The terminal then uses an MR device to virtually project the house onto the real site, allowing the user to realistically check how the light hits it, the shadows, color scheme, and more.

[0807] 6. Maintenance and renovation forecasts

[0808] The server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need renovation in five years.

[0809] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them maintain a comfortable home over the long term.

[0810] 7. Adjusting recommendations with an emotion engine

[0811] While the user is viewing a virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied.

[0812] The server adjusts its suggestions and displays based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server can quickly generate alternatives and suggest new interior designs or architectural conditions to the user.

[0813] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[0814] The processing flow will be explained below.

[0815] Step 1:

[0816] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[0817] Step 2:

[0818] Users input information about the home they desire (e.g., location "central Tokyo," construction requirements "a three-story home with a modern design," design preferences "simple and modern") into the terminal.

[0819] Step 3:

[0820] The terminal transmits the information entered by the user to the server.

[0821] Step 4:

[0822] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[0823] Step 5:

[0824] The server converts the generated 3D model into VR format and sends the data to the terminal.

[0825] Step 6:

[0826] The terminal prepares the VR device and guides the user to put it on.

[0827] Step 7:

[0828] Users put on a VR device and experience a virtual home.

[0829] Step 8:

[0830] The terminal displays an interior input form that allows the user to enter information about the desired interior design and the placement of furniture and appliances.

[0831] Step 9:

[0832] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[0833] Step 10:

[0834] The terminal transmits the interior information input by the user to the server.

[0835] Step 11:

[0836] The server uses generative AI to generate a specific interior design based on the received information.

[0837] Step 12:

[0838] The server converts the generated interior design into VR format and sends the data to the terminal.

[0839] Step 13:

[0840] The terminal again uses a VR device to allow the user to experience a virtual interior.

[0841] Step 14:

[0842] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[0843] Step 15:

[0844] The user inputs real-world site information into the terminal.

[0845] Step 16:

[0846] The terminal transmits the site information entered by the user to the server.

[0847] Step 17:

[0848] Based on the site information it receives, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site.

[0849] Step 18:

[0850] The server transmits the generated MR projection data to the terminal.

[0851] Step 19:

[0852] The terminal uses an MR device to allow the user to virtually project a house onto a real-world site.

[0853] Step 20:

[0854] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[0855] Step 21:

[0856] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[0857] Step 22:

[0858] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[0859] Step 23:

[0860] The server transmits the generated proposal data to the terminal.

[0861] Step 24:

[0862] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[0863] Step 25:

[0864] The device uses an emotion engine to analyze the user's emotional state in real time while they are experiencing the virtual world.

[0865] Step 26:

[0866] The emotion engine determines the user's emotional state based on biometric information such as facial expressions, voice, and gestures.

[0867] Step 27:

[0868] The server adjusts the suggestions and display content based on feedback from the emotion engine.

[0869] Step 28:

[0870] If the emotion engine detects a user's dissatisfaction, the server immediately generates an alternative and sends it to the device.

[0871] Step 29:

[0872] The device presents alternatives to the user and suggests new interior designs and architectural requirements.

[0873] Example 2

[0874] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0875] Conventional home design systems were unable to fully reflect the user's wishes and emotions, and virtual and interior experiences were limited. It was also difficult for users to predict when future renovations or maintenance would be required. There was also no real-time proposal adjustment that took the user's emotions into account, which led to problems with the quality of the user experience.

[0876] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0877] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a 3D model of the house using a generative AI based on the information input through the input means; virtual experience means for converting the 3D model generated by the modeling means into a VR format to allow the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; interior virtual experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; site input means for inputting information about the actual site; mixed reality projection means for virtually projecting the house onto the actual site based on the information input through the site input means by combining generative AI and mixed reality technology; prediction means for predicting deterioration of the house and changes in lifestyle and proposing appropriate timing for maintenance and improvements; and an emotion engine for analyzing the user's emotional state in real time and adjusting the content of the proposals. This will enable specific home designs that reflect the user's wishes and emotions, improving the quality of virtual and interior experiences, and even making it possible to accurately predict the timing of future renovations and maintenance.

[0878] "User" refers to an individual who uses this system to receive home design and interior design proposals and have a virtual experience.

[0879] "Input means" refers to the interface that allows users to input their desired location, architectural conditions, and design preferences.

[0880] "Modeling means" refers to the function of generating a 3D model of a house using generative AI based on information entered by the user.

[0881] "Virtual experience means" refers to the function of converting the generated 3D model into VR format, allowing users to experience the house in a virtual space.

[0882] "Interior input means" refers to an interface that allows users to input desired information regarding interior design and the placement of furniture and home appliances.

[0883] "Design generation means" refers to a function that uses generation AI to propose interior designs based on information input through the interior input means.

[0884] "Interior virtual experience means" refers to a function that provides users with the generated interior design in VR format, allowing them to experience it in a virtual space.

[0885] "Site input means" refers to an interface for a user to input real site information.

[0886] "Mixed reality projection means" refers to a function that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[0887] "Predictive tools" refers to functions that predict deterioration of the home and changes in lifestyle, and suggest appropriate timing for maintenance and renovations.

[0888] The "emotion engine" refers to a function that analyzes the user's emotional state in real time and adjusts the content of suggestions.

[0889] MODE FOR CARRYING OUT THE INVENTION

[0890] An embodiment of this invention is a system that specifically designs a home desired by a user, provides a virtual experience, and further combines an emotion engine to make suggestions based on the user's emotions. This system is composed of the following main components: input means, modeling means, virtual experience means, interior input means, design generation means, interior virtual experience means, site input means, mixed reality projection means, prediction means, and emotion engine.

[0891] Collection of User Information

[0892] The terminal displays a data entry form for users to enter their desired location, architectural conditions, and design preferences. For example, users might enter information such as "Central Tokyo," "Modern design three-story home," and "Simple and modern."

[0893] 3D modeling generation

[0894] The device sends the information entered by the user to the server. The server uses the received information to generate a 3D model of the house using a generative AI model. For example, GPT-4 can be used as the generative AI model. The following prompt sentence is used as input:

[0895] User Information:

[0896] Location: Central Tokyo

[0897] Building requirements: Modern design, three-storey house

[0898] Design preference: Simple and modern

[0899] Generate a 3D model of the house according to the user's wishes.

[0900] Creating and experiencing VR prototypes

[0901] The server converts the generated 3D model into a VR format and sends the data to the terminal, which then provides the user with a virtual experience through a VR device such as Oculus Rift. The user can then use the VR device to virtually explore the inside and outside of the building.

[0902] Interior design proposals

[0903] The device displays an interior input tool for the user to input their desired information regarding interior design, furniture, and appliance placement, prompting them to enter information such as "natural-style interior," "large TV," and "open kitchen." The server uses generative AI based on the collected information to generate a detailed interior design and sends the design data to the device. The user then uses the VR device again to view the virtual interior design.

[0904] Projection of the site using MR technology

[0905] The user inputs real-world site information and sends it to the server via their device. For example, they might specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines generative AI and mixed reality technology to generate data for virtually projecting the house onto the real site, and sends that data to the device. The device then uses an MR device such as Microsoft HoloLens to virtually project the house onto the real site.

[0906] Maintenance and renovation forecasts

[0907] The server predicts deterioration of the home and changes in the user's lifestyle. Specifically, it makes predictions such as "simulating deterioration after 10 years of construction" and "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term.

[0908] Tailoring suggestions with an emotion engine

[0909] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions and voice to determine whether the user is satisfied or dissatisfied. The server then adjusts the proposal based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server will quickly generate alternative proposals and suggest new interior designs or architectural conditions to the user.

[0910] In this way, the system combined with the emotion engine provides users with a more personalized experience, enabling optimal home design based on their emotions, greatly improving user satisfaction and further facilitating the realization of their ideal home.

[0911] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0912] Step 1:

[0913] Users use a data input form on their device to input their desired location, architectural requirements, and design preferences. The input is in the form of "Central Tokyo," "Modern design three-story house," "Simple and modern," etc. The input data is sent to the server in JSON format.

[0914] Step 2:

[0915] The server parses the received JSON formatted user information and generates a prompt to be input to the generative AI model. The generated prompt is in the following text format:

[0916] User Information:

[0917] Location: Central Tokyo

[0918] Building requirements: Modern design, three-storey house

[0919] Design preference: Simple and modern

[0920] Generate a 3D model of the house according to the user's wishes.

[0921] Based on this prompt, the server sends the data to a generative AI model (e.g., GPT-4) to generate a 3D model.

[0922] Step 3:

[0923] The server receives the 3D model data output by the generative AI model and converts it into a VR format (e.g., Unity format), which allows users to experience the data in a virtual space.

[0924] Step 4:

[0925] The server sends the converted 3D model data to the device via an HTTP POST request, and the device then provides the user with a virtual experience through a VR device such as an Oculus Rift. The user can explore the building's interior and exterior in the virtual space.

[0926] Step 5:

[0927] The user uses the device interface to input desired interior design information, such as "natural-style interior," "large TV," and "open kitchen." The input data is sent to the server in JSON format.

[0928] Step 6:

[0929] The server generates a detailed interior design using a generative AI model based on the received interior information. The generated interior design data is then converted back to JSON format and sent to the device.

[0930] Step 7:

[0931] The terminal then provides the received interior design data to the user via a VR device, enabling them to check the interior in a virtual space. The user puts on the VR device and visually checks the interior layout and design in the virtual space.

[0932] Step 8:

[0933] The user inputs real-world site information using a terminal and sends it to the server. For example, a "500 square meter site in the suburbs" is specified. The input data is sent in JSON format.

[0934] Step 9:

[0935] Based on the received site information, the server combines generative AI and mixed reality technology to generate data that virtually projects the house onto the real site. The generated MR data is then sent to the device.

[0936] Step 10:

[0937] The device then provides the received MR data to the user via a mixed reality device such as Microsoft HoloLens, projecting the virtual home onto the real site. By wearing the MR device, the user can visually confirm the layout and appearance of the home on the real site.

[0938] Step 11:

[0939] The server runs an algorithm that predicts deterioration of the home and changes in lifestyle, generating predictive data such as "simulation of deterioration after 10 years of construction" or "need to renovate the children's room in five years." The generated predictive data is sent to the device.

[0940] Step 12:

[0941] The device then notifies the user of the received forecast data and suggests appropriate maintenance and renovation timing, allowing the user to create a maintenance plan to maintain a comfortable home for the long term.

[0942] Step 13:

[0943] While the user is viewing the virtual experience or interior design, the device uses a camera and microphone to capture the user's facial expressions and voice, inputting the data into an emotion engine, which analyzes the user's emotional state in real time and sends feedback to the server.

[0944] Step 14:

[0945] The server adjusts the proposals based on the user's emotions based on feedback from the emotion engine. If the user expresses dissatisfaction, the server quickly generates alternative proposals and suggests new interior designs or architectural conditions to the user, thereby improving user satisfaction.

[0946] (Application example 2)

[0947] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0948] Conventional home design systems have difficulty providing personalized proposals based on the user's emotions and preferences. Furthermore, they lack the functionality to adjust proposals based on the user's emotional responses when viewing homes and interiors in real time through virtual experiences. This has hindered user satisfaction and purchase intent.

[0949] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for inputting the user's desired location, architectural conditions, and design preferences, a modeling means for generating a 3D model of a house using a generation AI based on the information input through the input means, a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience, an interior input means for inputting the user's desired information regarding the interior design and the layout of furniture and home appliances, a design generation means for proposing an interior design using a generation AI based on the information input through the interior input means, and a virtual object in VR format for the interior design generated by the design generation means. The system includes an interior VR experience means for allowing users to experience a virtual home on a real site, a site input means for inputting real site information, an MR projection means for virtually projecting a home onto the real site by combining generative AI and MR technology based on the information input through the site input means, a prediction means for predicting deterioration of the home and changes in lifestyle and proposing appropriate timing for maintenance and renovation, an emotion engine means for analyzing the user's emotional state in real time and adjusting the content of proposals and displays, a biometric information analysis means for photographing the user's face during the virtual experience and analyzing emotions in real time, and a proposal adjustment means for customizing interior design and product proposals according to the user's emotional state. This makes it possible to provide optimal home design and interior proposals taking into account the user's emotional state.

[0950] "Input means" refers to a device or system that allows a user to input information such as desired location, architectural conditions, and design preferences.

[0951] A "modeling means" is a system or algorithm that uses generative AI to generate a 3D model of a house based on input information.

[0952] The "VR experience means" is a system that converts the generated 3D model into VR format, allowing users to view the house through a virtual experience.

[0953] "Interior input means" refers to a device or system for inputting information regarding the user's desired interior design and the layout of furniture and home appliances.

[0954] The "design generation means" is a system or algorithm that uses generative AI to propose interior designs based on information input through the interior input means.

[0955] The "Interior VR Experience Method" is a system that allows users to virtually experience the generated interior design in VR format.

[0956] "Site input means" refers to a device or system that allows a user to input actual site information.

[0957] The "MR projection means" is a system or algorithm that combines generative AI and MR technology to virtually project a house onto a real site based on information input through the site input means.

[0958] The "prediction method" is a system that predicts deterioration of a home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation.

[0959] An "emotion engine means" is a system or algorithm that analyzes a user's emotional state in real time and adjusts the content of suggestions and displays.

[0960] "Biometric information analysis means" refers to a system or device that photographs the user's face during a virtual experience and analyzes emotions in real time.

[0961] A "proposal tailoring means" is a system or algorithm that customizes interior design and product proposals based on the user's emotional state.

[0962] This invention is a system for specifically designing a home desired by a user and providing a virtual experience of the design. This system has the function of analyzing the user's emotional state and adjusting the proposed content. The detailed configuration and processing method of the system are explained below.

[0963] System configuration and hardware / software usage

[0964] This system mainly uses the following hardware and software:

[0965] Hardware:

[0966] Smartphone

[0967] VR devices (e.g., Oculus Rift, HTC Vive, etc.)

[0968] Mixed reality devices (e.g. Microsoft HoloLens)

[0969] Camera (captures the user's facial expression)

[0970] software:

[0971] Python

[0972] OpenCV (real-time image processing and facial expression analysis)

[0973] EmotionRecognizer (emotion recognition library)

[0974] AI model generation library

[0975] VRHandler (VR processing)

[0976] MRHandler (MR processing)

[0977] System processing overview

[0978] 1. Collection of User Information

[0979] The device (smartphone) displays a data entry form for the user to enter their desired location, architectural requirements, and design preferences. For example, the user might enter "Central Tokyo" as the location information, "Modern three-story house" as the architectural requirement, and "Simple and modern" as the design preference.

[0980] 2. Information transmission and 3D modeling

[0981] The device sends the information entered by the user to a server, which uses the received information to generate a 3D model of the home that meets the user's requirements using a generative AI model.

[0982] 3. Creating and experiencing a VR prototype

[0983] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[0984] 4. Interior design proposals

[0985] The terminal displays an interior design input tool that allows users to input their desired conditions regarding interior design, furniture, and appliance placement, such as "natural-style interior," "large TV," and "open kitchen."

[0986] The server uses the collected information to generate a detailed interior design using a generative AI model, and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[0987] 5. Input of site information and projection of the site using MR technology

[0988] The user inputs real-world site information and sends it to the server via their device. For example, the user may specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines a generative AI model and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site.

[0989] 6. Maintenance and renovation forecasts

[0990] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it makes predictions such as "simulating deterioration after 10 years of construction" or "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term.

[0991] 7. Adjusting recommendations with an emotion engine

[0992] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied. The server adjusts the suggestions and display content based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server immediately generates alternatives and suggests new interior designs or architectural conditions to the user.

[0993] Specific examples

[0994] 1. The user enters their preferences into the app (e.g., modern, minimalist style, gray sofa).

[0995] 2. The app identifies the user's facial expression (e.g., the user is smiling).

[0996] 3. The app uses generative AI to suggest furniture that suits the user's preferences and generate a 3D model of it.

[0997] 4. Check the furniture layout in a virtual store using VR or MR devices.

[0998] Prompt Sentence Examples

[0999] An example of a prompt to be fed to a generative AI model:

[1000] Generate interior designs based on user preferences, such as:

[1001] Style: Modern, Simple

[1002] Furniture: Gray sofa, white table

[1003] If the user is smiling, suggest light colors and soft materials.

[1004] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[1005] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1006] Step 1:

[1007] The terminal displays a data input form for the user to input their desired location, architectural conditions, and design preferences. The user enters information such as "Central Tokyo," "Modern three-story house," and "Simple and modern" into the input form. This becomes the input data.

[1008] Step 2:

[1009] The device sends the information entered by the user to the server, which uses the generative AI model to generate a 3D model of the home that meets the user's requirements. This 3D model becomes the output data.

[1010] Step 3:

[1011] The server converts the generated 3D model into VR format and sends the data to the terminal. The terminal allows the user to experience the virtual house through a VR device. The user puts on the VR device and virtually walks around inside and outside the building. The 3D model in VR format becomes the output data.

[1012] Step 4:

[1013] The terminal displays an interior design input tool for inputting desired information regarding interior design, furniture, and appliance placement. Users input information such as "natural-style interior," "large-screen TV," and "open kitchen." This becomes the input data for the interior information.

[1014] Step 5:

[1015] The device sends the interior information entered by the user to the server, which then uses a generative AI model based on the collected interior information to generate a detailed interior design. This interior design becomes the output data.

[1016] Step 6:

[1017] The server converts the generated interior design into VR format and sends the data to the terminal. The terminal then uses a VR device to allow the user to view the virtual interior design. The VR-format interior design becomes the output data.

[1018] Step 7:

[1019] The user inputs information about the actual site and sends it to the server via their terminal. For example, they might input information like, "I want to build on a 500 square meter site in the suburbs." This becomes the input data for the site information.

[1020] Step 8:

[1021] Based on the received site information, the server combines the generative AI model and MR technology to generate data for virtually projecting the house onto the real site, and sends that data to the terminal. The terminal then uses an MR device to allow the user to virtually project the house onto the real site. The MR format data is the output data.

[1022] Step 9:

[1023] The server predicts the deterioration of the home and changes in the user's lifestyle, and generates data suggesting the appropriate timing and content of maintenance and renovations, which it then sends to the device. For example, suggestions include "simulation of deterioration after 10 years of construction" and "renovation of the children's room is necessary in 5 years." This is the output data.

[1024] Step 10:

[1025] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied. The result is output data.

[1026] Step 11:

[1027] Based on the analysis of the user's emotional state, the server adjusts the suggestions and display content. For example, if the user shows a dissatisfied expression, the server will immediately generate alternative suggestions and suggest new interior designs or architectural conditions to the user. This is the output data of the adjusted suggestions.

[1028] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1029] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1030] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1031] [Third embodiment]

[1032] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1033] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1035] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1036] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1037] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1039] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1040] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1042] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1043] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1044] This invention is a system that allows users to design and virtually experience the home they desire in the most specific way. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, and prediction means.

[1045] 1. User Information Collection Practices

[1046] First, the terminal provides an input method for the user to input their desired location, architectural conditions, and design preferences. For example, suppose a user requests a "modern three-story house in central Tokyo."

[1047] 2. Generate 3D modeling

[1048] The device sends the collected information to a server, which uses generative AI to generate a 3D model of a home based on the user's specifications. For example, a 3D model could be generated based on the user's criteria: "A three-story home with a modern design."

[1049] 3. Creating and experiencing a VR prototype

[1050] The server converts the generated 3D model into a VR format and sends it to the terminal, which then allows the user to experience the virtual house through a VR device. For example, the user can wear the VR device and virtually walk around inside and outside the building.

[1051] 4. Interior design proposals

[1052] Next, the device provides an interior design input means for users to input their desired conditions regarding interior design, furniture, and the placement of home appliances. For example, if a user requests a "natural interior," "large TV," or "open kitchen," that information is collected.

[1053] The server uses generative AI to propose detailed interior designs based on the collected information and sends the design data to the device, where the user can then use the VR device to view the virtual interior design.

[1054] 5. Projecting the site using MR technology

[1055] Users input real-world site information and send it to the server via their device. For example, if a user specifies, "I want to build on a 500-square-meter site in the suburbs," that information is used.

[1056] The server combines generative AI and MR technology to generate data for projecting a virtual house onto a real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site, allowing the user to see how the light hits it, how the shadows are, and the color scheme in a realistic way.

[1057] 6. Maintenance and renovation forecasts

[1058] Finally, the server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need to be renovated in five years.

[1059] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them to live comfortably in the home for the long term.

[1060] In this way, users can visualize the specific design in detail and virtually experience it before actually building it.The system also provides the ability to determine the appropriate timing for maintenance and renovations, allowing users to live comfortably in their home for many years to come.

[1061] The processing flow will be explained below.

[1062] Step 1:

[1063] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[1064] Step 2:

[1065] Users input information about the home they desire (e.g., location "Minato Ward, Tokyo," architectural requirements "a three-story home with a modern design," design preferences "simple and modern," etc.) into the terminal.

[1066] Step 3:

[1067] The terminal transmits the information entered by the user to the server.

[1068] Step 4:

[1069] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[1070] Step 5:

[1071] The server converts the generated 3D model into VR format and sends the data to the terminal.

[1072] Step 6:

[1073] The terminal sends the received VR data to a VR device, allowing the user to experience the virtual home in VR.

[1074] Step 7:

[1075] The device displays an interior input form to collect information about the user's desired interior design and the placement of furniture and appliances.

[1076] Step 8:

[1077] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[1078] Step 9:

[1079] The terminal transmits the interior information input by the user to the server.

[1080] Step 10:

[1081] The server uses generative AI to generate a specific interior design based on the received information.

[1082] Step 11:

[1083] The server converts the generated interior design into VR format and sends the data to the terminal.

[1084] Step 12:

[1085] The terminal again uses a VR device to allow the user to experience a virtual interior.

[1086] Step 13:

[1087] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[1088] Step 14:

[1089] The user inputs real-world site information into the terminal.

[1090] Step 15:

[1091] The terminal transmits the site information entered by the user to the server.

[1092] Step 16:

[1093] Based on the received site information, the server combines generation AI and MR technology to generate data for virtually projecting the house onto the real site.

[1094] Step 17:

[1095] The server transmits the generated MR projection data to the terminal.

[1096] Step 18:

[1097] The terminal uses an MR device to allow the user to project a virtual home onto a real property.

[1098] Step 19:

[1099] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[1100] Step 20:

[1101] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[1102] Step 21:

[1103] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[1104] Step 22:

[1105] The server transmits the generated proposal data to the terminal.

[1106] Step 23:

[1107] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[1108] Example 1

[1109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1110] Conventional home design systems lacked the functionality to allow users to specifically simulate and virtually experience their desired location, architectural conditions, and detailed design preferences. They also lacked the means to properly predict interior design, site projection, and the timing of long-term maintenance and renovations. This could prevent users from fully realizing their ideal home design, leading to lower satisfaction with the completed home.

[1111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1112] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a three-dimensional model of the house using a generating AI based on the information input through the input means; virtual reality experience means for converting the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generating AI based on the information input through the interior input means; interior virtual reality experience means for allowing the interior design generated by the design generation means to be virtually experienced in a virtual reality format; site input means for inputting real site information; mixed reality projection means for virtually projecting the house onto the real site by combining generating AI and mixed reality technology based on the information input through the site input means; and prediction means for predicting deterioration of the house and changes in lifestyle, and suggesting appropriate timing for maintenance and renovation.

[1113] This allows users to simulate and virtually experience the detailed design of their desired home. It also allows them to get a detailed understanding of the interior design, projection onto the site, and the timing of long-term maintenance and renovations, which increases satisfaction after the home is completed.

[1114] "Input means" refers to a device or interface that allows a user to input information such as desired location, architectural conditions, and design preferences.

[1115] "Modeling means" refers to a device or software that uses generation AI to generate a three-dimensional model of a house based on information input through the input means.

[1116] The "virtual reality experience means" refers to a device or software that converts the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience.

[1117] "Interior input means" refers to a device or interface for inputting desired information regarding the user's interior design and the placement of furniture and home appliances.

[1118] The "design generation means" refers to a device or software that uses generation AI to propose interior designs based on information input through the interior input means.

[1119] The "interior virtual reality experience means" refers to a device or software that allows the interior design generated by the design generation means to be virtually experienced in virtual reality format.

[1120] The "site input means" is a device or interface for inputting actual site information.

[1121] A "mixed reality projection means" is a device or software that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[1122] "Predictive tools" are devices or software that predict deterioration of homes and changes in lifestyles, and suggest appropriate timing for maintenance and renovations.

[1123] MODE FOR CARRYING OUT THE INVENTION

[1124] This invention is a system for allowing users to design and virtually experience the home they desire. This system generates a three-dimensional model of the home based on user input information and provides the user with a virtual experience using virtual reality and mixed reality technologies.

[1125] Hardware and software used

[1126] Hardware:

[1127] Device: The device (e.g., PC, tablet, smartphone) that collects user input and provides the virtual experience.

[1128] Server: A high-performance computer that runs generative AI models and processes data

[1129] VR device: A device that allows users to experience a virtual home (e.g., Oculus Rift, HTC Vive)

[1130] MR device: A device that provides a mixed reality experience (e.g., Microsoft HoloLens)

[1131] software:

[1132] Generative AI model: An AI system that generates 3D models and interior designs based on user-entered information

[1133] Modeling software: Tools for creating 3D models (e.g. Blender, 3ds Max)

[1134] VR software: Tools for converting 3D models into virtual reality formats (e.g. Unity, Unreal Engine)

[1135] MR Software: Tools for converting virtual homes into mixed reality formats

[1136] Database: A system for managing user input information and generated model data.

[1137] Program processing

[1138] 1. User Information Collection

[1139] Users use the terminal to input their desired location, architectural requirements, and design preferences. The terminal then sends this information to the server. For example, if a user wants a "modern, three-story house in central Tokyo," they would enter those detailed requirements.

[1140] 2. Generation of 3D modeling

[1141] The server uses the generative AI model to generate a 3D model of the house based on the received information. For example, a specific 3D model can be generated based on the criteria of "a three-story house with a modern design."

[1142] 3. Creating and experiencing a VR prototype

[1143] The server converts the generated 3D model into a VR format and sends it to the terminal. The terminal then allows the user to experience the virtual home through a VR device. The user puts on the VR device and can walk around inside and outside the building in the virtual space.

[1144] 4. Interior design proposals

[1145] Next, the user inputs their desired information about the interior design, furniture, and appliance placement into the device. For example, if a user requests a natural interior, a large TV, and an open kitchen, this information is sent to the server. The server then uses a generative AI model to generate a detailed interior design and sends the design data to the device. The user can then view the virtual interior design using the VR device.

[1146] 5. Projecting the site using MR technology

[1147] The user inputs real-world site information into the device and sends it to the server. For example, if a user specifies "I want to build on a 500-square-meter site in the suburbs," that information is processed by the server. The server combines a generative AI model with MR technology to generate data for projecting a virtual home onto the real site, which is then sent to the device. The device then uses an MR device to virtually project the home onto the real site, allowing the user to check the lighting, shadows, color scheme, and other aspects in real time.

[1148] 6. Maintenance and renovation forecasts

[1149] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it generates information such as "a simulation of deterioration after 10 years of construction" or "renovation of the children's room is necessary in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovation, and sends it to the device. The device notifies the user of this information, supporting a comfortable home over the long term.

[1150] Examples of concrete examples and prompts

[1151] As a concrete example, if a user requests a "two-story house with a natural design in the suburbs of Tokyo," the processing steps are as follows:

[1152] 1. The user enters desired conditions such as "suburbs of Tokyo," "natural design," and "two-story building" on the terminal.

[1153] 2. The device sends this information to a server, which uses a generative AI model to create a 3D model based on the conditions.

[1154] 3. The server converts the 3D model into a VR format and sends it to the terminal, where the user begins the virtual experience using the VR device.

[1155] 4. If the user requests "wood-style furniture," "eco-friendly appliances," or "a living room with an open ceiling space" as interior design preferences, this information will be collected from the device.

[1156] 5. The server generates a detailed interior design based on the collected information and sends it to the device, where the user can again view the virtual interior in their VR device.

[1157] 6. When the user selects "build on a 200-square-meter lot," the server uses MR technology to generate data for projecting a virtual house onto the real lot and sends it to the device. The user can then check how the light and shadows are reflected on the MR device.

[1158] 7. Finally, the server performs a "10-year deterioration simulation" and a "5-year renovation prediction" and sends the results to the terminal. The terminal then notifies the user of the results.

[1159] Example prompt sentence:

[1160] "As a user, I would like you to design a two-story house with a natural design in the suburbs of Tokyo. Specifically, I would like to use wood-style furniture and incorporate eco-friendly appliances. I would also like a living room with an open ceiling. I plan to build it on a 200-square-meter lot, and I would like to be able to use mixed reality technology to create a virtual experience. I would also like you to propose maintenance and renovations for 10 years from now."

[1161] In this way, the system allows users to imagine the design of their home in detail, experience it in a virtual space, and even create long-term maintenance plans.

[1162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1163] Step 1:

[1164] The user enters information

[1165] Users use the terminal to input their desired location, architectural requirements, and design preferences. This information includes the desired location, detailed architectural requirements, interior style, etc. If a user wants a "modern three-story home in central Tokyo," they input specific details.

[1166] Input: location, architectural conditions, design preferences

[1167] Output: Detailed data about the user's preferences

[1168] Step 2:

[1169] Send the collected information to a server

[1170] The device transmits the desired information collected from the user to the server, which stores this information and uses it for subsequent data processing.

[1171] Input: User's desired information

[1172] Output: User preference data stored on the server

[1173] Step 3:

[1174] The server generates the 3D model

[1175] The server uses the generative AI model to generate a 3D model of the house based on the information requested by the user. In this process, the AI ​​model generates a design based on the input information and creates 3D model data.

[1176] Input: User preference data

[1177] Data processing: Generating 3D models using generative AI models

[1178] Output: 3D model data

[1179] Step 4:

[1180] Converting 3D models into VR format

[1181] The server converts the generated 3D model data into VR format, allowing users to experience the home in a virtual space.

[1182] Input: 3D model data

[1183] Data processing: Converting 3D models into VR format

[1184] Output: VR format housing data

[1185] Step 5:

[1186] The user experiences VR

[1187] The terminal transmits the VR format house data received from the server to the VR device, providing the user with a virtual experience. The user wears the VR device and visually experiences the inside and outside of the virtual house.

[1188] Input: VR format housing data

[1189] Output: User's virtual experience information

[1190] Step 6:

[1191] Enter your interior design preferences

[1192] Users input their preferences for interior design, furniture, and appliance placement into the terminal, such as "natural interior," "large TV," and "open kitchen."

[1193] Input: Interior design, furniture, and home appliance preferences

[1194] Output: User's interior design preferences data

[1195] Step 7:

[1196] The collected interior preference information is sent to the server.

[1197] The device sends the interior design preferences collected from the user to the server, which receives the information and uses it for subsequent data processing.

[1198] Input: User's interior design preferences

[1199] Output: Interior preference data saved on the server

[1200] Step 8:

[1201] The server generates the interior design

[1202] The server generates a detailed interior design using a generative AI model based on the received interior design preferences.

[1203] Input: Interior design preferences data

[1204] Data processing: Generating interior designs using generative AI models

[1205] Output: Interior design data

[1206] Step 9:

[1207] Convert interior designs into VR format and provide them to users

[1208] The server converts the generated interior design data into VR format and sends it to the terminal, which then uses a VR device to provide the user with a virtual interior experience.

[1209] Input: Interior design data

[1210] Data processing: Converting interior design into VR format

[1211] Output: VR format interior data

[1212] Step 10:

[1213] Users input real site information

[1214] The user enters information about the lot on which they want to build a house into the terminal, such as "a 500-square-meter lot in the suburbs."

[1215] Input: Site information

[1216] Output: User's site information data

[1217] Step 11:

[1218] Send the collected site information to the server

[1219] The terminal sends the site information collected from the user to the server, which uses this information to perform the following processes:

[1220] Input: User's site information

[1221] Output: Site information data stored on the server

[1222] Step 12:

[1223] The server generates data to project the virtual house onto the site.

[1224] The server combines site information with a generative AI model and MR technology to generate data that projects a virtual home onto the real site.

[1225] Input: Site information data

[1226] Data processing: Generating residential projection data using generative AI and MR technology

[1227] Output: Virtual house projection data in MR format

[1228] Step 13:

[1229] Projecting a virtual house onto a real site through an MR device

[1230] The server then sends the generated MR-formatted virtual house projection data to the terminal, which then uses the MR device to virtually project the house onto the user's screen.The user can then use the MR device to check the location of the house, how the light hits it, and the shadows.

[1231] Input: MR format virtual house projection data

[1232] Output: User's projection confirmation information

[1233] Step 14:

[1234] The server predicts the deterioration and renovation of the home.

[1235] The server processes data to predict deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction and predicting renovation needs after five years.

[1236] Input: Housing information, user lifestyle information

[1237] Data processing: Deterioration and renovation prediction using generative AI

[1238] Output: Deterioration simulation data, renovation prediction data

[1239] Step 15:

[1240] Notifying users when maintenance or renovation is required

[1241] The server sends the generated simulation data and prediction data to the terminal, which then notifies the user of appropriate maintenance and renovation timing.

[1242] Input: Deterioration simulation data, renovation prediction data

[1243] Output: Notification information for the user

[1244] In this way, the present invention provides a system that allows users to design a home in the most specific way, virtually experience it, and even plan maintenance and renovations.

[1245] (Application example 1)

[1246] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1247] In home and interior design, users often have difficulty experiencing and confirming their preferences in a concrete and intuitive way. In particular, in physical stores, there are limited ways to check the actual layout and design of interior products before purchasing them, which can lead to lower post-purchase satisfaction. Furthermore, when users want to make revisions to detailed interior designs while experiencing them virtually, the process is cumbersome and they are unable to receive timely feedback.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1249] In this invention, the server includes: an input means for inputting a user's desired location, architectural conditions, and design preferences; a modeling means for generating a 3D model of a house using a generative AI based on the information input through the input means; a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience; an interior input means for inputting the user's desired information regarding the interior design and layout of furniture and home appliances; a design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; an interior VR experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; a site input means for inputting information about a real site; an MR projection means for virtually projecting a house onto the real site based on the information input through the site input means by combining the generative AI and MR technology; and an experience means for allowing the user to wear smart glasses or a head-mounted display at an interior shop and experience the interior in both real and virtual spaces. This allows the user to virtually experience interior products at the interior shop and intuitively and concretely check the layout and design.

[1250] "Input means" refers to the means by which a user inputs desired location, architectural conditions, and design preferences into a terminal.

[1251] The "modeling means" is a means for generating a 3D model of a house using a generation AI based on information input through the input means.

[1252] The "VR experience means" is a means for converting the 3D model generated by the modeling means into a VR format, allowing the user to have a virtual experience.

[1253] The "interior input means" is a means for inputting desired information regarding the user's interior design and the placement of furniture and home appliances into the terminal.

[1254] The "design generation means" is a means for proposing an interior design using a generation AI based on the information input through the interior input means.

[1255] The "interior VR experience means" is a means that allows the interior design generated by the design generation means to be virtually experienced in VR format.

[1256] The "site input means" is a means for inputting actual site information into the terminal.

[1257] The "MR projection means" is a means for virtually projecting a house onto a real site by combining generation AI and MR technology based on information input through the site input means.

[1258] The "experience means" refers to a means by which users can wear smart glasses or a head-mounted display in an interior design shop and experience the interior design in both real and virtual spaces.

[1259] A "prediction method" is a means of predicting when a home will deteriorate or require maintenance based on the user's lifestyle and environmental conditions.

[1260] The system for carrying out the present invention is composed of multiple components that allow a user to virtually experience the home design and interior design they desire. The system includes the following elements:

[1261] First, the user terminal provides an input means for the user to input their desired location, architectural conditions, and design preferences. The input means includes an interface through which the user can input their preferences, such as "a three-story house with a modern design in central Tokyo."

[1262] The server then uses the collected information to generate a 3D model of the house using a generative AI model. During the generation process, for example, the server sends the following prompt to the generative AI:

[1263] "Create a 3D model of a modern, three-story house in central Tokyo."

[1264] The server converts the generated 3D model into VR format and sends it to the user's device, which then allows the user to experience the virtual home through a VR device. This process uses software such as OpenVR and PyOpenGL.

[1265] Next, an interior design input interface is provided for users to input their preferences regarding interior design, furniture, and appliance placement. Users can use this interface to input preferences such as "natural interior," "large TV," and "open kitchen."

[1266] The server then uses the collected interior information to propose a detailed interior design using generative AI again. This time, prompts are used as well. For example:

[1267] "Create a 3D model of a living room with natural decor, a large TV, and an open kitchen."

[1268] The generated interior design is then converted back into VR format and sent to the user's device, where the user can view the virtual interior design through the interior VR experience.

[1269] Additionally, users can use the site entry tool to input real-world site information, for example, a "500 square meter site in the suburbs."

[1270] Based on this information, the server combines generative AI and MR technology to generate data for projecting a virtual house onto the real site. This MR projection method allows users to use an MR device to virtually project the house onto the real site, allowing them to realistically check how the light hits it, the shadows, the color scheme, and more.

[1271] The server also uses predictive tools to forecast deterioration of the home and changes in lifestyle. For example, predictions include simulations of deterioration after 10 years of construction and the need to renovate the children's room in five years.

[1272] Finally, an experience tool is provided to users in the interior design shop's physical store. This tool allows users to wear smart glasses or a head-mounted display and experience interior design products in both real and virtual spaces. This allows users to check the actual layout and design of the products before purchasing them.

[1273] As described above, this system connects multiple components to enable users to have a concrete and intuitive virtual experience of home and interior design, ultimately helping them make the best choice.

[1274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1275] Step 1:

[1276] An input method is used to input the user's desired location, architectural conditions, and design preferences. In this step, a user terminal is used, and the user inputs information such as "a three-story house with a modern design in central Tokyo" through the interface. The input information is converted into a data format within the terminal and sent to the server.

[1277] Step 2:

[1278] The server receives the information entered by the user. It then uses the generative AI model to generate a 3D model of the house. In this step, it sends the prompt "Please create a 3D model of a modern, three-story house located in central Tokyo" to the generative AI. It then receives the 3D model data returned by the generative AI and converts it into a VR format.

[1279] Step 3:

[1280] The server sends the generated 3D model data in VR format to the user's terminal. The user's terminal uses a VR device (e.g., a head-mounted display) to provide the user with a virtual home experience. The user can wear the VR device and virtually walk around inside and outside the building.

[1281] Step 4:

[1282] The user uses the interior input means to input desired information regarding the interior design, furniture, and placement of home appliances. In this step, the user inputs the desired interior style and specific furniture items, which are then sent to the server by the terminal. For example, information such as "natural interior," "large TV," and "open kitchen" can be input.

[1283] Step 5:

[1284] The server uses the generative AI model to propose a detailed interior design based on the information input through the interior input means. At this time, it sends a prompt to the generative AI: "Please create a 3D model of a living room with natural interior, a large TV, and an open kitchen." The interior design data returned by the generative AI is converted into VR format.

[1285] Step 6:

[1286] The server transmits the generated VR data of the interior design to the user terminal. The user terminal provides the user with a virtual interior experience using the interior VR experience means. The user can then use the VR device again to virtually check the interior design and walk around inside the house.

[1287] Step 7:

[1288] The user uses a site input means to input information about a real site. For example, the user inputs information about a "500 square meter site in the suburbs" and transmits it to the server via the terminal.

[1289] Step 8:

[1290] Based on the information entered through the site input means, the server combines AI generation and MR technology to generate data for virtually projecting the house onto the real site. In doing so, it generates detailed data such as lighting, shadows, and color schemes so that the virtual house is properly projected onto the real site.

[1291] Step 9:

[1292] The server transmits the generated MR data to the user's device, which then projects the virtual house onto the real site, allowing the user to see how the house will look in the real environment.

[1293] Step 10:

[1294] The server predicts deterioration of the home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation. Based on this prediction method, it generates simulation results, such as "simulation of deterioration after 10 years of construction" or "children's room renovations are necessary in 5 years," and sends these to the user's device.

[1295] Step 11:

[1296] Users can wear smart glasses or a head-mounted display in an interior design shop to experience the interior design in both real and virtual space. This experience allows users to check the actual layout and design of the product before purchasing it, allowing them to make the best choice.

[1297] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1298] This invention is a system for designing the home desired by the user in the most specific way and providing a virtual experience, and by combining it with an emotion engine, the system's suggestions and display content can be adjusted according to the user's emotional state. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, prediction means, and the emotion engine.

[1299] 1. User Information Collection Practices

[1300] The terminal displays a data entry form for users to enter their desired location, construction conditions, and design preferences. Users enter information such as location (Central Tokyo), construction conditions (Modern design, three-story house), and design preference (Simple and modern).

[1301] 2. Generate 3D modeling

[1302] The device sends the information entered by the user to a server, which then uses generative AI to generate a 3D model of the home that meets the user's needs.

[1303] 3. Creating and experiencing a VR prototype

[1304] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[1305] 4. Interior design proposals

[1306] The terminal displays an interior input means for users to input their desired conditions regarding interior design, furniture, and appliance placement, and users can enter information such as "natural-style interior," "large TV," and "open kitchen."

[1307] The server uses generative AI to generate a detailed interior design based on the collected information and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[1308] 5. Projecting the site using MR technology

[1309] The user inputs real-world site information and sends it to the server via their device. For example, the user might specify, "I want to build on a 500-square-meter site in the suburbs."

[1310] Based on the received site information, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the terminal. The terminal then uses an MR device to virtually project the house onto the real site, allowing the user to realistically check how the light hits it, the shadows, color scheme, and more.

[1311] 6. Maintenance and renovation forecasts

[1312] The server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need renovation in five years.

[1313] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them maintain a comfortable home over the long term.

[1314] 7. Adjusting recommendations with an emotion engine

[1315] While the user is viewing a virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied.

[1316] The server adjusts its suggestions and displays based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server can quickly generate alternatives and suggest new interior designs or architectural conditions to the user.

[1317] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[1318] The processing flow will be explained below.

[1319] Step 1:

[1320] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[1321] Step 2:

[1322] Users input information about the home they desire (e.g., location "central Tokyo," construction requirements "a three-story home with a modern design," design preferences "simple and modern") into the terminal.

[1323] Step 3:

[1324] The terminal transmits the information entered by the user to the server.

[1325] Step 4:

[1326] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[1327] Step 5:

[1328] The server converts the generated 3D model into VR format and sends the data to the terminal.

[1329] Step 6:

[1330] The terminal prepares the VR device and guides the user to put it on.

[1331] Step 7:

[1332] Users put on a VR device and experience a virtual home.

[1333] Step 8:

[1334] The terminal displays an interior input form that allows the user to enter information about the desired interior design and the placement of furniture and appliances.

[1335] Step 9:

[1336] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[1337] Step 10:

[1338] The terminal transmits the interior information input by the user to the server.

[1339] Step 11:

[1340] The server uses generative AI to generate a specific interior design based on the received information.

[1341] Step 12:

[1342] The server converts the generated interior design into VR format and sends the data to the terminal.

[1343] Step 13:

[1344] The terminal again uses a VR device to allow the user to experience a virtual interior.

[1345] Step 14:

[1346] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[1347] Step 15:

[1348] The user inputs real-world site information into the terminal.

[1349] Step 16:

[1350] The terminal transmits the site information entered by the user to the server.

[1351] Step 17:

[1352] Based on the site information it receives, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site.

[1353] Step 18:

[1354] The server transmits the generated MR projection data to the terminal.

[1355] Step 19:

[1356] The terminal uses an MR device to allow the user to virtually project a house onto a real-world site.

[1357] Step 20:

[1358] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[1359] Step 21:

[1360] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[1361] Step 22:

[1362] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[1363] Step 23:

[1364] The server transmits the generated proposal data to the terminal.

[1365] Step 24:

[1366] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[1367] Step 25:

[1368] The device uses an emotion engine to analyze the user's emotional state in real time while they are experiencing the virtual world.

[1369] Step 26:

[1370] The emotion engine determines the user's emotional state based on biometric information such as facial expressions, voice, and gestures.

[1371] Step 27:

[1372] The server adjusts the suggestions and display content based on feedback from the emotion engine.

[1373] Step 28:

[1374] If the emotion engine detects a user's dissatisfaction, the server immediately generates an alternative and sends it to the device.

[1375] Step 29:

[1376] The device presents alternatives to the user and suggests new interior designs and architectural requirements.

[1377] Example 2

[1378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1379] Conventional home design systems were unable to fully reflect the user's wishes and emotions, and virtual and interior experiences were limited. It was also difficult for users to predict when future renovations or maintenance would be required. There was also no real-time proposal adjustment that took the user's emotions into account, which led to problems with the quality of the user experience.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1381] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a 3D model of the house using a generative AI based on the information input through the input means; virtual experience means for converting the 3D model generated by the modeling means into a VR format to allow the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; interior virtual experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; site input means for inputting information about the actual site; mixed reality projection means for virtually projecting the house onto the actual site based on the information input through the site input means by combining generative AI and mixed reality technology; prediction means for predicting deterioration of the house and changes in lifestyle and proposing appropriate timing for maintenance and improvements; and an emotion engine for analyzing the user's emotional state in real time and adjusting the content of the proposals. This will enable specific home designs that reflect the user's wishes and emotions, improving the quality of virtual and interior experiences, and even making it possible to accurately predict the timing of future renovations and maintenance.

[1382] "User" refers to an individual who uses this system to receive home design and interior design proposals and have a virtual experience.

[1383] "Input means" refers to the interface that allows users to input their desired location, architectural conditions, and design preferences.

[1384] "Modeling means" refers to the function of generating a 3D model of a house using generative AI based on information entered by the user.

[1385] "Virtual experience means" refers to the function of converting the generated 3D model into VR format, allowing users to experience the house in a virtual space.

[1386] "Interior input means" refers to an interface that allows users to input desired information regarding interior design and the placement of furniture and home appliances.

[1387] "Design generation means" refers to a function that uses generation AI to propose interior designs based on information input through the interior input means.

[1388] "Interior virtual experience means" refers to a function that provides users with the generated interior design in VR format, allowing them to experience it in a virtual space.

[1389] "Site input means" refers to an interface for a user to input real site information.

[1390] "Mixed reality projection means" refers to a function that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[1391] "Predictive tools" refers to functions that predict deterioration of the home and changes in lifestyle, and suggest appropriate timing for maintenance and renovations.

[1392] The "emotion engine" refers to a function that analyzes the user's emotional state in real time and adjusts the content of suggestions.

[1393] MODE FOR CARRYING OUT THE INVENTION

[1394] An embodiment of this invention is a system that specifically designs a home desired by a user, provides a virtual experience, and further combines an emotion engine to make suggestions based on the user's emotions. This system is composed of the following main components: input means, modeling means, virtual experience means, interior input means, design generation means, interior virtual experience means, site input means, mixed reality projection means, prediction means, and emotion engine.

[1395] Collection of User Information

[1396] The terminal displays a data entry form for users to enter their desired location, architectural conditions, and design preferences. For example, users might enter information such as "Central Tokyo," "Modern design three-story home," and "Simple and modern."

[1397] 3D modeling generation

[1398] The device sends the information entered by the user to the server. The server uses the received information to generate a 3D model of the house using a generative AI model. For example, GPT-4 can be used as the generative AI model. The following prompt sentence is used as input:

[1399] User Information:

[1400] Location: Central Tokyo

[1401] Building requirements: Modern design, three-storey house

[1402] Design preference: Simple and modern

[1403] Generate a 3D model of the house according to the user's wishes.

[1404] Creating and experiencing VR prototypes

[1405] The server converts the generated 3D model into a VR format and sends the data to the terminal, which then provides the user with a virtual experience through a VR device such as Oculus Rift. The user can then use the VR device to virtually explore the inside and outside of the building.

[1406] Interior design proposals

[1407] The device displays an interior input tool for the user to input their desired information regarding interior design, furniture, and appliance placement, prompting them to enter information such as "natural-style interior," "large TV," and "open kitchen." The server uses generative AI based on the collected information to generate a detailed interior design and sends the design data to the device. The user then uses the VR device again to view the virtual interior design.

[1408] Projection of the site using MR technology

[1409] The user inputs real-world site information and sends it to the server via their device. For example, they might specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines generative AI and mixed reality technology to generate data for virtually projecting the house onto the real site, and sends that data to the device. The device then uses an MR device such as Microsoft HoloLens to virtually project the house onto the real site.

[1410] Maintenance and renovation forecasts

[1411] The server predicts deterioration of the home and changes in the user's lifestyle. Specifically, it makes predictions such as "simulating deterioration after 10 years of construction" and "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term.

[1412] Tailoring suggestions with an emotion engine

[1413] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions and voice to determine whether the user is satisfied or dissatisfied. The server then adjusts the proposal based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server will quickly generate alternative proposals and suggest new interior designs or architectural conditions to the user.

[1414] In this way, the system combined with the emotion engine provides users with a more personalized experience, enabling optimal home design based on their emotions, greatly improving user satisfaction and further facilitating the realization of their ideal home.

[1415] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1416] Step 1:

[1417] Users use a data input form on their device to input their desired location, architectural requirements, and design preferences. The input is in the form of "Central Tokyo," "Modern design three-story house," "Simple and modern," etc. The input data is sent to the server in JSON format.

[1418] Step 2:

[1419] The server parses the received JSON formatted user information and generates a prompt to be input to the generative AI model. The generated prompt is in the following text format:

[1420] User Information:

[1421] Location: Central Tokyo

[1422] Building requirements: Modern design, three-storey house

[1423] Design preference: Simple and modern

[1424] Generate a 3D model of the house according to the user's wishes.

[1425] Based on this prompt, the server sends the data to a generative AI model (e.g., GPT-4) to generate a 3D model.

[1426] Step 3:

[1427] The server receives the 3D model data output by the generative AI model and converts it into a VR format (e.g., Unity format), which allows users to experience the data in a virtual space.

[1428] Step 4:

[1429] The server sends the converted 3D model data to the device via an HTTP POST request, and the device then provides the user with a virtual experience through a VR device such as an Oculus Rift. The user can explore the building's interior and exterior in the virtual space.

[1430] Step 5:

[1431] The user uses the device interface to input desired interior design information, such as "natural-style interior," "large TV," and "open kitchen." The input data is sent to the server in JSON format.

[1432] Step 6:

[1433] The server generates a detailed interior design using a generative AI model based on the received interior information. The generated interior design data is then converted back to JSON format and sent to the device.

[1434] Step 7:

[1435] The terminal then provides the received interior design data to the user via a VR device, enabling them to check the interior in a virtual space. The user puts on the VR device and visually checks the interior layout and design in the virtual space.

[1436] Step 8:

[1437] The user inputs real-world site information using a terminal and sends it to the server. For example, a "500 square meter site in the suburbs" is specified. The input data is sent in JSON format.

[1438] Step 9:

[1439] Based on the received site information, the server combines generative AI and mixed reality technology to generate data that virtually projects the house onto the real site. The generated MR data is then sent to the device.

[1440] Step 10:

[1441] The device then provides the received MR data to the user via a mixed reality device such as Microsoft HoloLens, projecting the virtual home onto the real site. By wearing the MR device, the user can visually confirm the layout and appearance of the home on the real site.

[1442] Step 11:

[1443] The server runs an algorithm that predicts deterioration of the home and changes in lifestyle, generating predictive data such as "simulation of deterioration after 10 years of construction" or "need to renovate the children's room in five years." The generated predictive data is sent to the device.

[1444] Step 12:

[1445] The device then notifies the user of the received forecast data and suggests appropriate maintenance and renovation timing, allowing the user to create a maintenance plan to maintain a comfortable home for the long term.

[1446] Step 13:

[1447] While the user is viewing the virtual experience or interior design, the device uses a camera and microphone to capture the user's facial expressions and voice, inputting the data into an emotion engine, which analyzes the user's emotional state in real time and sends feedback to the server.

[1448] Step 14:

[1449] The server adjusts the proposals based on the user's emotions based on feedback from the emotion engine. If the user expresses dissatisfaction, the server quickly generates alternative proposals and suggests new interior designs or architectural conditions to the user, thereby improving user satisfaction.

[1450] (Application example 2)

[1451] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1452] Conventional home design systems have difficulty providing personalized proposals based on the user's emotions and preferences. Furthermore, they lack the functionality to adjust proposals based on the user's emotional responses when viewing homes and interiors in real time through virtual experiences. This has hindered user satisfaction and purchase intent.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for inputting the user's desired location, architectural conditions, and design preferences, a modeling means for generating a 3D model of a house using a generation AI based on the information input through the input means, a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience, an interior input means for inputting the user's desired information regarding the interior design and the layout of furniture and home appliances, a design generation means for proposing an interior design using a generation AI based on the information input through the interior input means, and a virtual object in VR format for the interior design generated by the design generation means. The system includes an interior VR experience means for allowing users to experience a virtual home on a real site, a site input means for inputting real site information, an MR projection means for virtually projecting a home onto the real site by combining generative AI and MR technology based on the information input through the site input means, a prediction means for predicting deterioration of the home and changes in lifestyle and proposing appropriate timing for maintenance and renovation, an emotion engine means for analyzing the user's emotional state in real time and adjusting the content of proposals and displays, a biometric information analysis means for photographing the user's face during the virtual experience and analyzing emotions in real time, and a proposal adjustment means for customizing interior design and product proposals according to the user's emotional state. This makes it possible to provide optimal home design and interior proposals taking into account the user's emotional state.

[1454] "Input means" refers to a device or system that allows a user to input information such as desired location, architectural conditions, and design preferences.

[1455] A "modeling means" is a system or algorithm that uses generative AI to generate a 3D model of a house based on input information.

[1456] The "VR experience means" is a system that converts the generated 3D model into VR format, allowing users to view the house through a virtual experience.

[1457] "Interior input means" refers to a device or system for inputting information regarding the user's desired interior design and the layout of furniture and home appliances.

[1458] The "design generation means" is a system or algorithm that uses generative AI to propose interior designs based on information input through the interior input means.

[1459] The "Interior VR Experience Method" is a system that allows users to virtually experience the generated interior design in VR format.

[1460] "Site input means" refers to a device or system that allows a user to input actual site information.

[1461] The "MR projection means" is a system or algorithm that combines generative AI and MR technology to virtually project a house onto a real site based on information input through the site input means.

[1462] The "prediction method" is a system that predicts deterioration of a home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation.

[1463] An "emotion engine means" is a system or algorithm that analyzes a user's emotional state in real time and adjusts the content of suggestions and displays.

[1464] "Biometric information analysis means" refers to a system or device that photographs the user's face during a virtual experience and analyzes emotions in real time.

[1465] A "proposal tailoring means" is a system or algorithm that customizes interior design and product proposals based on the user's emotional state.

[1466] This invention is a system for specifically designing a home desired by a user and providing a virtual experience of the design. This system has the function of analyzing the user's emotional state and adjusting the proposed content. The detailed configuration and processing method of the system are explained below.

[1467] System configuration and hardware / software usage

[1468] This system mainly uses the following hardware and software:

[1469] Hardware:

[1470] Smartphone

[1471] VR devices (e.g., Oculus Rift, HTC Vive, etc.)

[1472] Mixed reality devices (e.g. Microsoft HoloLens)

[1473] Camera (captures the user's facial expression)

[1474] software:

[1475] Python

[1476] OpenCV (real-time image processing and facial expression analysis)

[1477] EmotionRecognizer (emotion recognition library)

[1478] AI model generation library

[1479] VRHandler (VR processing)

[1480] MRHandler (MR processing)

[1481] System processing overview

[1482] 1. Collection of User Information

[1483] The device (smartphone) displays a data entry form for the user to enter their desired location, architectural requirements, and design preferences. For example, the user might enter "Central Tokyo" as the location information, "Modern three-story house" as the architectural requirement, and "Simple and modern" as the design preference.

[1484] 2. Information transmission and 3D modeling

[1485] The device sends the information entered by the user to a server, which uses the received information to generate a 3D model of the home that meets the user's requirements using a generative AI model.

[1486] 3. Creating and experiencing a VR prototype

[1487] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[1488] 4. Interior design proposals

[1489] The terminal displays an interior design input tool that allows users to input their desired conditions regarding interior design, furniture, and appliance placement, such as "natural-style interior," "large TV," and "open kitchen."

[1490] The server uses the collected information to generate a detailed interior design using a generative AI model, and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[1491] 5. Input of site information and projection of the site using MR technology

[1492] The user inputs real-world site information and sends it to the server via their device. For example, the user may specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines a generative AI model and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site.

[1493] 6. Maintenance and renovation forecasts

[1494] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it makes predictions such as "simulating deterioration after 10 years of construction" or "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term.

[1495] 7. Adjusting recommendations with an emotion engine

[1496] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied. The server adjusts the suggestions and display content based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server immediately generates alternatives and suggests new interior designs or architectural conditions to the user.

[1497] Specific examples

[1498] 1. The user enters their preferences into the app (e.g., modern, minimalist style, gray sofa).

[1499] 2. The app identifies the user's facial expression (e.g., the user is smiling).

[1500] 3. The app uses generative AI to suggest furniture that suits the user's preferences and generate a 3D model of it.

[1501] 4. Check the furniture layout in a virtual store using VR or MR devices.

[1502] Prompt Sentence Examples

[1503] An example of a prompt to be fed to a generative AI model:

[1504] Generate interior designs based on user preferences, such as:

[1505] Style: Modern, Simple

[1506] Furniture: Gray sofa, white table

[1507] If the user is smiling, suggest light colors and soft materials.

[1508] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[1509] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1510] Step 1:

[1511] The terminal displays a data input form for the user to input their desired location, architectural conditions, and design preferences. The user enters information such as "Central Tokyo," "Modern three-story house," and "Simple and modern" into the input form. This becomes the input data.

[1512] Step 2:

[1513] The device sends the information entered by the user to the server, which uses the generative AI model to generate a 3D model of the home that meets the user's requirements. This 3D model becomes the output data.

[1514] Step 3:

[1515] The server converts the generated 3D model into VR format and sends the data to the terminal. The terminal allows the user to experience the virtual house through a VR device. The user puts on the VR device and virtually walks around inside and outside the building. The 3D model in VR format becomes the output data.

[1516] Step 4:

[1517] The terminal displays an interior design input tool for inputting desired information regarding interior design, furniture, and appliance placement. Users input information such as "natural-style interior," "large-screen TV," and "open kitchen." This becomes the input data for the interior information.

[1518] Step 5:

[1519] The device sends the interior information entered by the user to the server, which then uses a generative AI model based on the collected interior information to generate a detailed interior design. This interior design becomes the output data.

[1520] Step 6:

[1521] The server converts the generated interior design into VR format and sends the data to the terminal. The terminal then uses a VR device to allow the user to view the virtual interior design. The VR-format interior design becomes the output data.

[1522] Step 7:

[1523] The user inputs information about the actual site and sends it to the server via their terminal. For example, they might input information like, "I want to build on a 500 square meter site in the suburbs." This becomes the input data for the site information.

[1524] Step 8:

[1525] Based on the received site information, the server combines the generative AI model and MR technology to generate data for virtually projecting the house onto the real site, and sends that data to the terminal. The terminal then uses an MR device to allow the user to virtually project the house onto the real site. The MR format data is the output data.

[1526] Step 9:

[1527] The server predicts the deterioration of the home and changes in the user's lifestyle, and generates data suggesting the appropriate timing and content of maintenance and renovations, which it then sends to the device. For example, suggestions include "simulation of deterioration after 10 years of construction" and "renovation of the children's room is necessary in 5 years." This is the output data.

[1528] Step 10:

[1529] While the user is viewing the virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied. The result is output data.

[1530] Step 11:

[1531] Based on the analysis of the user's emotional state, the server adjusts the suggestions and display content. For example, if the user shows a dissatisfied expression, the server will immediately generate alternative suggestions and suggest new interior designs or architectural conditions to the user. This is the output data of the adjusted suggestions.

[1532] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1533] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1534] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1535] [Fourth embodiment]

[1536] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1537] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1538] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1539] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1540] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1541] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1542] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1543] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1544] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1545] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1546] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1547] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1548] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1549] This invention is a system that allows users to design and virtually experience the home they desire in the most specific way. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, and prediction means.

[1550] 1. User Information Collection Practices

[1551] First, the terminal provides an input method for the user to input their desired location, architectural conditions, and design preferences. For example, suppose a user requests a "modern three-story house in central Tokyo."

[1552] 2. Generate 3D modeling

[1553] The device sends the collected information to a server, which uses generative AI to generate a 3D model of a home based on the user's specifications. For example, a 3D model could be generated based on the user's criteria: "A three-story home with a modern design."

[1554] 3. Creating and experiencing a VR prototype

[1555] The server converts the generated 3D model into a VR format and sends it to the terminal, which then allows the user to experience the virtual house through a VR device. For example, the user can wear the VR device and virtually walk around inside and outside the building.

[1556] 4. Interior design proposals

[1557] Next, the device provides an interior design input means for users to input their desired conditions regarding interior design, furniture, and the placement of home appliances. For example, if a user requests a "natural interior," "large TV," or "open kitchen," that information is collected.

[1558] The server uses generative AI to propose detailed interior designs based on the collected information and sends the design data to the device, where the user can then use the VR device to view the virtual interior design.

[1559] 5. Projecting the site using MR technology

[1560] Users input real-world site information and send it to the server via their device. For example, if a user specifies, "I want to build on a 500-square-meter site in the suburbs," that information is used.

[1561] The server combines generative AI and MR technology to generate data for projecting a virtual house onto a real site, and sends it to the device. The device then uses an MR device to virtually project the house onto the real site, allowing the user to see how the light hits it, how the shadows are, and the color scheme in a realistic way.

[1562] 6. Maintenance and renovation forecasts

[1563] Finally, the server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need to be renovated in five years.

[1564] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them to live comfortably in the home for the long term.

[1565] In this way, users can visualize the specific design in detail and virtually experience it before actually building it.The system also provides the ability to determine the appropriate timing for maintenance and renovations, allowing users to live comfortably in their home for many years to come.

[1566] The processing flow will be explained below.

[1567] Step 1:

[1568] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[1569] Step 2:

[1570] Users input information about the home they desire (e.g., location "Minato Ward, Tokyo," architectural requirements "a three-story home with a modern design," design preferences "simple and modern," etc.) into the terminal.

[1571] Step 3:

[1572] The terminal transmits the information entered by the user to the server.

[1573] Step 4:

[1574] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[1575] Step 5:

[1576] The server converts the generated 3D model into VR format and sends the data to the terminal.

[1577] Step 6:

[1578] The terminal sends the received VR data to a VR device, allowing the user to experience the virtual home in VR.

[1579] Step 7:

[1580] The device displays an interior input form to collect information about the user's desired interior design and the placement of furniture and appliances.

[1581] Step 8:

[1582] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[1583] Step 9:

[1584] The terminal transmits the interior information input by the user to the server.

[1585] Step 10:

[1586] The server uses generative AI to generate a specific interior design based on the received information.

[1587] Step 11:

[1588] The server converts the generated interior design into VR format and sends the data to the terminal.

[1589] Step 12:

[1590] The terminal again uses a VR device to allow the user to experience a virtual interior.

[1591] Step 13:

[1592] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[1593] Step 14:

[1594] The user inputs real-world site information into the terminal.

[1595] Step 15:

[1596] The terminal transmits the site information entered by the user to the server.

[1597] Step 16:

[1598] Based on the received site information, the server combines generation AI and MR technology to generate data for virtually projecting the house onto the real site.

[1599] Step 17:

[1600] The server transmits the generated MR projection data to the terminal.

[1601] Step 18:

[1602] The terminal uses an MR device to allow the user to project a virtual home onto a real property.

[1603] Step 19:

[1604] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[1605] Step 20:

[1606] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[1607] Step 21:

[1608] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[1609] Step 22:

[1610] The server transmits the generated proposal data to the terminal.

[1611] Step 23:

[1612] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[1613] Example 1

[1614] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1615] Conventional home design systems lacked the functionality to allow users to specifically simulate and virtually experience their desired location, architectural conditions, and detailed design preferences. They also lacked the means to properly predict interior design, site projection, and the timing of long-term maintenance and renovations. This could prevent users from fully realizing their ideal home design, leading to lower satisfaction with the completed home.

[1616] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1617] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a three-dimensional model of the house using a generating AI based on the information input through the input means; virtual reality experience means for converting the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generating AI based on the information input through the interior input means; interior virtual reality experience means for allowing the interior design generated by the design generation means to be virtually experienced in a virtual reality format; site input means for inputting real site information; mixed reality projection means for virtually projecting the house onto the real site by combining generating AI and mixed reality technology based on the information input through the site input means; and prediction means for predicting deterioration of the house and changes in lifestyle, and suggesting appropriate timing for maintenance and renovation.

[1618] This allows users to simulate and virtually experience the detailed design of their desired home. It also allows them to get a detailed understanding of the interior design, projection onto the site, and the timing of long-term maintenance and renovations, which increases satisfaction after the home is completed.

[1619] "Input means" refers to a device or interface that allows a user to input information such as desired location, architectural conditions, and design preferences.

[1620] "Modeling means" refers to a device or software that uses generation AI to generate a three-dimensional model of a house based on information input through the input means.

[1621] The "virtual reality experience means" refers to a device or software that converts the three-dimensional model generated by the modeling means into a virtual reality format, allowing the user to have a virtual experience.

[1622] "Interior input means" refers to a device or interface for inputting desired information regarding the user's interior design and the placement of furniture and home appliances.

[1623] The "design generation means" refers to a device or software that uses generation AI to propose interior designs based on information input through the interior input means.

[1624] The "interior virtual reality experience means" refers to a device or software that allows the interior design generated by the design generation means to be virtually experienced in virtual reality format.

[1625] The "site input means" is a device or interface for inputting actual site information.

[1626] A "mixed reality projection means" is a device or software that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[1627] "Predictive tools" are devices or software that predict deterioration of homes and changes in lifestyles, and suggest appropriate timing for maintenance and renovations.

[1628] MODE FOR CARRYING OUT THE INVENTION

[1629] This invention is a system for allowing users to design and virtually experience the home they desire. This system generates a three-dimensional model of the home based on user input information and provides the user with a virtual experience using virtual reality and mixed reality technologies.

[1630] Hardware and software used

[1631] Hardware:

[1632] Device: The device (e.g., PC, tablet, smartphone) that collects user input and provides the virtual experience.

[1633] Server: A high-performance computer that runs generative AI models and processes data

[1634] VR device: A device that allows users to experience a virtual home (e.g., Oculus Rift, HTC Vive)

[1635] MR device: A device that provides a mixed reality experience (e.g., Microsoft HoloLens)

[1636] software:

[1637] Generative AI model: An AI system that generates 3D models and interior designs based on user-entered information

[1638] Modeling software: Tools for creating 3D models (e.g. Blender, 3ds Max)

[1639] VR software: Tools for converting 3D models into virtual reality formats (e.g. Unity, Unreal Engine)

[1640] MR Software: Tools for converting virtual homes into mixed reality formats

[1641] Database: A system for managing user input information and generated model data.

[1642] Program processing

[1643] 1. User Information Collection

[1644] Users use the terminal to input their desired location, architectural requirements, and design preferences. The terminal then sends this information to the server. For example, if a user wants a "modern, three-story house in central Tokyo," they would enter those detailed requirements.

[1645] 2. Generation of 3D modeling

[1646] The server uses the generative AI model to generate a 3D model of the house based on the received information. For example, a specific 3D model can be generated based on the criteria of "a three-story house with a modern design."

[1647] 3. Creating and experiencing a VR prototype

[1648] The server converts the generated 3D model into a VR format and sends it to the terminal. The terminal then allows the user to experience the virtual home through a VR device. The user puts on the VR device and can walk around inside and outside the building in the virtual space.

[1649] 4. Interior design proposals

[1650] Next, the user inputs their desired information about the interior design, furniture, and appliance placement into the device. For example, if a user requests a natural interior, a large TV, and an open kitchen, this information is sent to the server. The server then uses a generative AI model to generate a detailed interior design and sends the design data to the device. The user can then view the virtual interior design using the VR device.

[1651] 5. Projecting the site using MR technology

[1652] The user inputs real-world site information into the device and sends it to the server. For example, if a user specifies "I want to build on a 500-square-meter site in the suburbs," that information is processed by the server. The server combines a generative AI model with MR technology to generate data for projecting a virtual home onto the real site, which is then sent to the device. The device then uses an MR device to virtually project the home onto the real site, allowing the user to check the lighting, shadows, color scheme, and other aspects in real time.

[1653] 6. Maintenance and renovation forecasts

[1654] The server predicts deterioration of the home and changes in the user's lifestyle. For example, it generates information such as "a simulation of deterioration after 10 years of construction" or "renovation of the children's room is necessary in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovation, and sends it to the device. The device notifies the user of this information, supporting a comfortable home over the long term.

[1655] Examples of concrete examples and prompts

[1656] As a concrete example, if a user requests a "two-story house with a natural design in the suburbs of Tokyo," the processing steps are as follows:

[1657] 1. The user enters desired conditions such as "suburbs of Tokyo," "natural design," and "two-story building" on the terminal.

[1658] 2. The device sends this information to a server, which uses a generative AI model to create a 3D model based on the conditions.

[1659] 3. The server converts the 3D model into a VR format and sends it to the terminal, where the user begins the virtual experience using the VR device.

[1660] 4. If the user requests "wood-style furniture," "eco-friendly appliances," or "a living room with an open ceiling space" as interior design preferences, this information will be collected from the device.

[1661] 5. The server generates a detailed interior design based on the collected information and sends it to the device, where the user can again view the virtual interior in their VR device.

[1662] 6. When the user selects "build on a 200-square-meter lot," the server uses MR technology to generate data for projecting a virtual house onto the real lot and sends it to the device. The user can then check how the light and shadows are reflected on the MR device.

[1663] 7. Finally, the server performs a "10-year deterioration simulation" and a "5-year renovation prediction" and sends the results to the terminal. The terminal then notifies the user of the results.

[1664] Example prompt sentence:

[1665] "As a user, I would like you to design a two-story house with a natural design in the suburbs of Tokyo. Specifically, I would like to use wood-style furniture and incorporate eco-friendly appliances. I would also like a living room with an open ceiling. I plan to build it on a 200-square-meter lot, and I would like to be able to use mixed reality technology to create a virtual experience. I would also like you to propose maintenance and renovations for 10 years from now."

[1666] In this way, the system allows users to imagine the design of their home in detail, experience it in a virtual space, and even create long-term maintenance plans.

[1667] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1668] Step 1:

[1669] The user enters information

[1670] Users use the terminal to input their desired location, architectural requirements, and design preferences. This information includes the desired location, detailed architectural requirements, interior style, etc. If a user wants a "modern three-story home in central Tokyo," they input specific details.

[1671] Input: location, architectural conditions, design preferences

[1672] Output: Detailed data about the user's preferences

[1673] Step 2:

[1674] Send the collected information to a server

[1675] The device transmits the desired information collected from the user to the server, which stores this information and uses it for subsequent data processing.

[1676] Input: User's desired information

[1677] Output: User preference data stored on the server

[1678] Step 3:

[1679] The server generates the 3D model

[1680] The server uses the generative AI model to generate a 3D model of the house based on the information requested by the user. In this process, the AI ​​model generates a design based on the input information and creates 3D model data.

[1681] Input: User preference data

[1682] Data processing: Generating 3D models using generative AI models

[1683] Output: 3D model data

[1684] Step 4:

[1685] Converting 3D models into VR format

[1686] The server converts the generated 3D model data into VR format, allowing users to experience the home in a virtual space.

[1687] Input: 3D model data

[1688] Data processing: Converting 3D models into VR format

[1689] Output: VR format housing data

[1690] Step 5:

[1691] The user experiences VR

[1692] The terminal transmits the VR format house data received from the server to the VR device, providing the user with a virtual experience. The user wears the VR device and visually experiences the inside and outside of the virtual house.

[1693] Input: VR format housing data

[1694] Output: User's virtual experience information

[1695] Step 6:

[1696] Enter your interior design preferences

[1697] Users input their preferences for interior design, furniture, and appliance placement into the terminal, such as "natural interior," "large TV," and "open kitchen."

[1698] Input: Interior design, furniture, and home appliance preferences

[1699] Output: User's interior design preferences data

[1700] Step 7:

[1701] The collected interior preference information is sent to the server.

[1702] The device sends the interior design preferences collected from the user to the server, which receives the information and uses it for subsequent data processing.

[1703] Input: User's interior design preferences

[1704] Output: Interior preference data saved on the server

[1705] Step 8:

[1706] The server generates the interior design

[1707] The server generates a detailed interior design using a generative AI model based on the received interior design preferences.

[1708] Input: Interior design preferences data

[1709] Data processing: Generating interior designs using generative AI models

[1710] Output: Interior design data

[1711] Step 9:

[1712] Convert interior designs into VR format and provide them to users

[1713] The server converts the generated interior design data into VR format and sends it to the terminal, which then uses a VR device to provide the user with a virtual interior experience.

[1714] Input: Interior design data

[1715] Data processing: Converting interior design into VR format

[1716] Output: VR format interior data

[1717] Step 10:

[1718] Users input real site information

[1719] The user enters information about the lot on which they want to build a house into the terminal, such as "a 500-square-meter lot in the suburbs."

[1720] Input: Site information

[1721] Output: User's site information data

[1722] Step 11:

[1723] Send the collected site information to the server

[1724] The terminal sends the site information collected from the user to the server, which uses this information to perform the following processes:

[1725] Input: User's site information

[1726] Output: Site information data stored on the server

[1727] Step 12:

[1728] The server generates data to project the virtual house onto the site.

[1729] The server combines site information with a generative AI model and MR technology to generate data that projects a virtual home onto the real site.

[1730] Input: Site information data

[1731] Data processing: Generating residential projection data using generative AI and MR technology

[1732] Output: Virtual house projection data in MR format

[1733] Step 13:

[1734] Projecting a virtual house onto a real site through an MR device

[1735] The server then sends the generated MR-formatted virtual house projection data to the terminal, which then uses the MR device to virtually project the house onto the user's screen.The user can then use the MR device to check the location of the house, how the light hits it, and the shadows.

[1736] Input: MR format virtual house projection data

[1737] Output: User's projection confirmation information

[1738] Step 14:

[1739] The server predicts the deterioration and renovation of the home.

[1740] The server processes data to predict deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction and predicting renovation needs after five years.

[1741] Input: Housing information, user lifestyle information

[1742] Data processing: Deterioration and renovation prediction using generative AI

[1743] Output: Deterioration simulation data, renovation prediction data

[1744] Step 15:

[1745] Notifying users when maintenance or renovation is required

[1746] The server sends the generated simulation data and prediction data to the terminal, which then notifies the user of appropriate maintenance and renovation timing.

[1747] Input: Deterioration simulation data, renovation prediction data

[1748] Output: Notification information for the user

[1749] In this way, the present invention provides a system that allows users to design a home in the most specific way, virtually experience it, and even plan maintenance and renovations.

[1750] (Application example 1)

[1751] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1752] In home and interior design, users often have difficulty experiencing and confirming their preferences in a concrete and intuitive way. In particular, in physical stores, there are limited ways to check the actual layout and design of interior products before purchasing them, which can lead to lower post-purchase satisfaction. Furthermore, when users want to make revisions to detailed interior designs while experiencing them virtually, the process is cumbersome and they are unable to receive timely feedback.

[1753] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1754] In this invention, the server includes: an input means for inputting a user's desired location, architectural conditions, and design preferences; a modeling means for generating a 3D model of a house using a generative AI based on the information input through the input means; a VR experience means for converting the 3D model generated by the modeling means into a VR format so that the user can have a virtual experience; an interior input means for inputting the user's desired information regarding the interior design and layout of furniture and home appliances; a design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; an interior VR experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; a site input means for inputting information about a real site; an MR projection means for virtually projecting a house onto the real site based on the information input through the site input means by combining the generative AI and MR technology; and an experience means for allowing the user to wear smart glasses or a head-mounted display at an interior shop and experience the interior in both real and virtual spaces. This allows the user to virtually experience interior products at the interior shop and intuitively and concretely check the layout and design.

[1755] "Input means" refers to the means by which a user inputs desired location, architectural conditions, and design preferences into a terminal.

[1756] The "modeling means" is a means for generating a 3D model of a house using a generation AI based on information input through the input means.

[1757] The "VR experience means" is a means for converting the 3D model generated by the modeling means into a VR format, allowing the user to have a virtual experience.

[1758] The "interior input means" is a means for inputting desired information regarding the user's interior design and the placement of furniture and home appliances into the terminal.

[1759] The "design generation means" is a means for proposing an interior design using a generation AI based on the information input through the interior input means.

[1760] The "interior VR experience means" is a means that allows the interior design generated by the design generation means to be virtually experienced in VR format.

[1761] The "site input means" is a means for inputting actual site information into the terminal.

[1762] The "MR projection means" is a means for virtually projecting a house onto a real site by combining generation AI and MR technology based on information input through the site input means.

[1763] The "experience means" refers to a means by which users can wear smart glasses or a head-mounted display in an interior design shop and experience the interior design in both real and virtual spaces.

[1764] A "prediction method" is a means of predicting when a home will deteriorate or require maintenance based on the user's lifestyle and environmental conditions.

[1765] The system for carrying out the present invention is composed of multiple components that allow a user to virtually experience the home design and interior design they desire. The system includes the following elements:

[1766] First, the user terminal provides an input means for the user to input their desired location, architectural conditions, and design preferences. The input means includes an interface through which the user can input their preferences, such as "a three-story house with a modern design in central Tokyo."

[1767] The server then uses the collected information to generate a 3D model of the house using a generative AI model. During the generation process, for example, the server sends the following prompt to the generative AI:

[1768] "Create a 3D model of a modern, three-story house in central Tokyo."

[1769] The server converts the generated 3D model into VR format and sends it to the user's device, which then allows the user to experience the virtual home through a VR device. This process uses software such as OpenVR and PyOpenGL.

[1770] Next, an interior design input interface is provided for users to input their preferences regarding interior design, furniture, and appliance placement. Users can use this interface to input preferences such as "natural interior," "large TV," and "open kitchen."

[1771] The server then uses the collected interior information to propose a detailed interior design using generative AI again. This time, prompts are used as well. For example:

[1772] "Create a 3D model of a living room with natural decor, a large TV, and an open kitchen."

[1773] The generated interior design is then converted back into VR format and sent to the user's device, where the user can view the virtual interior design through the interior VR experience.

[1774] Additionally, users can use the site entry tool to input real-world site information, for example, a "500 square meter site in the suburbs."

[1775] Based on this information, the server combines generative AI and MR technology to generate data for projecting a virtual house onto the real site. This MR projection method allows users to use an MR device to virtually project the house onto the real site, allowing them to realistically check how the light hits it, the shadows, the color scheme, and more.

[1776] The server also uses predictive tools to forecast deterioration of the home and changes in lifestyle. For example, predictions include simulations of deterioration after 10 years of construction and the need to renovate the children's room in five years.

[1777] Finally, an experience tool is provided to users in the interior design shop's physical store. This tool allows users to wear smart glasses or a head-mounted display and experience interior design products in both real and virtual spaces. This allows users to check the actual layout and design of the products before purchasing them.

[1778] As described above, this system connects multiple components to enable users to have a concrete and intuitive virtual experience of home and interior design, ultimately helping them make the best choice.

[1779] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1780] Step 1:

[1781] An input method is used to input the user's desired location, architectural conditions, and design preferences. In this step, a user terminal is used, and the user inputs information such as "a three-story house with a modern design in central Tokyo" through the interface. The input information is converted into a data format within the terminal and sent to the server.

[1782] Step 2:

[1783] The server receives the information entered by the user. It then uses the generative AI model to generate a 3D model of the house. In this step, it sends the prompt "Please create a 3D model of a modern, three-story house located in central Tokyo" to the generative AI. It then receives the 3D model data returned by the generative AI and converts it into a VR format.

[1784] Step 3:

[1785] The server sends the generated 3D model data in VR format to the user's terminal. The user's terminal uses a VR device (e.g., a head-mounted display) to provide the user with a virtual home experience. The user can wear the VR device and virtually walk around inside and outside the building.

[1786] Step 4:

[1787] The user uses the interior input means to input desired information regarding the interior design, furniture, and placement of home appliances. In this step, the user inputs the desired interior style and specific furniture items, which are then sent to the server by the terminal. For example, information such as "natural interior," "large TV," and "open kitchen" can be input.

[1788] Step 5:

[1789] The server uses the generative AI model to propose a detailed interior design based on the information input through the interior input means. At this time, it sends a prompt to the generative AI: "Please create a 3D model of a living room with natural interior, a large TV, and an open kitchen." The interior design data returned by the generative AI is converted into VR format.

[1790] Step 6:

[1791] The server transmits the generated VR data of the interior design to the user terminal. The user terminal provides the user with a virtual interior experience using the interior VR experience means. The user can then use the VR device again to virtually check the interior design and walk around inside the house.

[1792] Step 7:

[1793] The user uses a site input means to input information about a real site. For example, the user inputs information about a "500 square meter site in the suburbs" and transmits it to the server via the terminal.

[1794] Step 8:

[1795] Based on the information entered through the site input means, the server combines AI generation and MR technology to generate data for virtually projecting the house onto the real site. In doing so, it generates detailed data such as lighting, shadows, and color schemes so that the virtual house is properly projected onto the real site.

[1796] Step 9:

[1797] The server transmits the generated MR data to the user's device, which then projects the virtual house onto the real site, allowing the user to see how the house will look in the real environment.

[1798] Step 10:

[1799] The server predicts deterioration of the home and changes in lifestyle, and suggests appropriate timing for maintenance and renovation. Based on this prediction method, it generates simulation results, such as "simulation of deterioration after 10 years of construction" or "children's room renovations are necessary in 5 years," and sends these to the user's device.

[1800] Step 11:

[1801] Users can wear smart glasses or a head-mounted display in an interior design shop to experience the interior design in both real and virtual space. This experience allows users to check the actual layout and design of the product before purchasing it, allowing them to make the best choice.

[1802] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1803] This invention is a system for designing the home desired by the user in the most specific way and providing a virtual experience, and by combining it with an emotion engine, the system's suggestions and display content can be adjusted according to the user's emotional state. This system is composed of the following main components: input means, modeling means, VR experience means, interior input means, design generation means, interior VR experience means, site input means, MR projection means, prediction means, and the emotion engine.

[1804] 1. User Information Collection Practices

[1805] The terminal displays a data entry form for users to enter their desired location, construction conditions, and design preferences. Users enter information such as location (Central Tokyo), construction conditions (Modern design, three-story house), and design preference (Simple and modern).

[1806] 2. Generate 3D modeling

[1807] The device sends the information entered by the user to a server, which then uses generative AI to generate a 3D model of the home that meets the user's needs.

[1808] 3. Creating and experiencing a VR prototype

[1809] The server converts the generated 3D model into VR format and sends the data to the terminal, which then allows the user to experience the virtual home through a VR device. The user can wear the VR device and virtually walk around inside and outside the building.

[1810] 4. Interior design proposals

[1811] The terminal displays an interior input means for users to input their desired conditions regarding interior design, furniture, and appliance placement, and users can enter information such as "natural-style interior," "large TV," and "open kitchen."

[1812] The server uses generative AI to generate a detailed interior design based on the collected information and sends the design data to the device, where the user can then view the virtual interior design using a VR device.

[1813] 5. Projecting the site using MR technology

[1814] The user inputs real-world site information and sends it to the server via their device. For example, the user might specify, "I want to build on a 500-square-meter site in the suburbs."

[1815] Based on the received site information, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site, and sends it to the terminal. The terminal then uses an MR device to virtually project the house onto the real site, allowing the user to realistically check how the light hits it, the shadows, color scheme, and more.

[1816] 6. Maintenance and renovation forecasts

[1817] The server predicts the deterioration of the home and changes in the user's lifestyle, for example, simulating deterioration after 10 years of construction or predicting whether the children's room will need renovation in five years.

[1818] The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device, which then notifies the user of this information, helping them maintain a comfortable home over the long term.

[1819] 7. Adjusting recommendations with an emotion engine

[1820] While the user is viewing a virtual experience or interior design, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine analyzes the user's facial expressions, voice, gestures, and other biometric information to determine whether the user is satisfied or dissatisfied.

[1821] The server adjusts its suggestions and displays based on feedback from the emotion engine. For example, if the user shows a dissatisfied expression, the server can quickly generate alternatives and suggest new interior designs or architectural conditions to the user.

[1822] In this way, a system incorporating an emotion engine can provide users with a more personalized experience, enabling optimal home design based on their own emotions. The introduction of an emotion engine will greatly improve user satisfaction and further promote the realization of their ideal home.

[1823] The processing flow will be explained below.

[1824] Step 1:

[1825] The terminal displays a data entry form where the user can input their desired location, architectural requirements, and design preferences.

[1826] Step 2:

[1827] Users input information about the home they desire (e.g., location "central Tokyo," construction requirements "a three-story home with a modern design," design preferences "simple and modern") into the terminal.

[1828] Step 3:

[1829] The terminal transmits the information entered by the user to the server.

[1830] Step 4:

[1831] The server uses generative AI based on the received information to generate a 3D model of a home that meets the user's requirements.

[1832] Step 5:

[1833] The server converts the generated 3D model into VR format and sends the data to the terminal.

[1834] Step 6:

[1835] The terminal prepares the VR device and guides the user to put it on.

[1836] Step 7:

[1837] Users put on a VR device and experience a virtual home.

[1838] Step 8:

[1839] The terminal displays an interior input form that allows the user to enter information about the desired interior design and the placement of furniture and appliances.

[1840] Step 9:

[1841] Users input their desired interior design (e.g., natural-style interior, large TV, open kitchen, etc.) into the terminal.

[1842] Step 10:

[1843] The terminal transmits the interior information input by the user to the server.

[1844] Step 11:

[1845] The server uses generative AI to generate a specific interior design based on the received information.

[1846] Step 12:

[1847] The server converts the generated interior design into VR format and sends the data to the terminal.

[1848] Step 13:

[1849] The terminal again uses a VR device to allow the user to experience a virtual interior.

[1850] Step 14:

[1851] The terminal displays a site input form for the user to input real site information (site size, location, sunlight conditions, etc.).

[1852] Step 15:

[1853] The user inputs real-world site information into the terminal.

[1854] Step 16:

[1855] The terminal transmits the site information entered by the user to the server.

[1856] Step 17:

[1857] Based on the site information it receives, the server combines generative AI and MR technology to generate data for virtually projecting the house onto the real site.

[1858] Step 18:

[1859] The server transmits the generated MR projection data to the terminal.

[1860] Step 19:

[1861] The terminal uses an MR device to allow the user to virtually project a house onto a real-world site.

[1862] Step 20:

[1863] Through the MR device, users can check how the light hits the house, the shadows, the color scheme, and more.

[1864] Step 21:

[1865] The server collects and analyzes data to predict deterioration of the home and changes in the user's lifestyle.

[1866] Step 22:

[1867] Based on the prediction results, the server uses generative AI to generate data that suggests the appropriate timing and content of maintenance and renovations.

[1868] Step 23:

[1869] The server transmits the generated proposal data to the terminal.

[1870] Step 24:

[1871] The device notifies the user of the received proposal data and informs them of the necessary maintenance and renovations.

[1872] Step 25:

[1873] The device uses an emotion engine to analyze the user's emotional state in real time while they are experiencing the virtual world.

[1874] Step 26:

[1875] The emotion engine determines the user's emotional state based on biometric information such as facial expressions, voice, and gestures.

[1876] Step 27:

[1877] The server adjusts the suggestions and display content based on feedback from the emotion engine.

[1878] Step 28:

[1879] If the emotion engine detects a user's dissatisfaction, the server immediately generates an alternative and sends it to the device.

[1880] Step 29:

[1881] The device presents alternatives to the user and suggests new interior designs and architectural requirements.

[1882] Example 2

[1883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1884] Conventional home design systems were unable to fully reflect the user's wishes and emotions, and virtual and interior experiences were limited. It was also difficult for users to predict when future renovations or maintenance would be required. There was also no real-time proposal adjustment that took the user's emotions into account, which led to problems with the quality of the user experience.

[1885] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1886] In this invention, the server includes: input means for inputting the user's desired location, architectural conditions, and design preferences; modeling means for generating a 3D model of the house using a generative AI based on the information input through the input means; virtual experience means for converting the 3D model generated by the modeling means into a VR format to allow the user to have a virtual experience; interior input means for inputting the user's desired information regarding the interior design and the arrangement of furniture and home appliances; design generation means for proposing an interior design using a generative AI based on the information input through the interior input means; interior virtual experience means for allowing the user to virtually experience the interior design generated by the design generation means in a VR format; site input means for inputting information about the actual site; mixed reality projection means for virtually projecting the house onto the actual site based on the information input through the site input means by combining generative AI and mixed reality technology; prediction means for predicting deterioration of the house and changes in lifestyle and proposing appropriate timing for maintenance and improvements; and an emotion engine for analyzing the user's emotional state in real time and adjusting the content of the proposals. This will enable specific home designs that reflect the user's wishes and emotions, improving the quality of virtual and interior experiences, and even making it possible to accurately predict the timing of future renovations and maintenance.

[1887] "User" refers to an individual who uses this system to receive home design and interior design proposals and have a virtual experience.

[1888] "Input means" refers to the interface that allows users to input their desired location, architectural conditions, and design preferences.

[1889] "Modeling means" refers to the function of generating a 3D model of a house using generative AI based on information entered by the user.

[1890] "Virtual experience means" refers to the function of converting the generated 3D model into VR format, allowing users to experience the house in a virtual space.

[1891] "Interior input means" refers to an interface that allows users to input desired information regarding interior design and the placement of furniture and home appliances.

[1892] "Design generation means" refers to a function that uses generation AI to propose interior designs based on information input through the interior input means.

[1893] "Interior virtual experience means" refers to a function that provides users with the generated interior design in VR format, allowing them to experience it in a virtual space.

[1894] "Site input means" refers to an interface for a user to input real site information.

[1895] "Mixed reality projection means" refers to a function that combines generative AI and mixed reality technology to virtually project a house onto a real site based on information input through the site input means.

[1896] "Predictive tools" refers to functions that predict deterioration of the home and changes in lifestyle, and suggest appropriate timing for maintenance and renovations.

[1897] The "emotion engine" refers to a function that analyzes the user's emotional state in real time and adjusts the content of suggestions.

[1898] MODE FOR CARRYING OUT THE INVENTION

[1899] An embodiment of this invention is a system that specifically designs a home desired by a user, provides a virtual experience, and further combines an emotion engine to make suggestions based on the user's emotions. This system is composed of the following main components: input means, modeling means, virtual experience means, interior input means, design generation means, interior virtual experience means, site input means, mixed reality projection means, prediction means, and emotion engine.

[1900] Collection of User Information

[1901] The terminal displays a data entry form for users to enter their desired location, architectural conditions, and design preferences. For example, users might enter information such as "Central Tokyo," "Modern design three-story home," and "Simple and modern."

[1902] 3D modeling generation

[1903] The device sends the information entered by the user to the server. The server uses the received information to generate a 3D model of the house using a generative AI model. For example, GPT-4 can be used as the generative AI model. The following prompt sentence is used as input:

[1904] User Information:

[1905] Location: Central Tokyo

[1906] Building requirements: Modern design, three-storey house

[1907] Design preference: Simple and modern

[1908] Generate a 3D model of the house according to the user's wishes.

[1909] Creating and experiencing VR prototypes

[1910] The server converts the generated 3D model into a VR format and sends the data to the terminal, which then provides the user with a virtual experience through a VR device such as Oculus Rift. The user can then use the VR device to virtually explore the inside and outside of the building.

[1911] Interior design proposals

[1912] The device displays an interior input tool for the user to input their desired information regarding interior design, furniture, and appliance placement, prompting them to enter information such as "natural-style interior," "large TV," and "open kitchen." The server uses generative AI based on the collected information to generate a detailed interior design and sends the design data to the device. The user then uses the VR device again to view the virtual interior design.

[1913] Projection of the site using MR technology

[1914] The user inputs real-world site information and sends it to the server via their device. For example, they might specify, "I want to build on a 500-square-meter site in the suburbs." Based on the site information received, the server combines generative AI and mixed reality technology to generate data for virtually projecting the house onto the real site, and sends that data to the device. The device then uses an MR device such as Microsoft HoloLens to virtually project the house onto the real site.

[1915] Maintenance and renovation forecasts

[1916] The server predicts deterioration of the home and changes in the user's lifestyle. Specifically, it makes predictions such as "simulating deterioration after 10 years of construction" and "renovating the children's room in five years." The server generates data suggesting the appropriate timing and content of maintenance and renovations, and sends it to the device. The device notifies the user of this information, helping them to live comfortably in the home over the long term. ...

Claims

1. An input means for the user to input their desired location, architectural requirements, and design preferences; a modeling means for generating a 3D model of a house using a generation AI based on the information input through the input means; a VR experience means for converting the 3D model generated by the modeling means into a VR format so that a user can have a virtual experience; an interior input means for inputting desired information regarding the user's interior design and the arrangement of furniture and home appliances; a design generation means for proposing an interior design using a generation AI based on the information input through the interior input means; an interior VR experience means for enabling a virtual experience of the interior design generated by the design generation means in a VR format; a site input means for inputting actual site information; an MR projection means for virtually projecting a house onto a real site by combining a generation AI and MR technology based on the information input through the site input means; A predictive tool that predicts deterioration of homes and changes in lifestyles, and suggests appropriate timing for maintenance and renovations. A system including:

2. 2. The system according to claim 1, wherein the prediction means predicts the timing of deterioration of the house or the need for maintenance based on the user's lifestyle and environmental conditions.

3. 2. The system according to claim 1, wherein the VR experience means includes a function that allows a user to input modifications to architectural conditions and design during the virtual experience, and that reflects the modifications in real time.

Images