system

Abstract

We provide the system. [Solution] A means for users to upload image data of their rooms, A means for analyzing the aforementioned image data to extract the characteristics of the room, A means for generating interior design proposals using a generative artificial intelligence model based on extracted feature data, A means of presenting the generated interior design proposals to the user, A means of obtaining and presenting information on products related to the selected interior design plan, A means for accepting the purchase selection of the aforementioned product, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, in order for general customers to carry out room interior coordination and purchase related furniture, specialized knowledge is required, and it takes a lot of labor and time. Also, in order for customers to obtain a coordination plan suitable for their specific room, they need to consult a professional consultant or spend a lot of time researching by themselves. As a result, many customers are currently unable to realize an ideal interior.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a system that allows users to easily upload image data of a room and generates an optimal interior design plan for that room using AI technology. Specifically, the system includes means for receiving image data of a room from a user, means for analyzing the image data to extract the characteristics of the room, means for generating an interior design plan using a generation AI model, means for presenting the generated plan to the user, means for presenting information on related products based on the design plan selected by the user, and means for selecting to purchase them, thereby providing a system that enables users to quickly and easily coordinate their interiors and purchase the necessary furniture.

[0006] A "user" is a regular customer who uses the system to coordinate the interior design of their room.

[0007] "Image data" refers to photographs or floor plans that users upload to the system to show the condition of their room.

[0008] "Analysis" is the process of extracting room features from image data, and includes processes for determining color, shape, arrangement, etc.

[0009] "Features" refer to information extracted through analysis, such as the room's colors, shape, and existing furniture arrangement.

[0010] A "generative artificial intelligence model" is a computational model that utilizes AI technology to generate optimal interior design proposals using room characteristic data as input.

[0011] An "interior design proposal" is a design suggestion that includes combinations of furniture and decorative items, as well as their arrangement, based on the characteristics of the room, as proposed by a generated AI model.

[0012] "Related products" refer to items such as furniture and accessories that are presented with the assumption that the user will purchase them based on the proposed interior design plan.

[0013] "Presentation" refers to the act of displaying information or suggestions to the user through a system.

[0014] "Purchase selection" refers to the action of a user choosing the items they wish to buy from the related products presented and proceeding with the purchase process. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The interior design system of this invention begins with the user uploading photos and floor plans of their room to the system. This data is transmitted to the server via the user's terminal.

[0037] The server first preprocesses the received image data in order to analyze it. This preprocessing includes resizing and denoising the images. Next, the server uses computer vision technology to extract features from the image data, such as the color of the room walls, the type of flooring, and the location of the windows.

[0038] Next, the server inputs the extracted feature data into a generating AI model to generate optimal interior design proposals. This generating artificial intelligence model can generate multiple design proposals, taking into account color harmony, furniture style, and efficient use of space.

[0039] The generated interior design proposals are sent from the server to the user's terminal and presented to the user through the interface. The user can review the presented proposals and choose their preferred style.

[0040] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The search results are then filtered based on price, style, ratings, etc., and presented to the user as a list of suggestions.

[0041] Users can select the items they want to purchase from this suggestion list and proceed to the purchase process on the e-commerce site via the link. This allows users to smoothly complete the entire process, from selecting furniture to purchasing it, online without having to visit a physical store.

[0042] As a concrete example, consider a scenario where a user uploads a photo of their living room to the system. The server extracts the characteristics of the space, such as white walls, wooden flooring, and large windows. The generative AI model then generates a simple Scandinavian-style interior design proposal, suggesting items like a light-colored sofa, a wood-toned coffee table, and simple curtains. Based on the suggestions, the user can select the necessary furniture and easily purchase it online. Throughout this entire process, the user can effortlessly achieve the perfect interior design for their room.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user uses their device to select and upload photos and floor plans of rooms through the system interface. The device then prepares to send these selected image data to the server.

[0046] Step 2:

[0047] The server receives image data sent from the terminal. After receiving the data, it preprocesses it, which includes resizing and noise reduction.

[0048] Step 3:

[0049] The server uses computer vision technology to extract room features from pre-processed image data. These features include wall color, floor material, window placement, and existing furniture arrangement.

[0050] Step 4:

[0051] The server inputs feature data into an artificial intelligence model to generate interior design proposals based on the room's characteristics. The generated proposals take into account color harmony, furniture style, and efficient placement.

[0052] Step 5:

[0053] The server sends the generated interior design proposals to the user's terminal and displays them. The terminal displays these proposals on its interface, allowing the user to browse the designs and select their preferred style.

[0054] Step 6:

[0055] Based on the user's selected outfit, the server searches for relevant furniture and accessory product information from e-commerce sites. The server then creates a product list to suggest to the user based on the search results.

[0056] Step 7:

[0057] The server sends the suggested product list to the user's device, and the user reviews the list. The user selects the products they like and accesses the purchase page on the e-commerce site from their device.

[0058] Step 8:

[0059] The user proceeds with the purchase process on the e-commerce site and buys the selected product. Payment information and shipping settings are handled according to the instructions on the e-commerce site until the purchase is completed.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In selecting designs for modern living spaces, consumers require considerable knowledge and time to compare options and achieve highly personalized designs. Furthermore, the effort involved in quickly and effectively searching for and purchasing a wide variety of interior products is also a problem. To address these challenges, a system is needed that easily and efficiently proposes designs suitable for living spaces, and allows for the easy selection and purchase of related products based on those proposals.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for the user to transmit image data of the living space using a communication device, means for preprocessing and standardizing the image data, and means for using a machine learning algorithm to analyze the image data and extract the attributes of the living space. This enables the user to efficiently find the design they want and smoothly select and purchase related products.

[0065] A "user" is an individual or legal entity that uses this system to obtain design suggestions for living spaces and information on related products.

[0066] "Residential space" refers to the indoor space used by individuals or corporations for their daily lives, and this system is the subject of design proposals.

[0067] "Image data" refers to digital information files that visually record living spaces, and it is the input data for this system.

[0068] A "communication device" is an electronic device used to transmit data, and is a means used when a user sends image data of their living space to a server.

[0069] "Preprocessing" refers to the initial processing performed on image data, which is a technical process to standardize the data and reduce noise prior to analysis.

[0070] A "machine learning algorithm" is a computational method used by computers to learn patterns from data, and in this system, it is a method used to analyze and extract attributes of living spaces.

[0071] A "knowledge generation model" is an AI technology that learns from large datasets and generates new information and suggestions. In this system, it is used to generate design suggestions.

[0072] "Design proposals" refer to creative ideas generated by the server regarding the arrangement or style of interiors suitable for a living space.

[0073] "Items" refers to products and goods related to design proposals for living spaces that are provided to users through this system.

[0074] This interior design system allows users to easily obtain design proposals for their living spaces. First, the user takes or selects image data of their living space using their own device and uploads it to the server via a communication device. The uploaded image data undergoes preprocessing and standardization on the server. This preprocessing includes image resizing and noise reduction.

[0075] Subsequently, the server uses machine learning algorithms to analyze the image data and extract attributes of the living space. This process utilizes computer vision libraries (e.g., OpenCV and TENSORFLOW®) to collect information such as wall color, flooring material, and window location. The extracted attribute data is then input into a knowledge generation model (e.g., a generative AI model).

[0076] The server generates design suggestions by passing prompt statements to the AI ​​model, which then utilizes relevant knowledge to produce design proposals. An example of a prompt statement is, "Analyze a photo of a living room and propose Scandinavian-style interior design ideas." Based on this prompt, the model outputs multiple design suggestions suitable for the living space.

[0077] The generated design proposals are sent from the server to the user's terminal, where the user can review the proposals and select their preferred design. After the user selects a design proposal, the server retrieves information on related items through the e-commerce site and presents it to the user. The user can then select the items they wish to purchase from the proposed items and proceed with the online purchase process. This entire process allows users to efficiently realize their interior design for their living spaces.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user takes a photo of their living space using their device or selects an existing image. This image data is uploaded to the server using a communication device. The input is the selected image file, and the output is the image data sent to the server. Specifically, the user launches the application on their device, selects an image according to the instructions, and presses the upload button.

[0081] Step 2:

[0082] The server performs preprocessing on the received image data. Specifically, it standardizes the image size and removes noise using filtering techniques. The input is the unprocessed image data sent by the user, and the output is the clean image data after preprocessing. This process ensures accurate subsequent feature extraction.

[0083] Step 3:

[0084] The server uses machine learning algorithms to analyze preprocessed image data and extract attributes of the living space. The input is preprocessed image data, and the output is extracted attribute data. Specifically, the server calls a computer vision library and analyzes pixel information in the image to detect wall color, flooring material, window positions, etc.

[0085] Step 4:

[0086] The server inputs the extracted attribute data into a knowledge generation model to generate design proposals. At this time, prompt statements are used to instruct the AI ​​model on specific design themes. The input consists of attribute data and prompt statements, and the output is multiple candidate design proposals. For example, the server sends the prompt "Analyze a photo of a living room and propose a Scandinavian-style interior design plan."

[0087] Step 5:

[0088] The generated design proposals are sent from the server to the user's terminal. The user reviews the proposals through the interface on their terminal and selects their preferred design. The input is the design proposals sent from the server, and the output is the design selected by the user. Specifically, the user scrolls through the displayed proposal images and confirms their selection by pressing a selection button.

[0089] Step 6:

[0090] The server searches for items related to the design proposal selected by the user and retrieves that information from e-commerce sites. The input is the design selected by the user and the category information of the related items, and the output is a list of suggested items. Specifically, the server uses APIs to request product prices and review information from multiple e-commerce sites.

[0091] Step 7:

[0092] Users can select items they wish to purchase from a presented list. After selection, users proceed with the online purchase process using a link on their device. The input is the user's item selection, and the output is the progress towards the purchase of the selected items. Specifically, the user checks the details page for each item and completes the payment process by clicking the purchase button.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] Modern consumers demand quick and accurate suggestions when choosing interior design products in a physical space, but traditional methods are time-consuming and labor-intensive. This is especially true in brick-and-mortar stores, where providing prompt and personalized interior design suggestions tailored to customer needs is difficult, hindering immediate purchase decisions.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes a unit for users to upload image data of a room, a unit for analyzing the image data and extracting the characteristics of the room, and a unit for generating interior layout plans using a generated information processing model based on the extracted characteristic data. This makes it possible to immediately propose the optimal interior layout plan when serving customers in a physical store, using a wearable visual enhancement device within the store.

[0098] A "user-uploadable room image data unit" is a system component that has the function of allowing users to transmit visual information of their own room or any physical space in digital format.

[0099] A "unit that analyzes image data to extract room characteristics" refers to an algorithm or device that processes visual information and extracts useful information by identifying the features and components of a space.

[0100] A "generative information processing model" refers to artificial intelligence or learning algorithms that perform advanced optimization and generate layout proposals based on input data.

[0101] A "unit for generating interior layout plans" is a system element that has the function of constructing an optimal interior design tailored to the user's preferences and the purpose of the space, based on pre-analyzed spatial data.

[0102] A "visual enhancement device" is a device that assists a user's visual information and can overlay digital information on it. This refers to wearable devices such as smart glasses.

[0103] This invention is a system that enables quick and appropriate interior design proposals in physical stores. The system is configured as follows:

[0104] The server receives image data of the room transmitted from a visual enhancement device worn by the user within the store. The image data is captured using a device such as smart glasses and sent to the server. The server analyzes the received image data and extracts its characteristics. This is done using software that applies computer vision technology, specifically the OpenCV library. This allows for the analysis of wall color schemes, furniture arrangement, and spatial shape.

[0105] Based on the analyzed characteristic data, the server automatically generates interior layout plans using a generative information processing model. This process employs a generative AI model and machine learning frameworks such as TensorFlow and PyTorch. As a result, multiple interior layout plans are generated, taking into account specific themes and customer preferences.

[0106] The generated placement options are displayed on the user's visual enhancement device. This user interface was developed using Unity and ARKit, allowing the user to see the optimal placement options in real time.

[0107] As a concrete example, if a user sends data using a visual enhancement device to the living room section of a store, the server can present a simple Scandinavian-style interior layout plan. In this case, an example of a prompt message might be "Living room / Scandinavian style / Warm-toned wooden furniture." This allows the user to quickly and easily make a purchase decision without hesitation.

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The user captures image data of the room using a visual enhancement device they are wearing. This image data is transmitted to the server via a terminal. The input is visual information of the real space, and the server receives the image data as output. This supplies the server with the necessary image information.

[0111] Step 2:

[0112] The server analyzes the received image data. Here, computer vision techniques are used to denoise and resize the images. Specifically, the OpenCV library is used to remove noise from the images and convert them to an appropriate resolution. The input is the image data obtained in step 1, and the output is the denoised, analyzable image data.

[0113] Step 3:

[0114] The server extracts room characteristics from the analyzed image data. Specifically, it applies algorithms to identify spatial features such as color and shape. The input is the image data processed in step 2, and the output is characteristic data such as the room's color and furniture arrangement.

[0115] Step 4:

[0116] The server inputs the extracted characteristic data into a generative information processing model to generate interior layout proposals. During this process, a generative AI model using TensorFlow or PyTorch is executed. The input is the characteristic data obtained in step 3, and the output is multiple interior layout proposals.

[0117] Step 5:

[0118] The server displays the generated interior layout plan on the display of the user's visual augmentation device. The user can then visually confirm this. The input is the interior layout plan generated in step 4, and the output is the graphical display information shown on the visual augmentation device.

[0119] Step 6:

[0120] The user selects their preferred interior layout from the presented options and proceeds to the purchasing process. The selected layout is sent to the server, where relevant product information is retrieved and presented to the user. The input is the user's selection of the layout presented in step 5, and the output is information on purchasable products. This sequence of actions allows the user to immediately proceed to purchasing.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] This invention is a system that proposes interior design using image data of a room provided by the user, taking into account the user's emotional state. First, the user uploads photos and floor plans of the room to the system via a terminal. This image data is then transmitted from the user's terminal to the server.

[0123] The server uses computer vision technology to process the image data received from the terminal, extracting features such as wall color, floor material, and room shape. Based on this, the artificial intelligence model generates interior design proposals. These proposals take into account the colors, layout, and style of the interior.

[0124] Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions. The terminal uses the user's camera and microphone to input the user's facial expressions and voice into the emotion engine in real time. The server analyzes the input data and recognizes the user's emotional state. This emotional information is used to adjust the interior design proposal.

[0125] During the adjustment process, the suggested interior color palette and style are dynamically changed according to the user's emotional state. For example, if the user wants to relax, the generating AI model will prioritize suggesting interior designs with calming colors. Conversely, if the user is in an active emotional state, bright and vibrant colors will be suggested.

[0126] The generated interior design proposals are presented to the user on their device. The user can view the proposed designs and select the one they like. Based on the design proposal selected by the user, the server searches for relevant product information from e-commerce sites and displays it as a list of suggestions for the user. The user selects the products they wish to purchase from this list and proceeds with the purchase process through the e-commerce site.

[0127] For example, if the emotion engine determines that the user desires a calm atmosphere, the server, through its generative AI model, will propose an interior design plan based on cool colors such as blue and green. The user can then purchase furniture selected from the list according to this proposal, easily creating their ideal room. In this way, the present invention provides personalized interior design that responds to the user's emotions.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The user selects photos and floor plans of a room on their device and uploads them through the system interface. The device then prepares to send this data and sends the image data to the server.

[0131] Step 2:

[0132] The server receives the transmitted image data and begins analysis using computer vision technology. The analysis includes processes to extract room features such as wall color, furniture arrangement, and floor material from the image.

[0133] Step 3:

[0134] The server inputs the extracted feature data into an artificial intelligence model to generate optimal interior design proposals for the room. The generated proposals take into account color harmony and spatial efficiency.

[0135] Step 4:

[0136] The user uses their device to input facial expressions and voice into the emotion engine via the camera and microphone to acquire emotion data. The device then sends this data to the server.

[0137] Step 5:

[0138] The server analyzes the received facial and voice data to recognize the user's emotional state. Specifically, it evaluates emotions such as joy, sadness, and calmness.

[0139] Step 6:

[0140] The server takes the user's emotional state into consideration and adjusts the generated interior design suggestions accordingly. For example, if the user indicates a desire to relax, the server will adjust the colors to soft, warm tones.

[0141] Step 7:

[0142] The adjusted interior design proposals are sent from the server to the user's terminal and presented on the interface. The user can view multiple proposals and choose their preferred one.

[0143] Step 8:

[0144] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The server then creates a list of suggested products and sends it to the user's device.

[0145] Step 9:

[0146] Users can view a product list on their device and select the items they wish to purchase. They can then proceed with the purchase process on the e-commerce site via the purchase link for their selected items.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] Conventional interior design systems have struggled to flexibly respond to users' emotions and preferences, making it difficult to provide appropriate suggestions tailored to individual needs. Furthermore, while users desire a system that allows for interior adjustments that take their emotional state into consideration, suitable technology has not yet existed.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] In this invention, the server includes means for a user to transmit image data of a living space via an input device, a device for analyzing the image data and extracting features of the living space, and means for creating an interior layout plan using an artificial intelligence model generated based on the extracted feature data. This makes it possible to propose personalized interior coordination that responds to the user's emotions.

[0152] "Living space" refers to the indoor area where a user lives, and is a concept that includes rooms or parts of a house that are the subject of interior design.

[0153] An "input device" is a device used by a user to supply information to a system, and refers to electronic terminals such as smartphones, tablets, and computers.

[0154] "Image data" refers to digital information in which a user visually records the state of their living space, and includes photographs and drawing data.

[0155] A "server" is the central information processing device of a system, and its role is to collect, analyze, store, and distribute data.

[0156] A "generating artificial intelligence model" is a computer program equipped with a process for analyzing and generating data through learning, and is a technology used to propose optimal interior layout plans to users.

[0157] An "interior design plan" is a blueprint for the composition and layout of interiors in a living space, and includes proposals that encompass elements such as color, material, shape, and arrangement.

[0158] "Items" refer to parts, tools, furniture, etc., used to make up the interior of a living space, and are items that can be purchased as concrete products.

[0159] "Emotional state" refers to information that indicates a user's psychological reactions and emotional tendencies, and is an element used in adjusting interior design.

[0160] "Dynamic adjustment" refers to a process in which the system changes and optimizes results and suggestions in real time based on user input and circumstances.

[0161] The system based on this invention is primarily realized through three entities: a server, a terminal, and a user.

[0162] First, the user sends image data of their living space to the system via a device. This device can be a smartphone, tablet, or computer, and the images are sent over the internet, with the data being sent from the device to the server.

[0163] The server analyzes the received image data and extracts features of the living space. For this purpose, the server utilizes computer vision technology. Specifically, information processing libraries such as TensorFlow and OpenCV are used to extract features such as wall color, floor material, and room shape.

[0164] Next, the server utilizes an artificial intelligence model generated based on the extracted feature data to produce interior layout proposals. The generating AI model refers to a training database and constructs proposals considering the interior's colors, layout, and style. The interior coordination proposals are customized to suit the user's individual preferences.

[0165] Furthermore, the device uses the user's camera and microphone to acquire their emotional state. The emotion engine analyzes the user's facial expression and voice data to determine their emotional state, such as whether they are relaxed or active.

[0166] The server takes this emotional state into account and dynamically adjusts the interior design layout. For example, if the user desires a calm atmosphere, the server, through a generative AI model, will suggest and present interior designs in cool colors such as blue and green. An example of a prompt message to set in this case would be, "I want to make the room a relaxing space."

[0167] The generated interior design proposals are presented to the user again via the device. The user selects their preferred proposal from the presented options, and related item information is automatically collected, generating a suggestion list from e-commerce sites. The user then considers purchasing items from this list, and by actually buying the selected items, the interior design can be realized.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The user takes images of their living space using their device and uploads them to the system. The input is digital image data, and the server receives this data as output; therefore, the image data is sent to the server. Specifically, the user takes a picture of their room with their smartphone camera and operates the upload button within the app.

[0171] Step 2:

[0172] The server analyzes the received image data using computer vision technology. The input is the image data sent in step 1, and the output is feature data of the living space, such as wall color, floor material, and room shape. Specifically, it uses OpenCV to perform image processing such as edge detection on the image and identify each element of the room.

[0173] Step 3:

[0174] The server uses a generation AI model, taking feature data as input, to generate interior layout proposals. The output is an interior coordination proposal based on color, arrangement, and style. Specifically, using TensorFlow or similar tools, a model trained on an existing interior database generates multiple layout proposals.

[0175] Step 4:

[0176] The device uses its built-in camera and microphone to capture the user's facial expressions and voice, collecting emotional data in real time. Input is the user's visual and audio data, and output is data for emotion analysis. Specifically, the device application activates the camera and microphone to record the user's facial expressions and voice.

[0177] Step 5:

[0178] The server analyzes the user's emotional data obtained from the terminal to determine the user's emotional state. The input is the emotional data collected in step 4, and the output is an emotional state such as "relaxed" or "active." Specifically, it uses an emotion recognition API to analyze the data and categorize the results.

[0179] Step 6:

[0180] The server adjusts the layout suggestions obtained from the generating AI model according to the user's emotional state. The input is the layout suggestions from step 3 and the emotional state from step 5, and the output is the adjusted layout suggestions. For example, if a relaxed state is desired, the server will adjust the layout to emphasize cool color tones.

[0181] Step 7:

[0182] The device presents the user with a refined interior design plan. The input is the layout plan refined in step 6, and the output is the visual information displayed on the device screen. Specifically, multiple options are displayed in a list format on the app screen, which the user can scroll through to view them.

[0183] Step 8:

[0184] The server retrieves item information related to the selected interior design plan. The input is the plan selected by the user, and the output is a list of item information obtained from the e-commerce site. Specifically, the server calls the API of the partner e-commerce site to collect product information that matches the selected style.

[0185] Step 9:

[0186] The terminal presents the user with product information obtained from the e-commerce site and accepts their purchase selection. The input is the product information list from step 8, and the output is the user's purchase decision. Specifically, the interface displays a product list and provides a purchase button, allowing the user to proceed directly with the purchase process.

[0187] (Application Example 2)

[0188] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0189] Interior design proposals that meet the diverse needs of modern consumers rely on static designs, which has the problem of not adequately addressing the emotional states and personalities of individual customers. Furthermore, there is a lack of means to immediately respond to customers' emotions with product suggestions in physical stores, so there is a need to improve the customer experience.

[0190] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0191] In this invention, the server includes means for the user to acquire images of the facility, means for analyzing the image data to extract features of the facility, and means for generating interior design proposals using a generative artificial intelligence model based on the extracted feature data and the user's emotional state. This makes it possible to provide dynamic and personalized interior design suggestions and product information tailored to the emotions of individual customers.

[0192] "User" refers to an individual or group receiving interior design proposals.

[0193] "Images of the facility" are visual data acquired by users that show the internal environment of the facility.

[0194] "Feature extraction" is the process of identifying and analyzing important information such as color and shape from image data.

[0195] "Emotional state" refers to information that represents the user's current psychological state and emotions.

[0196] A "generative artificial intelligence model" is an algorithm or system that dynamically generates interior design proposals based on input data.

[0197] An "interior design proposal" is a proposal document for interior design provided to the user.

[0198] "Product information" refers to detailed data such as price, specifications, and stock status of products related to the proposed interior design plan.

[0199] The system of this invention enables users to receive interior design suggestions using devices such as smartphones and tablets within facilities such as physical stores or their homes. The system operates in the following steps:

[0200] First, the user uses their device's camera at any location within the facility to capture "images of the facility." This image data is immediately sent to the server. The server analyzes the received image data using computer vision technologies such as OpenCV to "extract features" of the facility. This allows elements such as color and shape to be clearly identified.

[0201] Next, the device's camera and microphone are used to analyze the user's "emotional state." This allows the system to recognize emotions in real time from the user's facial expressions and voice using technologies such as the Emotion AI API. This emotional information is crucial data that is input into the generative AI model.

[0202] The generative artificial intelligence model generates "interior design proposals" tailored to the user based on acquired feature data and emotional states. This process utilizes advanced AI algorithms such as GPT-3(registered trademark).5, dynamically generating designs based on prompt text.

[0203] For example, if a user desires a calm and relaxed shopping environment in a store, and this sentiment is recognized, the AI ​​will suggest interior design options based on blue and green tones. An example of a prompt message might be: "Based on the following information, please suggest an interior design that matches the customer's mood. The store images were taken by a customer who prefers a relaxed atmosphere."

[0204] Finally, the generated interior design proposals are displayed on the terminal and presented to the user along with relevant "product information." The user can instantly select the presented products and purchase them through online shopping.

[0205] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0206] Step 1:

[0207] The user uses a terminal to acquire images within the facility and sends the image data to the server. The input is still image data captured through the terminal's camera, and the output is an image file transferred to the server. Through this operation, the user provides basic data for interior evaluation.

[0208] Step 2:

[0209] The server analyzes the received image files using OpenCV and extracts features from the facility. Specifically, it identifies features such as color, shape, and texture. The input is the image file transferred to the server, and the output is the extracted feature data. This feature data serves as basic information for interior design proposals.

[0210] Step 3:

[0211] The device captures the user's facial expressions and voice in real time via the Emotion AI API and recognizes their emotional state. In this process, camera and microphone data are the input, and the analyzed emotional state information is the output. This emotional data reflects the user's psychological state and has a significant impact on interior design recommendations.

[0212] Step 4:

[0213] The server uses a generative artificial intelligence model to generate interior design proposals based on acquired feature data and emotional information. The input is feature data and emotional state information, and the output is the generated interior design proposal. Specifically, GPT-3.5 uses prompts to construct the optimal design proposal.

[0214] Step 5:

[0215] The server sends the generated interior design proposal to the terminal and presents it to the user. The input is the generated interior design proposal, and the output is a visual display of the proposal on the terminal. The user uses this proposal as a reference to select products.

[0216] Step 6:

[0217] The user selects items they like based on the presented interior design proposals. The server uses this selection data to build a list of related product information and presents it to the user. The input is the product information selected by the user, and the output is a product list that includes detailed information.

[0218] Step 7:

[0219] The user selects the items they wish to purchase from the displayed product list and places an order. The server confirms the order details and completes the transaction. The input is the user's purchase instruction, and the output is the completed transaction data. This allows the user to improve their interior design through their store experience.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] The interior design system of this invention begins with the user uploading photos and floor plans of their room to the system. This data is transmitted to the server via the user's terminal.

[0237] The server first preprocesses the received image data in order to analyze it. This preprocessing includes resizing and denoising the images. Next, the server uses computer vision technology to extract features from the image data, such as the color of the room walls, the type of flooring, and the location of the windows.

[0238] Next, the server inputs the extracted feature data into a generating AI model to generate optimal interior design proposals. This generating artificial intelligence model can generate multiple design proposals, taking into account color harmony, furniture style, and efficient use of space.

[0239] The generated interior design proposals are sent from the server to the user's terminal and presented to the user through the interface. The user can review the presented proposals and choose their preferred style.

[0240] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The search results are then filtered based on price, style, ratings, etc., and presented to the user as a list of suggestions.

[0241] Users can select the items they want to purchase from this suggestion list and proceed to the purchase process on the e-commerce site via the link. This allows users to smoothly complete the entire process, from selecting furniture to purchasing it, online without having to visit a physical store.

[0242] As a concrete example, consider a scenario where a user uploads a photo of their living room to the system. The server extracts the characteristics of the space, such as white walls, wooden flooring, and large windows. The generative AI model then generates a simple Scandinavian-style interior design proposal, suggesting items like a light-colored sofa, a wood-toned coffee table, and simple curtains. Based on the suggestions, the user can select the necessary furniture and easily purchase it online. Throughout this entire process, the user can effortlessly achieve the perfect interior design for their room.

[0243] The following describes the processing flow.

[0244] Step 1:

[0245] The user uses their device to select and upload photos and floor plans of rooms through the system interface. The device then prepares to send these selected image data to the server.

[0246] Step 2:

[0247] The server receives image data sent from the terminal. After receiving the data, it preprocesses it, which includes resizing and noise reduction.

[0248] Step 3:

[0249] The server uses computer vision technology to extract room features from pre-processed image data. These features include wall color, floor material, window placement, and existing furniture arrangement.

[0250] Step 4:

[0251] The server inputs feature data into an artificial intelligence model to generate interior design proposals based on the room's characteristics. The generated proposals take into account color harmony, furniture style, and efficient placement.

[0252] Step 5:

[0253] The server sends the generated interior design proposals to the user's terminal and displays them. The terminal displays these proposals on its interface, allowing the user to browse the designs and select their preferred style.

[0254] Step 6:

[0255] Based on the user's selected outfit, the server searches for relevant furniture and accessory product information from e-commerce sites. The server then creates a product list to suggest to the user based on the search results.

[0256] Step 7:

[0257] The server sends the suggested product list to the user's device, and the user reviews the list. The user selects the products they like and accesses the purchase page on the e-commerce site from their device.

[0258] Step 8:

[0259] The user proceeds with the purchase process on the e-commerce site and buys the selected product. Payment information and shipping settings are handled according to the instructions on the e-commerce site until the purchase is completed.

[0260] (Example 1)

[0261] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0262] In selecting designs for modern living spaces, consumers require considerable knowledge and time to compare options and achieve highly personalized designs. Furthermore, the effort involved in quickly and effectively searching for and purchasing a wide variety of interior products is also a problem. To address these challenges, a system is needed that easily and efficiently proposes designs suitable for living spaces, and allows for the easy selection and purchase of related products based on those proposals.

[0263] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0264] In this invention, the server includes means for the user to transmit image data of the living space using a communication device, means for preprocessing and standardizing the image data, and means for using a machine learning algorithm to analyze the image data and extract the attributes of the living space. This enables the user to efficiently find the design they want and smoothly select and purchase related products.

[0265] A "user" is an individual or legal entity that uses this system to obtain design suggestions for living spaces and information on related products.

[0266] "Residential space" refers to the indoor space used by individuals or corporations for their daily lives, and this system is the subject of design proposals.

[0267] "Image data" refers to digital information files that visually record living spaces, and it is the input data for this system.

[0268] A "communication device" is an electronic device used to transmit data, and is a means used when a user sends image data of their living space to a server.

[0269] "Preprocessing" refers to the initial processing performed on image data, which is a technical process to standardize the data and reduce noise prior to analysis.

[0270] A "machine learning algorithm" is a computational method used by computers to learn patterns from data, and in this system, it is a method used to analyze and extract attributes of living spaces.

[0271] A "knowledge generation model" is an AI technology that learns from large datasets and generates new information and suggestions. In this system, it is used to generate design suggestions.

[0272] "Design proposals" refer to creative ideas generated by the server regarding the arrangement or style of interiors suitable for a living space.

[0273] "Items" refers to products and goods related to design proposals for living spaces that are provided to users through this system.

[0274] This interior design system allows users to easily obtain design proposals for their living spaces. First, the user takes or selects image data of their living space using their own device and uploads it to the server via a communication device. The uploaded image data undergoes preprocessing and standardization on the server. This preprocessing includes image resizing and noise reduction.

[0275] Subsequently, the server uses machine learning algorithms to analyze the image data and extract attributes of the living space. This process utilizes computer vision libraries (e.g., OpenCV and TensorFlow) to collect information such as wall color, flooring material, and window location. The extracted attribute data is then input into a knowledge generation model (e.g., a generative AI model).

[0276] The server generates design suggestions by passing prompt statements to the AI ​​model, which then utilizes relevant knowledge to produce design proposals. An example of a prompt statement is, "Analyze a photo of a living room and propose Scandinavian-style interior design ideas." Based on this prompt, the model outputs multiple design suggestions suitable for the living space.

[0277] The generated design proposals are sent from the server to the user's terminal, where the user can review the proposals and select their preferred design. After the user selects a design proposal, the server retrieves information on related items through the e-commerce site and presents it to the user. The user can then select the items they wish to purchase from the proposed items and proceed with the online purchase process. This entire process allows users to efficiently realize their interior design for their living spaces.

[0278] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0279] Step 1:

[0280] The user uses the terminal to take a photo of the living space or select an existing image. This image data is uploaded to the server using the communication device. The input is the selected image file, and the output is the image data sent to the server. Specifically, the user launches the application on the terminal, selects an image according to the instructions, and presses the upload button.

[0281] Step 2:

[0282] The server performs information preprocessing on the received image data. Specifically, the size of the image is standardized, and noise is removed using filtering techniques. The input is the unprocessed image data sent by the user, and the output is the clean image data after preprocessing. This processing enables accurate subsequent feature extraction.

[0283] Step 3:

[0284] The server uses a machine learning algorithm to analyze the preprocessed image data and extract the attributes of the living space. The input is the preprocessed image data, and the output is the extracted attribute data. Specifically, the server calls a computer vision library and analyzes the pixel information in the image to detect the wall color, floor material, window position, etc.

[0285] Step 4:

[0286] The server inputs the extracted attribute data into the knowledge generation model to generate design proposals. At this time, a prompt sentence is used to instruct the specific design theme to the generation AI model. The input is the attribute data and the prompt sentence, and the output is multiple candidates for the generated design proposals. Specifically, the server sends a prompt such as "Analyze the photo of the living room and propose a Scandinavian-style interior coordination plan." to the model.

[0287] Step 5:

[0288] The generated design proposals are sent from the server to the user's terminal. The user reviews the proposals through the interface on their terminal and selects their preferred design. The input is the design proposals sent from the server, and the output is the design selected by the user. Specifically, the user scrolls through the displayed proposal images and confirms their selection by pressing a selection button.

[0289] Step 6:

[0290] The server searches for items related to the design proposal selected by the user and retrieves that information from e-commerce sites. The input is the design selected by the user and the category information of the related items, and the output is a list of suggested items. Specifically, the server uses APIs to request product prices and review information from multiple e-commerce sites.

[0291] Step 7:

[0292] Users can select items they wish to purchase from a presented list. After selection, users proceed with the online purchase process using a link on their device. The input is the user's item selection, and the output is the progress towards the purchase of the selected items. Specifically, the user checks the details page for each item and completes the payment process by clicking the purchase button.

[0293] (Application Example 1)

[0294] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0295] Modern consumers demand quick and accurate suggestions when choosing interior design products in a physical space, but traditional methods are time-consuming and labor-intensive. This is especially true in brick-and-mortar stores, where providing prompt and personalized interior design suggestions tailored to customer needs is difficult, hindering immediate purchase decisions.

[0296] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0297] In this invention, the server includes a unit for users to upload image data of a room, a unit for analyzing the image data and extracting the characteristics of the room, and a unit for generating interior layout plans using a generated information processing model based on the extracted characteristic data. This makes it possible to immediately propose the optimal interior layout plan when serving customers in a physical store, using a wearable visual enhancement device within the store.

[0298] A "user-uploadable room image data unit" is a system component that has the function of allowing users to transmit visual information of their own room or any physical space in digital format.

[0299] A "unit that analyzes image data to extract room characteristics" refers to an algorithm or device that processes visual information and extracts useful information by identifying the features and components of a space.

[0300] A "generative information processing model" refers to artificial intelligence or learning algorithms that perform advanced optimization and generate layout proposals based on input data.

[0301] A "unit for generating interior layout plans" is a system element that has the function of constructing an optimal interior design tailored to the user's preferences and the purpose of the space, based on pre-analyzed spatial data.

[0302] A "visual enhancement device" is a device that assists a user's visual information and can overlay digital information on it. This refers to wearable devices such as smart glasses.

[0303] This invention is a system that enables quick and appropriate interior design proposals in physical stores. The system is configured as follows:

[0304] The server receives the image data of the room transmitted from the visual enhancement device worn by the user in the store. The image data is captured using a device such as smart glasses and transmitted to the server. The server analyzes the received image data and extracts its characteristics. Here, software applying computer vision technology is used, specifically, the OpenCV library is used. Thereby, the wall color scheme, furniture arrangement, space shape, etc. are analyzed.

[0305] Based on the analyzed characteristic data, the server automatically generates interior layout plans using a generation information processing model. In this process, a generation AI model is used, and machine learning frameworks such as TensorFlow and PyTorch are used. As a result, multiple interior layout plans are generated, which take into account specific themes and customer preferences.

[0306] The generated layout plans are presented on the display of the visual enhancement device worn by the user. This user interface is developed using Unity or ARKit, and the user can confirm the optimal layout plan in real time.

[0307] As a specific example, when the user transmits data in the living room section of the store using the visual enhancement device, the server can present a simple interior layout plan in the Nordic style. In this case, an example of the prompt text is "Living room / Scandinavian style / Wooden furniture with warm colors". Thereby, the user can quickly move on to a purchase decision without hesitation.

[0308] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0309] Step 1:

[0310] The user captures image data of the room using a visual enhancement device they are wearing. This image data is transmitted to the server via a terminal. The input is visual information of the real space, and the server receives the image data as output. This supplies the server with the necessary image information.

[0311] Step 2:

[0312] The server analyzes the received image data. Here, computer vision techniques are used to denoise and resize the images. Specifically, the OpenCV library is used to remove noise from the images and convert them to an appropriate resolution. The input is the image data obtained in step 1, and the output is the denoised, analyzable image data.

[0313] Step 3:

[0314] The server extracts room characteristics from the analyzed image data. Specifically, it applies algorithms to identify spatial features such as color and shape. The input is the image data processed in step 2, and the output is characteristic data such as the room's color and furniture arrangement.

[0315] Step 4:

[0316] The server inputs the extracted characteristic data into a generative information processing model to generate interior layout proposals. During this process, a generative AI model using TensorFlow or PyTorch is executed. The input is the characteristic data obtained in step 3, and the output is multiple interior layout proposals.

[0317] Step 5:

[0318] The server displays the generated interior layout plan on the display of the user's visual augmentation device. The user can then visually confirm this. The input is the interior layout plan generated in step 4, and the output is the graphical display information shown on the visual augmentation device.

[0319] Step 6:

[0320] The user selects their preferred interior layout from the presented options and proceeds to the purchasing process. The selected layout is sent to the server, where relevant product information is retrieved and presented to the user. The input is the user's selection of the layout presented in step 5, and the output is information on purchasable products. This sequence of actions allows the user to immediately proceed to purchasing.

[0321] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0322] This invention is a system that proposes interior design using image data of a room provided by the user, taking into account the user's emotional state. First, the user uploads photos and floor plans of the room to the system via a terminal. This image data is then transmitted from the user's terminal to the server.

[0323] The server uses computer vision technology to process the image data received from the terminal, extracting features such as wall color, floor material, and room shape. Based on this, the artificial intelligence model generates interior design proposals. These proposals take into account the colors, layout, and style of the interior.

[0324] Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions. The terminal uses the user's camera and microphone to input the user's facial expressions and voice into the emotion engine in real time. The server analyzes the input data and recognizes the user's emotional state. This emotional information is used to adjust the interior design proposal.

[0325] During the adjustment process, the suggested interior color palette and style are dynamically changed according to the user's emotional state. For example, if the user wants to relax, the generating AI model will prioritize suggesting interior designs with calming colors. Conversely, if the user is in an active emotional state, bright and vibrant colors will be suggested.

[0326] The generated interior design proposals are presented to the user on their device. The user can view the proposed designs and select the one they like. Based on the design proposal selected by the user, the server searches for relevant product information from e-commerce sites and displays it as a list of suggestions for the user. The user selects the products they wish to purchase from this list and proceeds with the purchase process through the e-commerce site.

[0327] For example, if the emotion engine determines that the user desires a calm atmosphere, the server, through its generative AI model, will propose an interior design plan based on cool colors such as blue and green. The user can then purchase furniture selected from the list according to this proposal, easily creating their ideal room. In this way, the present invention provides personalized interior design that responds to the user's emotions.

[0328] The following describes the processing flow.

[0329] Step 1:

[0330] The user selects photos and floor plans of a room on their device and uploads them through the system interface. The device then prepares to send this data and sends the image data to the server.

[0331] Step 2:

[0332] The server receives the transmitted image data and begins analysis using computer vision technology. The analysis includes processes to extract room features such as wall color, furniture arrangement, and floor material from the image.

[0333] Step 3:

[0334] The server inputs the extracted feature data into an artificial intelligence model to generate optimal interior design proposals for the room. The generated proposals take into account color harmony and spatial efficiency.

[0335] Step 4:

[0336] The user uses their device to input facial expressions and voice into the emotion engine via the camera and microphone to acquire emotion data. The device then sends this data to the server.

[0337] Step 5:

[0338] The server analyzes the received facial and voice data to recognize the user's emotional state. Specifically, it evaluates emotions such as joy, sadness, and calmness.

[0339] Step 6:

[0340] The server takes the user's emotional state into consideration and adjusts the generated interior design suggestions accordingly. For example, if the user indicates a desire to relax, the server will adjust the colors to soft, warm tones.

[0341] Step 7:

[0342] The adjusted interior design proposals are sent from the server to the user's terminal and presented on the interface. The user can view multiple proposals and choose their preferred one.

[0343] Step 8:

[0344] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The server then creates a list of suggested products and sends it to the user's device.

[0345] Step 9:

[0346] Users can view a product list on their device and select the items they wish to purchase. They can then proceed with the purchase process on the e-commerce site via the purchase link for their selected items.

[0347] (Example 2)

[0348] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0349] Conventional interior design systems have struggled to flexibly respond to users' emotions and preferences, making it difficult to provide appropriate suggestions tailored to individual needs. Furthermore, while users desire a system that allows for interior adjustments that take their emotional state into consideration, suitable technology has not yet existed.

[0350] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0351] In this invention, the server includes means for a user to transmit image data of a living space via an input device, a device for analyzing the image data and extracting features of the living space, and means for creating an interior layout plan using an artificial intelligence model generated based on the extracted feature data. This makes it possible to propose personalized interior coordination that responds to the user's emotions.

[0352] "Living space" refers to the indoor area where a user lives, and is a concept that includes rooms or parts of a house that are the subject of interior design.

[0353] An "input device" is a device used by a user to supply information to a system, and refers to electronic terminals such as smartphones, tablets, and computers.

[0354] "Image data" refers to digital information in which a user visually records the state of their living space, and includes photographs and drawing data.

[0355] A "server" is the central information processing device of a system, and its role is to collect, analyze, store, and distribute data.

[0356] A "generating artificial intelligence model" is a computer program equipped with a process for analyzing and generating data through learning, and is a technology used to propose optimal interior layout plans to users.

[0357] An "interior design plan" is a blueprint for the composition and layout of interiors in a living space, and includes proposals that encompass elements such as color, material, shape, and arrangement.

[0358] "Items" refer to parts, tools, furniture, etc., used to make up the interior of a living space, and are items that can be purchased as concrete products.

[0359] "Emotional state" refers to information that indicates a user's psychological reactions and emotional tendencies, and is an element used in adjusting interior design.

[0360] "Dynamic adjustment" refers to a process in which the system changes and optimizes results and suggestions in real time based on user input and circumstances.

[0361] The system based on this invention is primarily realized through three entities: a server, a terminal, and a user.

[0362] First, the user sends image data of their living space to the system via a device. This device can be a smartphone, tablet, or computer, and the images are sent over the internet, with the data being sent from the device to the server.

[0363] The server analyzes the received image data and extracts features of the living space. For this purpose, the server utilizes computer vision technology. Specifically, information processing libraries such as TensorFlow and OpenCV are used to extract features such as wall color, floor material, and room shape.

[0364] Next, the server utilizes an artificial intelligence model generated based on the extracted feature data to produce interior layout proposals. The generating AI model refers to a training database and constructs proposals considering the interior's colors, layout, and style. The interior coordination proposals are customized to suit the user's individual preferences.

[0365] Furthermore, the device uses the user's camera and microphone to acquire their emotional state. The emotion engine analyzes the user's facial expression and voice data to determine their emotional state, such as whether they are relaxed or active.

[0366] The server takes this emotional state into account and dynamically adjusts the interior design layout. For example, if the user desires a calm atmosphere, the server, through a generative AI model, will suggest and present interior designs in cool colors such as blue and green. An example of a prompt message to set in this case would be, "I want to make the room a relaxing space."

[0367] The generated interior design proposals are presented to the user again via the device. The user selects their preferred proposal from the presented options, and related item information is automatically collected, generating a suggestion list from e-commerce sites. The user then considers purchasing items from this list, and by actually buying the selected items, the interior design can be realized.

[0368] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0369] Step 1:

[0370] The user takes images of their living space using their device and uploads them to the system. The input is digital image data, and the server receives this data as output; therefore, the image data is sent to the server. Specifically, the user takes a picture of their room with their smartphone camera and operates the upload button within the app.

[0371] Step 2:

[0372] The server analyzes the received image data using computer vision technology. The input is the image data sent in step 1, and the output is feature data of the living space, such as wall color, floor material, and room shape. Specifically, it uses OpenCV to perform image processing such as edge detection on the image and identify each element of the room.

[0373] Step 3:

[0374] The server uses a generation AI model, taking feature data as input, to generate interior layout proposals. The output is an interior coordination proposal based on color, arrangement, and style. Specifically, using TensorFlow or similar tools, a model trained on an existing interior database generates multiple layout proposals.

[0375] Step 4:

[0376] The device uses its built-in camera and microphone to capture the user's facial expressions and voice, collecting emotional data in real time. Input is the user's visual and audio data, and output is data for emotion analysis. Specifically, the device application activates the camera and microphone to record the user's facial expressions and voice.

[0377] Step 5:

[0378] The server analyzes the user's emotional data obtained from the terminal to determine the user's emotional state. The input is the emotional data collected in step 4, and the output is an emotional state such as "relaxed" or "active." Specifically, it uses an emotion recognition API to analyze the data and categorize the results.

[0379] Step 6:

[0380] The server adjusts the layout suggestions obtained from the generating AI model according to the user's emotional state. The input is the layout suggestions from step 3 and the emotional state from step 5, and the output is the adjusted layout suggestions. For example, if a relaxed state is desired, the server will adjust the layout to emphasize cool color tones.

[0381] Step 7:

[0382] The device presents the user with a refined interior design plan. The input is the layout plan refined in step 6, and the output is the visual information displayed on the device screen. Specifically, multiple options are displayed in a list format on the app screen, which the user can scroll through to view them.

[0383] Step 8:

[0384] The server retrieves item information related to the selected interior design plan. The input is the plan selected by the user, and the output is a list of item information obtained from the e-commerce site. Specifically, the server calls the API of the partner e-commerce site to collect product information that matches the selected style.

[0385] Step 9:

[0386] The terminal presents the user with product information obtained from the e-commerce site and accepts their purchase selection. The input is the product information list from step 8, and the output is the user's purchase decision. Specifically, the interface displays a product list and provides a purchase button, allowing the user to proceed directly with the purchase process.

[0387] (Application Example 2)

[0388] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0389] Interior design proposals that meet the diverse needs of modern consumers rely on static designs, which has the problem of not adequately addressing the emotional states and personalities of individual customers. Furthermore, there is a lack of means to immediately respond to customers' emotions with product suggestions in physical stores, so there is a need to improve the customer experience.

[0390] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0391] In this invention, the server includes means for the user to acquire images of the facility, means for analyzing the image data to extract features of the facility, and means for generating interior design proposals using a generative artificial intelligence model based on the extracted feature data and the user's emotional state. This makes it possible to provide dynamic and personalized interior design suggestions and product information tailored to the emotions of individual customers.

[0392] "User" refers to an individual or group receiving interior design proposals.

[0393] "Images of the facility" are visual data acquired by users that show the internal environment of the facility.

[0394] "Feature extraction" is the process of identifying and analyzing important information such as color and shape from image data.

[0395] "Emotional state" refers to information that represents the user's current psychological state and emotions.

[0396] A "generative artificial intelligence model" is an algorithm or system that dynamically generates interior design proposals based on input data.

[0397] An "interior design proposal" is a proposal document for interior design provided to the user.

[0398] "Product information" refers to detailed data such as price, specifications, and stock status of products related to the proposed interior design plan.

[0399] The system of this invention enables users to receive interior design suggestions using devices such as smartphones and tablets within facilities such as physical stores or their homes. The system operates in the following steps:

[0400] First, the user uses their device's camera at any location within the facility to capture "images of the facility." This image data is immediately sent to the server. The server analyzes the received image data using computer vision technologies such as OpenCV to "extract features" of the facility. This allows elements such as color and shape to be clearly identified.

[0401] Next, the device's camera and microphone are used to analyze the user's "emotional state." This allows the system to recognize emotions in real time from the user's facial expressions and voice using technologies such as the Emotion AI API. This emotional information is crucial data that is input into the generative AI model.

[0402] The generative artificial intelligence model generates "interior design proposals" tailored to the user based on acquired feature data and emotional states. This process utilizes advanced AI algorithms such as GPT-3.5, dynamically generating designs based on prompt text.

[0403] For example, if a user desires a calm and relaxed shopping environment in a store, and this sentiment is recognized, the AI ​​will suggest interior design options based on blue and green tones. An example of a prompt message might be: "Based on the following information, please suggest an interior design that matches the customer's mood. The store images were taken by a customer who prefers a relaxed atmosphere."

[0404] Finally, the generated interior design proposals are displayed on the terminal and presented to the user along with relevant "product information." The user can instantly select the presented products and purchase them through online shopping.

[0405] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0406] Step 1:

[0407] The user uses a terminal to acquire images within the facility and sends the image data to the server. The input is still image data captured through the terminal's camera, and the output is an image file transferred to the server. Through this operation, the user provides basic data for interior evaluation.

[0408] Step 2:

[0409] The server analyzes the received image files using OpenCV and extracts features from the facility. Specifically, it identifies features such as color, shape, and texture. The input is the image file transferred to the server, and the output is the extracted feature data. This feature data serves as basic information for interior design proposals.

[0410] Step 3:

[0411] The device captures the user's facial expressions and voice in real time via the Emotion AI API and recognizes their emotional state. In this process, camera and microphone data are the input, and the analyzed emotional state information is the output. This emotional data reflects the user's psychological state and has a significant impact on interior design recommendations.

[0412] Step 4:

[0413] The server uses a generative artificial intelligence model to generate interior design proposals based on acquired feature data and emotional information. The input is feature data and emotional state information, and the output is the generated interior design proposal. Specifically, GPT-3.5 uses prompts to construct the optimal design proposal.

[0414] Step 5:

[0415] The server sends the generated interior design proposal to the terminal and presents it to the user. The input is the generated interior design proposal, and the output is a visual display of the proposal on the terminal. The user uses this proposal as a reference to select products.

[0416] Step 6:

[0417] The user selects items they like based on the presented interior design proposals. The server uses this selection data to build a list of related product information and presents it to the user. The input is the product information selected by the user, and the output is a product list that includes detailed information.

[0418] Step 7:

[0419] The user selects the items they wish to purchase from the displayed product list and places an order. The server confirms the order details and completes the transaction. The input is the user's purchase instruction, and the output is the completed transaction data. This allows the user to improve their interior design through their store experience.

[0420] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0421] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0422] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0423] [Third Embodiment]

[0424] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0425] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0426] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0427] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0428] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0430] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0431] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0432] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0433] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0434] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0435] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0436] The interior design system of this invention begins with the user uploading photos and floor plans of their room to the system. This data is transmitted to the server via the user's terminal.

[0437] The server first preprocesses the received image data in order to analyze it. This preprocessing includes resizing and denoising the images. Next, the server uses computer vision technology to extract features from the image data, such as the color of the room walls, the type of flooring, and the location of the windows.

[0438] Next, the server inputs the extracted feature data into a generating AI model to generate optimal interior design proposals. This generating artificial intelligence model can generate multiple design proposals, taking into account color harmony, furniture style, and efficient use of space.

[0439] The generated interior design proposals are sent from the server to the user's terminal and presented to the user through the interface. The user can review the presented proposals and choose their preferred style.

[0440] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The search results are then filtered based on price, style, ratings, etc., and presented to the user as a list of suggestions.

[0441] Users can select the items they want to purchase from this suggestion list and proceed to the purchase process on the e-commerce site via the link. This allows users to smoothly complete the entire process, from selecting furniture to purchasing it, online without having to visit a physical store.

[0442] As a concrete example, consider a scenario where a user uploads a photo of their living room to the system. The server extracts the characteristics of the space, such as white walls, wooden flooring, and large windows. The generative AI model then generates a simple Scandinavian-style interior design proposal, suggesting items like a light-colored sofa, a wood-toned coffee table, and simple curtains. Based on the suggestions, the user can select the necessary furniture and easily purchase it online. Throughout this entire process, the user can effortlessly achieve the perfect interior design for their room.

[0443] The following describes the processing flow.

[0444] Step 1:

[0445] The user uses their device to select and upload photos and floor plans of rooms through the system interface. The device then prepares to send these selected image data to the server.

[0446] Step 2:

[0447] The server receives image data sent from the terminal. After receiving the data, it preprocesses it, which includes resizing and noise reduction.

[0448] Step 3:

[0449] The server uses computer vision technology to extract room features from pre-processed image data. These features include wall color, floor material, window placement, and existing furniture arrangement.

[0450] Step 4:

[0451] The server inputs feature data into an artificial intelligence model to generate interior design proposals based on the room's characteristics. The generated proposals take into account color harmony, furniture style, and efficient placement.

[0452] Step 5:

[0453] The server sends the generated interior design proposals to the user's terminal and displays them. The terminal displays these proposals on its interface, allowing the user to browse the designs and select their preferred style.

[0454] Step 6:

[0455] Based on the user's selected outfit, the server searches for relevant furniture and accessory product information from e-commerce sites. The server then creates a product list to suggest to the user based on the search results.

[0456] Step 7:

[0457] The server sends the suggested product list to the user's device, and the user reviews the list. The user selects the products they like and accesses the purchase page on the e-commerce site from their device.

[0458] Step 8:

[0459] The user proceeds with the purchase process on the e-commerce site and buys the selected product. Payment information and shipping settings are handled according to the instructions on the e-commerce site until the purchase is completed.

[0460] (Example 1)

[0461] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] In selecting designs for modern living spaces, consumers require considerable knowledge and time to compare options and achieve highly personalized designs. Furthermore, the effort involved in quickly and effectively searching for and purchasing a wide variety of interior products is also a problem. To address these challenges, a system is needed that easily and efficiently proposes designs suitable for living spaces, and allows for the easy selection and purchase of related products based on those proposals.

[0463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0464] In this invention, the server includes means for the user to transmit image data of the living space using a communication device, means for preprocessing and standardizing the image data, and means for using a machine learning algorithm to analyze the image data and extract the attributes of the living space. This enables the user to efficiently find the design they want and smoothly select and purchase related products.

[0465] A "user" is an individual or legal entity that uses this system to obtain design suggestions for living spaces and information on related products.

[0466] "Residential space" refers to the indoor space used by individuals or corporations for their daily lives, and this system is the subject of design proposals.

[0467] "Image data" refers to digital information files that visually record living spaces, and it is the input data for this system.

[0468] A "communication device" is an electronic device used to transmit data, and is a means used when a user sends image data of their living space to a server.

[0469] "Preprocessing" refers to the initial processing performed on image data, which is a technical process to standardize the data and reduce noise prior to analysis.

[0470] A "machine learning algorithm" is a computational method used by computers to learn patterns from data, and in this system, it is a method used to analyze and extract attributes of living spaces.

[0471] A "knowledge generation model" is an AI technology that learns from large datasets and generates new information and suggestions. In this system, it is used to generate design suggestions.

[0472] "Design proposals" refer to creative ideas generated by the server regarding the arrangement or style of interiors suitable for a living space.

[0473] "Items" refers to products and goods related to design proposals for living spaces that are provided to users through this system.

[0474] This interior design system allows users to easily obtain design proposals for their living spaces. First, the user takes or selects image data of their living space using their own device and uploads it to the server via a communication device. The uploaded image data undergoes preprocessing and standardization on the server. This preprocessing includes image resizing and noise reduction.

[0475] Subsequently, the server uses machine learning algorithms to analyze the image data and extract attributes of the living space. This process utilizes computer vision libraries (e.g., OpenCV and TensorFlow) to collect information such as wall color, flooring material, and window location. The extracted attribute data is then input into a knowledge generation model (e.g., a generative AI model).

[0476] The server generates design suggestions by passing prompt statements to the AI ​​model, which then utilizes relevant knowledge to produce design proposals. An example of a prompt statement is, "Analyze a photo of a living room and propose Scandinavian-style interior design ideas." Based on this prompt, the model outputs multiple design suggestions suitable for the living space.

[0477] The generated design proposals are sent from the server to the user's terminal, where the user can review the proposals and select their preferred design. After the user selects a design proposal, the server retrieves information on related items through the e-commerce site and presents it to the user. The user can then select the items they wish to purchase from the proposed items and proceed with the online purchase process. This entire process allows users to efficiently realize their interior design for their living spaces.

[0478] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0479] Step 1:

[0480] The user takes a photo of their living space using their device or selects an existing image. This image data is uploaded to the server using a communication device. The input is the selected image file, and the output is the image data sent to the server. Specifically, the user launches the application on their device, selects an image according to the instructions, and presses the upload button.

[0481] Step 2:

[0482] The server performs preprocessing on the received image data. Specifically, it standardizes the image size and removes noise using filtering techniques. The input is the unprocessed image data sent by the user, and the output is the clean image data after preprocessing. This process ensures accurate subsequent feature extraction.

[0483] Step 3:

[0484] The server uses machine learning algorithms to analyze preprocessed image data and extract attributes of the living space. The input is preprocessed image data, and the output is extracted attribute data. Specifically, the server calls a computer vision library and analyzes pixel information in the image to detect wall color, flooring material, window positions, etc.

[0485] Step 4:

[0486] The server inputs the extracted attribute data into a knowledge generation model to generate design proposals. At this time, prompt statements are used to instruct the AI ​​model on specific design themes. The input consists of attribute data and prompt statements, and the output is multiple candidate design proposals. For example, the server sends the prompt "Analyze a photo of a living room and propose a Scandinavian-style interior design plan."

[0487] Step 5:

[0488] The generated design proposals are sent from the server to the user's terminal. The user reviews the proposals through the interface on their terminal and selects their preferred design. The input is the design proposals sent from the server, and the output is the design selected by the user. Specifically, the user scrolls through the displayed proposal images and confirms their selection by pressing a selection button.

[0489] Step 6:

[0490] The server searches for items related to the design proposal selected by the user and retrieves that information from e-commerce sites. The input is the design selected by the user and the category information of the related items, and the output is a list of suggested items. Specifically, the server uses APIs to request product prices and review information from multiple e-commerce sites.

[0491] Step 7:

[0492] Users can select items they wish to purchase from a presented list. After selection, users proceed with the online purchase process using a link on their device. The input is the user's item selection, and the output is the progress towards the purchase of the selected items. Specifically, the user checks the details page for each item and completes the payment process by clicking the purchase button.

[0493] (Application Example 1)

[0494] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] Modern consumers demand quick and accurate suggestions when choosing interior design products in a physical space, but traditional methods are time-consuming and labor-intensive. This is especially true in brick-and-mortar stores, where providing prompt and personalized interior design suggestions tailored to customer needs is difficult, hindering immediate purchase decisions.

[0496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0497] In this invention, the server includes a unit for users to upload image data of a room, a unit for analyzing the image data and extracting the characteristics of the room, and a unit for generating interior layout plans using a generated information processing model based on the extracted characteristic data. This makes it possible to immediately propose the optimal interior layout plan when serving customers in a physical store, using a wearable visual enhancement device within the store.

[0498] A "user-uploadable room image data unit" is a system component that has the function of allowing users to transmit visual information of their own room or any physical space in digital format.

[0499] A "unit that analyzes image data to extract room characteristics" refers to an algorithm or device that processes visual information and extracts useful information by identifying the features and components of a space.

[0500] A "generative information processing model" refers to artificial intelligence or learning algorithms that perform advanced optimization and generate layout proposals based on input data.

[0501] A "unit for generating interior layout plans" is a system element that has the function of constructing an optimal interior design tailored to the user's preferences and the purpose of the space, based on pre-analyzed spatial data.

[0502] A "visual enhancement device" is a device that assists a user's visual information and can overlay digital information on it. This refers to wearable devices such as smart glasses.

[0503] This invention is a system that enables quick and appropriate interior design proposals in physical stores. The system is configured as follows:

[0504] The server receives image data of the room transmitted from a visual enhancement device worn by the user within the store. The image data is captured using a device such as smart glasses and sent to the server. The server analyzes the received image data and extracts its characteristics. This is done using software that applies computer vision technology, specifically the OpenCV library. This allows for the analysis of wall color schemes, furniture arrangement, and spatial shape.

[0505] Based on the analyzed characteristic data, the server automatically generates interior layout plans using a generative information processing model. This process employs a generative AI model and machine learning frameworks such as TensorFlow and PyTorch. As a result, multiple interior layout plans are generated, taking into account specific themes and customer preferences.

[0506] The generated placement options are displayed on the user's visual enhancement device. This user interface was developed using Unity and ARKit, allowing the user to see the optimal placement options in real time.

[0507] As a concrete example, if a user sends data using a visual enhancement device to the living room section of a store, the server can present a simple Scandinavian-style interior layout plan. In this case, an example of a prompt message might be "Living room / Scandinavian style / Warm-toned wooden furniture." This allows the user to quickly and easily make a purchase decision without hesitation.

[0508] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0509] Step 1:

[0510] The user captures image data of the room using a visual enhancement device they are wearing. This image data is transmitted to the server via a terminal. The input is visual information of the real space, and the server receives the image data as output. This supplies the server with the necessary image information.

[0511] Step 2:

[0512] The server analyzes the received image data. Here, computer vision techniques are used to denoise and resize the images. Specifically, the OpenCV library is used to remove noise from the images and convert them to an appropriate resolution. The input is the image data obtained in step 1, and the output is the denoised, analyzable image data.

[0513] Step 3:

[0514] The server extracts room characteristics from the analyzed image data. Specifically, it applies algorithms to identify spatial features such as color and shape. The input is the image data processed in step 2, and the output is characteristic data such as the room's color and furniture arrangement.

[0515] Step 4:

[0516] The server inputs the extracted characteristic data into a generative information processing model to generate interior layout proposals. During this process, a generative AI model using TensorFlow or PyTorch is executed. The input is the characteristic data obtained in step 3, and the output is multiple interior layout proposals.

[0517] Step 5:

[0518] The server displays the generated interior layout plan on the display of the user's visual augmentation device. The user can then visually confirm this. The input is the interior layout plan generated in step 4, and the output is the graphical display information shown on the visual augmentation device.

[0519] Step 6:

[0520] The user selects their preferred interior layout from the presented options and proceeds to the purchasing process. The selected layout is sent to the server, where relevant product information is retrieved and presented to the user. The input is the user's selection of the layout presented in step 5, and the output is information on purchasable products. This sequence of actions allows the user to immediately proceed to purchasing.

[0521] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0522] This invention is a system that proposes interior design using image data of a room provided by the user, taking into account the user's emotional state. First, the user uploads photos and floor plans of the room to the system via a terminal. This image data is then transmitted from the user's terminal to the server.

[0523] The server uses computer vision technology to process the image data received from the terminal, extracting features such as wall color, floor material, and room shape. Based on this, the artificial intelligence model generates interior design proposals. These proposals take into account the colors, layout, and style of the interior.

[0524] Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions. The terminal uses the user's camera and microphone to input the user's facial expressions and voice into the emotion engine in real time. The server analyzes the input data and recognizes the user's emotional state. This emotional information is used to adjust the interior design proposal.

[0525] During the adjustment process, the suggested interior color palette and style are dynamically changed according to the user's emotional state. For example, if the user wants to relax, the generating AI model will prioritize suggesting interior designs with calming colors. Conversely, if the user is in an active emotional state, bright and vibrant colors will be suggested.

[0526] The generated interior design proposals are presented to the user on their device. The user can view the proposed designs and select the one they like. Based on the design proposal selected by the user, the server searches for relevant product information from e-commerce sites and displays it as a list of suggestions for the user. The user selects the products they wish to purchase from this list and proceeds with the purchase process through the e-commerce site.

[0527] For example, if the emotion engine determines that the user desires a calm atmosphere, the server, through its generative AI model, will propose an interior design plan based on cool colors such as blue and green. The user can then purchase furniture selected from the list according to this proposal, easily creating their ideal room. In this way, the present invention provides personalized interior design that responds to the user's emotions.

[0528] The following describes the processing flow.

[0529] Step 1:

[0530] The user selects photos and floor plans of a room on their device and uploads them through the system interface. The device then prepares to send this data and sends the image data to the server.

[0531] Step 2:

[0532] The server receives the transmitted image data and begins analysis using computer vision technology. The analysis includes processes to extract room features such as wall color, furniture arrangement, and floor material from the image.

[0533] Step 3:

[0534] The server inputs the extracted feature data into an artificial intelligence model to generate optimal interior design proposals for the room. The generated proposals take into account color harmony and spatial efficiency.

[0535] Step 4:

[0536] The user uses their device to input facial expressions and voice into the emotion engine via the camera and microphone to acquire emotion data. The device then sends this data to the server.

[0537] Step 5:

[0538] The server analyzes the received facial and voice data to recognize the user's emotional state. Specifically, it evaluates emotions such as joy, sadness, and calmness.

[0539] Step 6:

[0540] The server takes the user's emotional state into consideration and adjusts the generated interior design suggestions accordingly. For example, if the user indicates a desire to relax, the server will adjust the colors to soft, warm tones.

[0541] Step 7:

[0542] The adjusted interior design proposals are sent from the server to the user's terminal and presented on the interface. The user can view multiple proposals and choose their preferred one.

[0543] Step 8:

[0544] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The server then creates a list of suggested products and sends it to the user's device.

[0545] Step 9:

[0546] Users can view a product list on their device and select the items they wish to purchase. They can then proceed with the purchase process on the e-commerce site via the purchase link for their selected items.

[0547] (Example 2)

[0548] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0549] Conventional interior design systems have struggled to flexibly respond to users' emotions and preferences, making it difficult to provide appropriate suggestions tailored to individual needs. Furthermore, while users desire a system that allows for interior adjustments that take their emotional state into consideration, suitable technology has not yet existed.

[0550] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0551] In this invention, the server includes means for a user to transmit image data of a living space via an input device, a device for analyzing the image data and extracting features of the living space, and means for creating an interior layout plan using an artificial intelligence model generated based on the extracted feature data. This makes it possible to propose personalized interior coordination that responds to the user's emotions.

[0552] "Living space" refers to the indoor area where a user lives, and is a concept that includes rooms or parts of a house that are the subject of interior design.

[0553] An "input device" is a device used by a user to supply information to a system, and refers to electronic terminals such as smartphones, tablets, and computers.

[0554] "Image data" refers to digital information in which a user visually records the state of their living space, and includes photographs and drawing data.

[0555] A "server" is the central information processing device of a system, and its role is to collect, analyze, store, and distribute data.

[0556] A "generating artificial intelligence model" is a computer program equipped with a process for analyzing and generating data through learning, and is a technology used to propose optimal interior layout plans to users.

[0557] An "interior design plan" is a blueprint for the composition and layout of interiors in a living space, and includes proposals that encompass elements such as color, material, shape, and arrangement.

[0558] "Items" refer to parts, tools, furniture, etc., used to make up the interior of a living space, and are items that can be purchased as concrete products.

[0559] "Emotional state" refers to information that indicates a user's psychological reactions and emotional tendencies, and is an element used in adjusting interior design.

[0560] "Dynamic adjustment" refers to a process in which the system changes and optimizes results and suggestions in real time based on user input and circumstances.

[0561] The system based on this invention is primarily realized through three entities: a server, a terminal, and a user.

[0562] First, the user sends image data of their living space to the system via a device. This device can be a smartphone, tablet, or computer, and the images are sent over the internet, with the data being sent from the device to the server.

[0563] The server analyzes the received image data and extracts features of the living space. For this purpose, the server utilizes computer vision technology. Specifically, information processing libraries such as TensorFlow and OpenCV are used to extract features such as wall color, floor material, and room shape.

[0564] Next, the server utilizes an artificial intelligence model generated based on the extracted feature data to produce interior layout proposals. The generating AI model refers to a training database and constructs proposals considering the interior's colors, layout, and style. The interior coordination proposals are customized to suit the user's individual preferences.

[0565] Furthermore, the device uses the user's camera and microphone to acquire their emotional state. The emotion engine analyzes the user's facial expression and voice data to determine their emotional state, such as whether they are relaxed or active.

[0566] The server takes this emotional state into account and dynamically adjusts the interior design layout. For example, if the user desires a calm atmosphere, the server, through a generative AI model, will suggest and present interior designs in cool colors such as blue and green. An example of a prompt message to set in this case would be, "I want to make the room a relaxing space."

[0567] The generated interior design proposals are presented to the user again via the device. The user selects their preferred proposal from the presented options, and related item information is automatically collected, generating a suggestion list from e-commerce sites. The user then considers purchasing items from this list, and by actually buying the selected items, the interior design can be realized.

[0568] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0569] Step 1:

[0570] The user takes images of their living space using their device and uploads them to the system. The input is digital image data, and the server receives this data as output; therefore, the image data is sent to the server. Specifically, the user takes a picture of their room with their smartphone camera and operates the upload button within the app.

[0571] Step 2:

[0572] The server analyzes the received image data using computer vision technology. The input is the image data sent in step 1, and the output is feature data of the living space, such as wall color, floor material, and room shape. Specifically, it uses OpenCV to perform image processing such as edge detection on the image and identify each element of the room.

[0573] Step 3:

[0574] The server uses a generation AI model, taking feature data as input, to generate interior layout proposals. The output is an interior coordination proposal based on color, arrangement, and style. Specifically, using TensorFlow or similar tools, a model trained on an existing interior database generates multiple layout proposals.

[0575] Step 4:

[0576] The device uses its built-in camera and microphone to capture the user's facial expressions and voice, collecting emotional data in real time. Input is the user's visual and audio data, and output is data for emotion analysis. Specifically, the device application activates the camera and microphone to record the user's facial expressions and voice.

[0577] Step 5:

[0578] The server analyzes the user's emotional data obtained from the terminal to determine the user's emotional state. The input is the emotional data collected in step 4, and the output is an emotional state such as "relaxed" or "active." Specifically, it uses an emotion recognition API to analyze the data and categorize the results.

[0579] Step 6:

[0580] The server adjusts the layout suggestions obtained from the generating AI model according to the user's emotional state. The input is the layout suggestions from step 3 and the emotional state from step 5, and the output is the adjusted layout suggestions. For example, if a relaxed state is desired, the server will adjust the layout to emphasize cool color tones.

[0581] Step 7:

[0582] The device presents the user with a refined interior design plan. The input is the layout plan refined in step 6, and the output is the visual information displayed on the device screen. Specifically, multiple options are displayed in a list format on the app screen, which the user can scroll through to view them.

[0583] Step 8:

[0584] The server retrieves item information related to the selected interior design plan. The input is the plan selected by the user, and the output is a list of item information obtained from the e-commerce site. Specifically, the server calls the API of the partner e-commerce site to collect product information that matches the selected style.

[0585] Step 9:

[0586] The terminal presents the user with product information obtained from the e-commerce site and accepts their purchase selection. The input is the product information list from step 8, and the output is the user's purchase decision. Specifically, the interface displays a product list and provides a purchase button, allowing the user to proceed directly with the purchase process.

[0587] (Application Example 2)

[0588] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] Interior design proposals that meet the diverse needs of modern consumers rely on static designs, which has the problem of not adequately addressing the emotional states and personalities of individual customers. Furthermore, there is a lack of means to immediately respond to customers' emotions with product suggestions in physical stores, so there is a need to improve the customer experience.

[0590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0591] In this invention, the server includes means for the user to acquire images of the facility, means for analyzing the image data to extract features of the facility, and means for generating interior design proposals using a generative artificial intelligence model based on the extracted feature data and the user's emotional state. This makes it possible to provide dynamic and personalized interior design suggestions and product information tailored to the emotions of individual customers.

[0592] "User" refers to an individual or group receiving interior design proposals.

[0593] "Images of the facility" are visual data acquired by users that show the internal environment of the facility.

[0594] "Feature extraction" is the process of identifying and analyzing important information such as color and shape from image data.

[0595] "Emotional state" refers to information that represents the user's current psychological state and emotions.

[0596] A "generative artificial intelligence model" is an algorithm or system that dynamically generates interior design proposals based on input data.

[0597] An "interior design proposal" is a proposal document for interior design provided to the user.

[0598] "Product information" refers to detailed data such as price, specifications, and stock status of products related to the proposed interior design plan.

[0599] The system of this invention enables users to receive interior design suggestions using devices such as smartphones and tablets within facilities such as physical stores or their homes. The system operates in the following steps:

[0600] First, the user uses their device's camera at any location within the facility to capture "images of the facility." This image data is immediately sent to the server. The server analyzes the received image data using computer vision technologies such as OpenCV to "extract features" of the facility. This allows elements such as color and shape to be clearly identified.

[0601] Next, the device's camera and microphone are used to analyze the user's "emotional state." This allows the system to recognize emotions in real time from the user's facial expressions and voice using technologies such as the Emotion AI API. This emotional information is crucial data that is input into the generative AI model.

[0602] The generative artificial intelligence model generates "interior design proposals" tailored to the user based on acquired feature data and emotional states. This process utilizes advanced AI algorithms such as GPT-3.5, dynamically generating designs based on prompt text.

[0603] For example, if a user desires a calm and relaxed shopping environment in a store, and this sentiment is recognized, the AI ​​will suggest interior design options based on blue and green tones. An example of a prompt message might be: "Based on the following information, please suggest an interior design that matches the customer's mood. The store images were taken by a customer who prefers a relaxed atmosphere."

[0604] Finally, the generated interior design proposals are displayed on the terminal and presented to the user along with relevant "product information." The user can instantly select the presented products and purchase them through online shopping.

[0605] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0606] Step 1:

[0607] The user uses a terminal to acquire images within the facility and sends the image data to the server. The input is still image data captured through the terminal's camera, and the output is an image file transferred to the server. Through this operation, the user provides basic data for interior evaluation.

[0608] Step 2:

[0609] The server analyzes the received image files using OpenCV and extracts features from the facility. Specifically, it identifies features such as color, shape, and texture. The input is the image file transferred to the server, and the output is the extracted feature data. This feature data serves as basic information for interior design proposals.

[0610] Step 3:

[0611] The device captures the user's facial expressions and voice in real time via the Emotion AI API and recognizes their emotional state. In this process, camera and microphone data are the input, and the analyzed emotional state information is the output. This emotional data reflects the user's psychological state and has a significant impact on interior design recommendations.

[0612] Step 4:

[0613] The server uses a generative artificial intelligence model to generate interior design proposals based on acquired feature data and emotional information. The input is feature data and emotional state information, and the output is the generated interior design proposal. Specifically, GPT-3.5 uses prompts to construct the optimal design proposal.

[0614] Step 5:

[0615] The server sends the generated interior design proposal to the terminal and presents it to the user. The input is the generated interior design proposal, and the output is a visual display of the proposal on the terminal. The user uses this proposal as a reference to select products.

[0616] Step 6:

[0617] The user selects items they like based on the presented interior design proposals. The server uses this selection data to build a list of related product information and presents it to the user. The input is the product information selected by the user, and the output is a product list that includes detailed information.

[0618] Step 7:

[0619] The user selects the items they wish to purchase from the displayed product list and places an order. The server confirms the order details and completes the transaction. The input is the user's purchase instruction, and the output is the completed transaction data. This allows the user to improve their interior design through their store experience.

[0620] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0621] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0622] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0623] [Fourth Embodiment]

[0624] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0625] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0626] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0627] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0628] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0629] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0630] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0631] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0632] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0633] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0634] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0635] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0636] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0637] The interior design system of this invention begins with the user uploading photos and floor plans of their room to the system. This data is transmitted to the server via the user's terminal.

[0638] The server first preprocesses the received image data in order to analyze it. This preprocessing includes resizing and denoising the images. Next, the server uses computer vision technology to extract features from the image data, such as the color of the room walls, the type of flooring, and the location of the windows.

[0639] Next, the server inputs the extracted feature data into a generating AI model to generate optimal interior design proposals. This generating artificial intelligence model can generate multiple design proposals, taking into account color harmony, furniture style, and efficient use of space.

[0640] The generated interior design proposals are sent from the server to the user's terminal and presented to the user through the interface. The user can review the presented proposals and choose their preferred style.

[0641] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The search results are then filtered based on price, style, ratings, etc., and presented to the user as a list of suggestions.

[0642] Users can select the items they want to purchase from this suggestion list and proceed to the purchase process on the e-commerce site via the link. This allows users to smoothly complete the entire process, from selecting furniture to purchasing it, online without having to visit a physical store.

[0643] As a concrete example, consider a scenario where a user uploads a photo of their living room to the system. The server extracts the characteristics of the space, such as white walls, wooden flooring, and large windows. The generative AI model then generates a simple Scandinavian-style interior design proposal, suggesting items like a light-colored sofa, a wood-toned coffee table, and simple curtains. Based on the suggestions, the user can select the necessary furniture and easily purchase it online. Throughout this entire process, the user can effortlessly achieve the perfect interior design for their room.

[0644] The following describes the processing flow.

[0645] Step 1:

[0646] The user uses their device to select and upload photos and floor plans of rooms through the system interface. The device then prepares to send these selected image data to the server.

[0647] Step 2:

[0648] The server receives image data sent from the terminal. After receiving the data, it preprocesses it, which includes resizing and noise reduction.

[0649] Step 3:

[0650] The server uses computer vision technology to extract room features from pre-processed image data. These features include wall color, floor material, window placement, and existing furniture arrangement.

[0651] Step 4:

[0652] The server inputs feature data into an artificial intelligence model to generate interior design proposals based on the room's characteristics. The generated proposals take into account color harmony, furniture style, and efficient placement.

[0653] Step 5:

[0654] The server sends the generated interior design proposals to the user's terminal and displays them. The terminal displays these proposals on its interface, allowing the user to browse the designs and select their preferred style.

[0655] Step 6:

[0656] Based on the user's selected outfit, the server searches for relevant furniture and accessory product information from e-commerce sites. The server then creates a product list to suggest to the user based on the search results.

[0657] Step 7:

[0658] The server sends the suggested product list to the user's device, and the user reviews the list. The user selects the products they like and accesses the purchase page on the e-commerce site from their device.

[0659] Step 8:

[0660] The user proceeds with the purchase process on the e-commerce site and buys the selected product. Payment information and shipping settings are handled according to the instructions on the e-commerce site until the purchase is completed.

[0661] (Example 1)

[0662] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0663] In selecting designs for modern living spaces, consumers require considerable knowledge and time to compare options and achieve highly personalized designs. Furthermore, the effort involved in quickly and effectively searching for and purchasing a wide variety of interior products is also a problem. To address these challenges, a system is needed that easily and efficiently proposes designs suitable for living spaces, and allows for the easy selection and purchase of related products based on those proposals.

[0664] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0665] In this invention, the server includes means for the user to transmit image data of the living space using a communication device, means for preprocessing and standardizing the image data, and means for using a machine learning algorithm to analyze the image data and extract the attributes of the living space. This enables the user to efficiently find the design they want and smoothly select and purchase related products.

[0666] A "user" is an individual or legal entity that uses this system to obtain design suggestions for living spaces and information on related products.

[0667] "Residential space" refers to the indoor space used by individuals or corporations for their daily lives, and this system is the subject of design proposals.

[0668] "Image data" refers to digital information files that visually record living spaces, and it is the input data for this system.

[0669] A "communication device" is an electronic device used to transmit data, and is a means used when a user sends image data of their living space to a server.

[0670] "Preprocessing" refers to the initial processing performed on image data, which is a technical process to standardize the data and reduce noise prior to analysis.

[0671] A "machine learning algorithm" is a computational method used by computers to learn patterns from data, and in this system, it is a method used to analyze and extract attributes of living spaces.

[0672] A "knowledge generation model" is an AI technology that learns from large datasets and generates new information and suggestions. In this system, it is used to generate design suggestions.

[0673] "Design proposals" refer to creative ideas generated by the server regarding the arrangement or style of interiors suitable for a living space.

[0674] "Items" refers to products and goods related to design proposals for living spaces that are provided to users through this system.

[0675] This interior design system allows users to easily obtain design proposals for their living spaces. First, the user takes or selects image data of their living space using their own device and uploads it to the server via a communication device. The uploaded image data undergoes preprocessing and standardization on the server. This preprocessing includes image resizing and noise reduction.

[0676] Subsequently, the server uses machine learning algorithms to analyze the image data and extract attributes of the living space. This process utilizes computer vision libraries (e.g., OpenCV and TensorFlow) to collect information such as wall color, flooring material, and window location. The extracted attribute data is then input into a knowledge generation model (e.g., a generative AI model).

[0677] The server generates design suggestions by passing prompt statements to the AI ​​model, which then utilizes relevant knowledge to produce design proposals. An example of a prompt statement is, "Analyze a photo of a living room and propose Scandinavian-style interior design ideas." Based on this prompt, the model outputs multiple design suggestions suitable for the living space.

[0678] The generated design proposals are sent from the server to the user's terminal, where the user can review the proposals and select their preferred design. After the user selects a design proposal, the server retrieves information on related items through the e-commerce site and presents it to the user. The user can then select the items they wish to purchase from the proposed items and proceed with the online purchase process. This entire process allows users to efficiently realize their interior design for their living spaces.

[0679] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0680] Step 1:

[0681] The user takes a photo of their living space using their device or selects an existing image. This image data is uploaded to the server using a communication device. The input is the selected image file, and the output is the image data sent to the server. Specifically, the user launches the application on their device, selects an image according to the instructions, and presses the upload button.

[0682] Step 2:

[0683] The server performs preprocessing on the received image data. Specifically, it standardizes the image size and removes noise using filtering techniques. The input is the unprocessed image data sent by the user, and the output is the clean image data after preprocessing. This process ensures accurate subsequent feature extraction.

[0684] Step 3:

[0685] The server uses machine learning algorithms to analyze preprocessed image data and extract attributes of the living space. The input is preprocessed image data, and the output is extracted attribute data. Specifically, the server calls a computer vision library and analyzes pixel information in the image to detect wall color, flooring material, window positions, etc.

[0686] Step 4:

[0687] The server inputs the extracted attribute data into a knowledge generation model to generate design proposals. At this time, prompt statements are used to instruct the AI ​​model on specific design themes. The input consists of attribute data and prompt statements, and the output is multiple candidate design proposals. For example, the server sends the prompt "Analyze a photo of a living room and propose a Scandinavian-style interior design plan."

[0688] Step 5:

[0689] The generated design proposals are sent from the server to the user's terminal. The user reviews the proposals through the interface on their terminal and selects their preferred design. The input is the design proposals sent from the server, and the output is the design selected by the user. Specifically, the user scrolls through the displayed proposal images and confirms their selection by pressing a selection button.

[0690] Step 6:

[0691] The server searches for items related to the design proposal selected by the user and retrieves that information from e-commerce sites. The input is the design selected by the user and the category information of the related items, and the output is a list of suggested items. Specifically, the server uses APIs to request product prices and review information from multiple e-commerce sites.

[0692] Step 7:

[0693] Users can select items they wish to purchase from a presented list. After selection, users proceed with the online purchase process using a link on their device. The input is the user's item selection, and the output is the progress towards the purchase of the selected items. Specifically, the user checks the details page for each item and completes the payment process by clicking the purchase button.

[0694] (Application Example 1)

[0695] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0696] Modern consumers demand quick and accurate suggestions when choosing interior design products in a physical space, but traditional methods are time-consuming and labor-intensive. This is especially true in brick-and-mortar stores, where providing prompt and personalized interior design suggestions tailored to customer needs is difficult, hindering immediate purchase decisions.

[0697] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0698] In this invention, the server includes a unit for users to upload image data of a room, a unit for analyzing the image data and extracting the characteristics of the room, and a unit for generating interior layout plans using a generated information processing model based on the extracted characteristic data. This makes it possible to immediately propose the optimal interior layout plan when serving customers in a physical store, using a wearable visual enhancement device within the store.

[0699] A "user-uploadable room image data unit" is a system component that has the function of allowing users to transmit visual information of their own room or any physical space in digital format.

[0700] A "unit that analyzes image data to extract room characteristics" refers to an algorithm or device that processes visual information and extracts useful information by identifying the features and components of a space.

[0701] A "generative information processing model" refers to artificial intelligence or learning algorithms that perform advanced optimization and generate layout proposals based on input data.

[0702] A "unit for generating interior layout plans" is a system element that has the function of constructing an optimal interior design tailored to the user's preferences and the purpose of the space, based on pre-analyzed spatial data.

[0703] A "visual enhancement device" is a device that assists a user's visual information and can overlay digital information on it. This refers to wearable devices such as smart glasses.

[0704] This invention is a system that enables quick and appropriate interior design proposals in physical stores. The system is configured as follows:

[0705] The server receives image data of the room transmitted from a visual enhancement device worn by the user within the store. The image data is captured using a device such as smart glasses and sent to the server. The server analyzes the received image data and extracts its characteristics. This is done using software that applies computer vision technology, specifically the OpenCV library. This allows for the analysis of wall color schemes, furniture arrangement, and spatial shape.

[0706] Based on the analyzed characteristic data, the server automatically generates interior layout plans using a generative information processing model. This process employs a generative AI model and machine learning frameworks such as TensorFlow and PyTorch. As a result, multiple interior layout plans are generated, taking into account specific themes and customer preferences.

[0707] The generated placement options are displayed on the user's visual enhancement device. This user interface was developed using Unity and ARKit, allowing the user to see the optimal placement options in real time.

[0708] As a concrete example, if a user sends data using a visual enhancement device to the living room section of a store, the server can present a simple Scandinavian-style interior layout plan. In this case, an example of a prompt message might be "Living room / Scandinavian style / Warm-toned wooden furniture." This allows the user to quickly and easily make a purchase decision without hesitation.

[0709] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0710] Step 1:

[0711] The user captures image data of the room using a visual enhancement device they are wearing. This image data is transmitted to the server via a terminal. The input is visual information of the real space, and the server receives the image data as output. This supplies the server with the necessary image information.

[0712] Step 2:

[0713] The server analyzes the received image data. Here, computer vision techniques are used to denoise and resize the images. Specifically, the OpenCV library is used to remove noise from the images and convert them to an appropriate resolution. The input is the image data obtained in step 1, and the output is the denoised, analyzable image data.

[0714] Step 3:

[0715] The server extracts room characteristics from the analyzed image data. Specifically, it applies algorithms to identify spatial features such as color and shape. The input is the image data processed in step 2, and the output is characteristic data such as the room's color and furniture arrangement.

[0716] Step 4:

[0717] The server inputs the extracted characteristic data into a generative information processing model to generate interior layout proposals. During this process, a generative AI model using TensorFlow or PyTorch is executed. The input is the characteristic data obtained in step 3, and the output is multiple interior layout proposals.

[0718] Step 5:

[0719] The server displays the generated interior layout plan on the display of the user's visual augmentation device. The user can then visually confirm this. The input is the interior layout plan generated in step 4, and the output is the graphical display information shown on the visual augmentation device.

[0720] Step 6:

[0721] The user selects their preferred interior layout from the presented options and proceeds to the purchasing process. The selected layout is sent to the server, where relevant product information is retrieved and presented to the user. The input is the user's selection of the layout presented in step 5, and the output is information on purchasable products. This sequence of actions allows the user to immediately proceed to purchasing.

[0722] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0723] This invention is a system that proposes interior design using image data of a room provided by the user, taking into account the user's emotional state. First, the user uploads photos and floor plans of the room to the system via a terminal. This image data is then transmitted from the user's terminal to the server.

[0724] The server uses computer vision technology to process the image data received from the terminal, extracting features such as wall color, floor material, and room shape. Based on this, the artificial intelligence model generates interior design proposals. These proposals take into account the colors, layout, and style of the interior.

[0725] Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions. The terminal uses the user's camera and microphone to input the user's facial expressions and voice into the emotion engine in real time. The server analyzes the input data and recognizes the user's emotional state. This emotional information is used to adjust the interior design proposal.

[0726] During the adjustment process, the suggested interior color palette and style are dynamically changed according to the user's emotional state. For example, if the user wants to relax, the generating AI model will prioritize suggesting interior designs with calming colors. Conversely, if the user is in an active emotional state, bright and vibrant colors will be suggested.

[0727] The generated interior design proposals are presented to the user on their device. The user can view the proposed designs and select the one they like. Based on the design proposal selected by the user, the server searches for relevant product information from e-commerce sites and displays it as a list of suggestions for the user. The user selects the products they wish to purchase from this list and proceeds with the purchase process through the e-commerce site.

[0728] For example, if the emotion engine determines that the user desires a calm atmosphere, the server, through its generative AI model, will propose an interior design plan based on cool colors such as blue and green. The user can then purchase furniture selected from the list according to this proposal, easily creating their ideal room. In this way, the present invention provides personalized interior design that responds to the user's emotions.

[0729] The following describes the processing flow.

[0730] Step 1:

[0731] The user selects photos and floor plans of a room on their device and uploads them through the system interface. The device then prepares to send this data and sends the image data to the server.

[0732] Step 2:

[0733] The server receives the transmitted image data and begins analysis using computer vision technology. The analysis includes processes to extract room features such as wall color, furniture arrangement, and floor material from the image.

[0734] Step 3:

[0735] The server inputs the extracted feature data into an artificial intelligence model to generate optimal interior design proposals for the room. The generated proposals take into account color harmony and spatial efficiency.

[0736] Step 4:

[0737] The user uses their device to input facial expressions and voice into the emotion engine via the camera and microphone to acquire emotion data. The device then sends this data to the server.

[0738] Step 5:

[0739] The server analyzes the received facial and voice data to recognize the user's emotional state. Specifically, it evaluates emotions such as joy, sadness, and calmness.

[0740] Step 6:

[0741] The server takes the user's emotional state into consideration and adjusts the generated interior design suggestions accordingly. For example, if the user indicates a desire to relax, the server will adjust the colors to soft, warm tones.

[0742] Step 7:

[0743] The adjusted interior design proposals are sent from the server to the user's terminal and presented on the interface. The user can view multiple proposals and choose their preferred one.

[0744] Step 8:

[0745] Based on the user's chosen outfit, the server searches for related furniture and accessories from e-commerce sites. The server then creates a list of suggested products and sends it to the user's device.

[0746] Step 9:

[0747] Users can view a product list on their device and select the items they wish to purchase. They can then proceed with the purchase process on the e-commerce site via the purchase link for their selected items.

[0748] (Example 2)

[0749] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0750] Conventional interior design systems have struggled to flexibly respond to users' emotions and preferences, making it difficult to provide appropriate suggestions tailored to individual needs. Furthermore, while users desire a system that allows for interior adjustments that take their emotional state into consideration, suitable technology has not yet existed.

[0751] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0752] In this invention, the server includes means for a user to transmit image data of a living space via an input device, a device for analyzing the image data and extracting features of the living space, and means for creating an interior layout plan using an artificial intelligence model generated based on the extracted feature data. This makes it possible to propose personalized interior coordination that responds to the user's emotions.

[0753] "Living space" refers to the indoor area where a user lives, and is a concept that includes rooms or parts of a house that are the subject of interior design.

[0754] An "input device" is a device used by a user to supply information to a system, and refers to electronic terminals such as smartphones, tablets, and computers.

[0755] "Image data" refers to digital information in which a user visually records the state of their living space, and includes photographs and drawing data.

[0756] A "server" is the central information processing device of a system, and its role is to collect, analyze, store, and distribute data.

[0757] A "generating artificial intelligence model" is a computer program equipped with a process for analyzing and generating data through learning, and is a technology used to propose optimal interior layout plans to users.

[0758] An "interior design plan" is a blueprint for the composition and layout of interiors in a living space, and includes proposals that encompass elements such as color, material, shape, and arrangement.

[0759] "Items" refer to parts, tools, furniture, etc., used to make up the interior of a living space, and are items that can be purchased as concrete products.

[0760] "Emotional state" refers to information that indicates a user's psychological reactions and emotional tendencies, and is an element used in adjusting interior design.

[0761] "Dynamic adjustment" refers to a process in which the system changes and optimizes results and suggestions in real time based on user input and circumstances.

[0762] The system based on this invention is primarily realized through three entities: a server, a terminal, and a user.

[0763] First, the user sends image data of their living space to the system via a device. This device can be a smartphone, tablet, or computer, and the images are sent over the internet, with the data being sent from the device to the server.

[0764] The server analyzes the received image data and extracts features of the living space. For this purpose, the server utilizes computer vision technology. Specifically, information processing libraries such as TensorFlow and OpenCV are used to extract features such as wall color, floor material, and room shape.

[0765] Next, the server utilizes an artificial intelligence model generated based on the extracted feature data to produce interior layout proposals. The generating AI model refers to a training database and constructs proposals considering the interior's colors, layout, and style. The interior coordination proposals are customized to suit the user's individual preferences.

[0766] Furthermore, the device uses the user's camera and microphone to acquire their emotional state. The emotion engine analyzes the user's facial expression and voice data to determine their emotional state, such as whether they are relaxed or active.

[0767] The server takes this emotional state into account and dynamically adjusts the interior design layout. For example, if the user desires a calm atmosphere, the server, through a generative AI model, will suggest and present interior designs in cool colors such as blue and green. An example of a prompt message to set in this case would be, "I want to make the room a relaxing space."

[0768] The generated interior design proposals are presented to the user again via the device. The user selects their preferred proposal from the presented options, and related item information is automatically collected, generating a suggestion list from e-commerce sites. The user then considers purchasing items from this list, and by actually buying the selected items, the interior design can be realized.

[0769] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0770] Step 1:

[0771] The user takes images of their living space using their device and uploads them to the system. The input is digital image data, and the server receives this data as output; therefore, the image data is sent to the server. Specifically, the user takes a picture of their room with their smartphone camera and operates the upload button within the app.

[0772] Step 2:

[0773] The server analyzes the received image data using computer vision technology. The input is the image data sent in step 1, and the output is feature data of the living space, such as wall color, floor material, and room shape. Specifically, it uses OpenCV to perform image processing such as edge detection on the image and identify each element of the room.

[0774] Step 3:

[0775] The server uses a generation AI model, taking feature data as input, to generate interior layout proposals. The output is an interior coordination proposal based on color, arrangement, and style. Specifically, using TensorFlow or similar tools, a model trained on an existing interior database generates multiple layout proposals.

[0776] Step 4:

[0777] The device uses its built-in camera and microphone to capture the user's facial expressions and voice, collecting emotional data in real time. Input is the user's visual and audio data, and output is data for emotion analysis. Specifically, the device application activates the camera and microphone to record the user's facial expressions and voice.

[0778] Step 5:

[0779] The server analyzes the user's emotional data obtained from the terminal to determine the user's emotional state. The input is the emotional data collected in step 4, and the output is an emotional state such as "relaxed" or "active." Specifically, it uses an emotion recognition API to analyze the data and categorize the results.

[0780] Step 6:

[0781] The server adjusts the layout suggestions obtained from the generating AI model according to the user's emotional state. The input is the layout suggestions from step 3 and the emotional state from step 5, and the output is the adjusted layout suggestions. For example, if a relaxed state is desired, the server will adjust the layout to emphasize cool color tones.

[0782] Step 7:

[0783] The device presents the user with a refined interior design plan. The input is the layout plan refined in step 6, and the output is the visual information displayed on the device screen. Specifically, multiple options are displayed in a list format on the app screen, which the user can scroll through to view them.

[0784] Step 8:

[0785] The server retrieves item information related to the selected interior design plan. The input is the plan selected by the user, and the output is a list of item information obtained from the e-commerce site. Specifically, the server calls the API of the partner e-commerce site to collect product information that matches the selected style.

[0786] Step 9:

[0787] The terminal presents the user with product information obtained from the e-commerce site and accepts their purchase selection. The input is the product information list from step 8, and the output is the user's purchase decision. Specifically, the interface displays a product list and provides a purchase button, allowing the user to proceed directly with the purchase process.

[0788] (Application Example 2)

[0789] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0790] Interior design proposals that meet the diverse needs of modern consumers rely on static designs, which has the problem of not adequately addressing the emotional states and personalities of individual customers. Furthermore, there is a lack of means to immediately respond to customers' emotions with product suggestions in physical stores, so there is a need to improve the customer experience.

[0791] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0792] In this invention, the server includes means for the user to acquire images of the facility, means for analyzing the image data to extract features of the facility, and means for generating interior design proposals using a generative artificial intelligence model based on the extracted feature data and the user's emotional state. This makes it possible to provide dynamic and personalized interior design suggestions and product information tailored to the emotions of individual customers.

[0793] "User" refers to an individual or group receiving interior design proposals.

[0794] "Images of the facility" are visual data acquired by users that show the internal environment of the facility.

[0795] "Feature extraction" is the process of identifying and analyzing important information such as color and shape from image data.

[0796] "Emotional state" refers to information that represents the user's current psychological state and emotions.

[0797] A "generative artificial intelligence model" is an algorithm or system that dynamically generates interior design proposals based on input data.

[0798] An "interior design proposal" is a proposal document for interior design provided to the user.

[0799] "Product information" refers to detailed data such as price, specifications, and stock status of products related to the proposed interior design plan.

[0800] The system of this invention enables users to receive interior design suggestions using devices such as smartphones and tablets within facilities such as physical stores or their homes. The system operates in the following steps:

[0801] First, the user uses their device's camera at any location within the facility to capture "images of the facility." This image data is immediately sent to the server. The server analyzes the received image data using computer vision technologies such as OpenCV to "extract features" of the facility. This allows elements such as color and shape to be clearly identified.

[0802] Next, the device's camera and microphone are used to analyze the user's "emotional state." This allows the system to recognize emotions in real time from the user's facial expressions and voice using technologies such as the Emotion AI API. This emotional information is crucial data that is input into the generative AI model.

[0803] The generative artificial intelligence model generates "interior design proposals" tailored to the user based on acquired feature data and emotional states. This process utilizes advanced AI algorithms such as GPT-3.5, dynamically generating designs based on prompt text.

[0804] For example, if a user desires a calm and relaxed shopping environment in a store, and this sentiment is recognized, the AI ​​will suggest interior design options based on blue and green tones. An example of a prompt message might be: "Based on the following information, please suggest an interior design that matches the customer's mood. The store images were taken by a customer who prefers a relaxed atmosphere."

[0805] Finally, the generated interior design proposals are displayed on the terminal and presented to the user along with relevant "product information." The user can instantly select the presented products and purchase them through online shopping.

[0806] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0807] Step 1:

[0808] The user uses a terminal to acquire images within the facility and sends the image data to the server. The input is still image data captured through the terminal's camera, and the output is an image file transferred to the server. Through this operation, the user provides basic data for interior evaluation.

[0809] Step 2:

[0810] The server analyzes the received image files using OpenCV and extracts features from the facility. Specifically, it identifies features such as color, shape, and texture. The input is the image file transferred to the server, and the output is the extracted feature data. This feature data serves as basic information for interior design proposals.

[0811] Step 3:

[0812] The device captures the user's facial expressions and voice in real time via the Emotion AI API and recognizes their emotional state. In this process, camera and microphone data are the input, and the analyzed emotional state information is the output. This emotional data reflects the user's psychological state and has a significant impact on interior design recommendations.

[0813] Step 4:

[0814] The server uses a generative artificial intelligence model to generate interior design proposals based on acquired feature data and emotional information. The input is feature data and emotional state information, and the output is the generated interior design proposal. Specifically, GPT-3.5 uses prompts to construct the optimal design proposal.

[0815] Step 5:

[0816] The server sends the generated interior design proposal to the terminal and presents it to the user. The input is the generated interior design proposal, and the output is a visual display of the proposal on the terminal. The user uses this proposal as a reference to select products.

[0817] Step 6:

[0818] The user selects items they like based on the presented interior design proposals. The server uses this selection data to build a list of related product information and presents it to the user. The input is the product information selected by the user, and the output is a product list that includes detailed information.

[0819] Step 7:

[0820] The user selects the items they wish to purchase from the displayed product list and places an order. The server confirms the order details and completes the transaction. The input is the user's purchase instruction, and the output is the completed transaction data. This allows the user to improve their interior design through their store experience.

[0821] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0822] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0823] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0824] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0825] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0826] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0827] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0828] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0829] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0830] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0831] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0832] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0833] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0834] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0835] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0836] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0837] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0838] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0839] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0840] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0841] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0842] The following is further disclosed regarding the embodiments described above.

[0843] (Claim 1)

[0844] A means for users to upload image data of their rooms,

[0845] A means for analyzing the aforementioned image data to extract the characteristics of the room,

[0846] A means for generating interior design proposals using a generative artificial intelligence model based on extracted feature data,

[0847] A means of presenting the generated interior design proposals to the user,

[0848] A means of obtaining and presenting information on products related to the selected interior design plan,

[0849] A means for accepting the purchase selection of the aforementioned product,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, wherein the analysis of the image data is performed using computer vision technology to determine the colors and shapes of the room.

[0853] (Claim 3)

[0854] The system according to claim 1, wherein the generating artificial intelligence model classifies coordination proposals based on interior coordination themes.

[0855] "Example 1"

[0856] (Claim 1)

[0857] A means by which a user transmits image data of their living space using a communication device,

[0858] A means for performing information preprocessing and standardization on the aforementioned image data,

[0859] A means of using a machine learning algorithm to analyze the aforementioned image data and extract the attributes of the living space,

[0860] A means of generating design proposals using a knowledge generation model based on extracted attribute data,

[0861] A means of displaying the generated design proposals to the user,

[0862] A means of obtaining and presenting information about items related to the design proposal selected by the user,

[0863] A means for accepting the purchase option of the aforementioned goods,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, wherein the analysis of the image data is performed using information processing technology to identify the color tone and structure of the living space.

[0867] (Claim 3)

[0868] The system according to claim 1, wherein the knowledge generation model classifies proposals based on design themes.

[0869] "Application Example 1"

[0870] (Claim 1)

[0871] A unit for users to upload image data of a room,

[0872] A unit that analyzes the aforementioned image data to extract the characteristics of the room,

[0873] A unit that generates interior layout plans using a generated information processing model based on extracted characteristic data,

[0874] A unit that presents the generated interior layout plan to the user,

[0875] A unit that retrieves and presents product information related to the selected interior layout plan,

[0876] A unit that accepts the purchase selection of the aforementioned product,

[0877] A unit that uses the generated information processing model to immediately propose interior layout plans using a wearable visual enhancement device in a store,

[0878] A system that includes this.

[0879] (Claim 2)

[0880] The system according to claim 1, wherein the analysis of the image data is performed using information processing technology to determine the color scheme and structure of the room.

[0881] (Claim 3)

[0882] The system according to claim 1, wherein the generated information processing model categorizes layout plans based on interior layout themes.

[0883] "Example 2 of combining an emotion engine"

[0884] (Claim 1)

[0885] A means for a user to transmit image data of their living space via an input device,

[0886] A device for analyzing the aforementioned image data and extracting features of the living space,

[0887] A means of creating interior layout plans using an artificial intelligence model generated based on extracted feature data,

[0888] A means of displaying the generated layout plan to the user,

[0889] A means for acquiring and displaying item information related to the selected layout plan,

[0890] Means for processing the selection of the aforementioned items to be purchased,

[0891] A means for detecting the user's emotional state,

[0892] A means for dynamically adjusting the layout plan based on the aforementioned emotional state,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, wherein the analysis of the image data is performed using information processing technology to determine the color and form of the living space.

[0896] (Claim 3)

[0897] The system according to claim 1, wherein the artificial intelligence model generated classifies layout plans based on interior layout themes, and further adjusts the layout plans by utilizing prompts based on the user's emotional state.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] A means for users to obtain images within the facility,

[0901] A means for analyzing the aforementioned image data to extract features within the facility,

[0902] A means for generating interior design proposals using a generative artificial intelligence model based on extracted feature data and the user's emotional state,

[0903] A means of presenting the generated interior design proposals to the user,

[0904] A means of obtaining and presenting information on products related to the selected interior design proposal,

[0905] A means of accepting the option to purchase the aforementioned product,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, wherein the analysis of the image data is performed using information processing technology to determine the colors and shapes within the facility.

[0909] (Claim 3)

[0910] The system according to claim 1, wherein the generating artificial intelligence model classifies proposals based on interior design themes. [Explanation of Symbols]

[0911] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to upload image data of their rooms, A means for analyzing the aforementioned image data to extract the characteristics of the room, A means for generating interior design proposals using a generative artificial intelligence model based on extracted feature data, A means of presenting the generated interior design proposals to the user, A means of obtaining and presenting information on products related to the selected interior design plan, A means for accepting the purchase selection of the aforementioned product, A system that includes this.

2. The system according to claim 1, wherein the analysis of the image data is performed using computer vision technology to determine the colors and shapes of the room.

3. The system according to claim 1, wherein the generating artificial intelligence model classifies coordination proposals based on interior coordination themes.

Images