system

Abstract

We provide the system. [Solution] A means for the user to select building elements, A means for the server to search the database based on the selected element and retrieve the corresponding image data, A means of displaying image data acquired by the terminal to the user, A means of reflecting user-selected elements onto the drawing, A means for generating a composite drawing based on a drawing reflected by a server, A means by which the terminal displays a composite diagram, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern architectural design, there is a problem that it is difficult for customers to have a specific image. Since there are few systems that can integrally image specific elements such as floor plans, cross-sections, fixtures, doors, and window sashes, customers feel difficulty in reflecting their own desires in the design. The purpose of the present invention is to make the customer's image clearer and streamline the design process by selecting and visualizing these elements.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for a user to select building elements, means for a server to search a database based on the selected elements and obtain corresponding image data, means for a terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the server to generate a composite drawing based on the reflected drawing, and means for the terminal to display the composite drawing. This allows the user to select specific elements of a desired building and visually confirm a composite drawing based on them.

[0006] A "user" is a person or entity that selects specific elements of a building and utilizes the system.

[0007] A "server" is a computer system that has the function of searching a database based on selected elements, retrieving corresponding image data, and generating composite diagrams.

[0008] A "terminal" is a device used by a user to select building elements and display acquired image data and composite diagrams.

[0009] "Building elements" refer to specific components or materials involved in the design and construction of a building, such as floor plans, wallpaper, fixtures, doors, and window frames.

[0010] A "database" is a collection of information, including image data related to the elements of a building, and is what a server searches for.

[0011] "Image data" refers to digital image files used to provide a visual representation of building elements.

[0012] A "drawing" refers to a picture or design plan used to visually represent the design and layout of a building.

[0013] A "composite diagram" is a visual image generated by the server that reflects the building elements selected by the user. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[0036] A natural language explanation of the program's processing.

[0037] The user selects a building element.

[0038] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[0039] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[0040] The server searches the database.

[0041] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[0042] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[0043] Acquisition and transmission of image data

[0044] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[0045] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[0046] Displaying images on a device

[0047] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[0048] The user selects how to apply the changes to the drawing.

[0049] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[0050] The terminal sends information about the selected element to the server.

[0051] Generation and display of composite diagrams

[0052] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[0053] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[0054] The terminal decodes the received composite image and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[0055] Specific example

[0056] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[0057] The system of this invention allows users to visualize in detail how selected elements fit into the overall building design. This streamlines the design process and improves user satisfaction.

[0058] The following describes the processing flow.

[0059] Step 1:

[0060] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[0061] Step 2:

[0062] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[0063] Step 3:

[0064] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[0065] Step 4:

[0066] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[0067] Step 5:

[0068] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[0069] SQL

[0070] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[0071] SELECT image FROM Images WHERE category='Cross' AND color='White';

[0072] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[0073] Step 6:

[0074] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[0075] Step 7:

[0076] The server converts this list into a data format such as JSON and sends it to the terminal.

[0077] Step 8:

[0078] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[0079] Step 9:

[0080] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[0081] Step 10:

[0082] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[0083] Step 11:

[0084] The device sends the ID and password of the image selected by the user to the server.

[0085] Step 12:

[0086] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[0087] Step 13:

[0088] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[0089] Step 14:

[0090] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[0091] Step 15:

[0092] The terminal decodes the received binary data into image data.

[0093] Step 16:

[0094] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[0095] Through these steps, users can visually see how their selected elements will be combined and create a building design that closely matches their vision.

[0096] (Example 1)

[0097] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] Existing building design systems had a problem where it was difficult for users to intuitively visualize how selected building elements would be reflected in actual drawings. Furthermore, there was a lack of systems that could generate and quickly display composite drawings based on user selections. This resulted in an inefficient design process and low user satisfaction.

[0099] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0100] In this invention, the server includes means for the user to select building elements, means for the terminal to transmit the user's selection information to the server, means for the server to generate a search query based on the selected elements, search a database and obtain corresponding image data, means for the server to transmit the obtained image data to the terminal in list format, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the terminal to transmit information about the selected elements to the server, means for the server to generate a composite drawing based on the reflected drawing, and means for the terminal to display the composite drawing to the user. As a result, the elements selected by the user are intuitively visualized, the design process is made more efficient, and user satisfaction is improved.

[0101] A "user" refers to an individual or group that uses the system to select building elements and participate in the design process.

[0102] A "terminal" refers to an electronic device (e.g., computer, smartphone, tablet) that a user uses to access and operate a system.

[0103] A "server" refers to a computer system that receives user selection information and performs processing such as database searches, image data acquisition, processing, and transmission.

[0104] "Building elements" refer to specific components that users can select in the design of a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0105] A "database" refers to a system for storing and managing information and image data related to the elements of a building.

[0106] A "search query" refers to a specific search command generated to retrieve relevant information from a database based on the user's selections.

[0107] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[0108] "List format" refers to a data format that enumerates and stores multiple pieces of data in a structured manner.

[0109] "User interface" refers to the screen display and operating means that allow a user to operate a system and visually confirm its output.

[0110] A "composite drawing" refers to a comprehensive design drawing generated by reflecting the building elements selected by the user.

[0111] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[0112] The user selects building elements.

[0113] The device provides a user interface, such as a web browser or a dedicated app. This interface displays dropdown menus and radio buttons for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). Through these interfaces, the user might, for example, select "floor plan" and then "spacious."

[0114] Send user selection information to the server

[0115] The device sends the user's selected information to the server as a JSON-formatted request. The selected information includes the "element type" and "selection options".

[0116] The server searches the database.

[0117] The server parses the received request and generates an appropriate search query. For example, if the element type is "floor plan" and the selection option is "spacious," it generates a query like "SELECT FROM images WHERE type='floor plan' AND option='spacious'." The server then executes this query in a database management system (e.g., MySQL®, PostgreSQL, etc.) to retrieve the corresponding image data.

[0118] Acquire and send image data.

[0119] The server compiles the image data retrieved from the database into a list format and sends it to the terminal in JSON format.

[0120] The device displays image data.

[0121] The terminal parses the JSON data received from the server and displays the images on the user interface. Specifically, the images are displayed in a list format as thumbnails. This allows the user to visually confirm the images.

[0122] The user selects how to apply the changes to the drawing.

[0123] The user selects the desired element from the displayed thumbnail image by clicking or tapping, and then specifies the placement of that element. The device then sends this selection and placement information back to the server in JSON format.

[0124] The server generates the composite image.

[0125] The server generates a composite image based on the elements selected by the user. Image processing techniques (e.g., OpenCV, Pillow, etc.) are used in this process. After generating the composite image, the server may encode this image data into binary format.

[0126] Send and display the composite image.

[0127] The server sends the generated composite diagram to the terminal, which decodes and displays it in the user interface. This allows the user to visually confirm how the selected elements are reflected in the actual design drawing.

[0128] Specific example

[0129] For example, when designing a new living room, the user first selects "spacious" for the "layout," "white" for the "wallpaper," and "wood grain" for the "door" through the UI. The device sends this selection information to the server in JSON format. The server parses the received information, retrieves the corresponding image data from the database, and sends it to the device. The device displays these images as thumbnails, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite drawing, which is then sent to the device. Finally, the user can view the composite drawing and see how their image is reflected in the actual design.

[0130] Example of a prompt

[0131] "When designing a new room, I'd like to make the living room spacious. I also want white wallpaper and wood-grain doors. I'd like to retrieve image data that meets these conditions from a server and generate a composite drawing that incorporates these elements into the floor plan."

[0132] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0133] Step 1:

[0134] The user selects elements of the building.

[0135] Specifically, the user selects a "floor plan" through the device's user interface, for example, using a dropdown menu or radio buttons, and then selects "spacious" from the options.

[0136] Input: User selection of elements (e.g., floor plan = spacious)

[0137] Output: Selected element information (Example: {"Element Type":"Floor Plan","Selection Option":"Spacious"})

[0138] Step 2:

[0139] The device sends the information selected by the user to the server.

[0140] Specifically, the terminal packets the selection information in JSON format and sends it to the server as an HTTP request.

[0141] Input: Selected element information (Example: {"Element Type": "Floor Plan","Selection Option": "Spacious"})

[0142] Output: Sending data to the server via HTTP request

[0143] Step 3:

[0144] The server parses the received request and generates a database search query.

[0145] Specifically, the server parses the JSON and generates a query such as "SELECT FROM images WHERE type='floor plan' AND option='spacious'".

[0146] Input: Element information received from the terminal (e.g., {"Element Type":"Floor Plan", "Selection Option":"Spacious"})

[0147] Output: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[0148] Step 4:

[0149] The server executes a search query on the database management system and retrieves the corresponding image data.

[0150] Specifically, the server executes queries against database management systems such as MySQL and PostgreSQL to retrieve relevant image data.

[0151] Input: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[0152] Output: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0153] Step 5:

[0154] The server sends the acquired image data to the terminal in JSON format.

[0155] Specifically, the server compiles the acquired data into a list format, encodes it into JSON, and sends it to the terminal as an HTTP response.

[0156] Input: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0157] Output: Image data in JSON format (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0158] Step 6:

[0159] The terminal parses the JSON data received from the server and displays the image on the user interface.

[0160] Specifically, the device parses the JSON data and displays the images in a list format as thumbnails.

[0161] Input: Image data in JSON format received from the server (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0162] Output: Thumbnail image displayed on the user interface

[0163] Step 7:

[0164] The user selects the desired element from the displayed thumbnail images and specifies its placement.

[0165] Specifically, users select a thumbnail image by clicking or tapping it, and then determine its placement by dragging and dropping it.

[0166] Input: Displayed thumbnail image and user selection and placement information

[0167] Output: Information on the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[0168] Step 8:

[0169] The terminal then sends the selected elements and their placement information back to the server in JSON format.

[0170] Specifically, the terminal packets the selected elements and their placement information and sends them to the server as an HTTP request.

[0171] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[0172] Output: Sending data to the server via HTTP request

[0173] Step 9:

[0174] The server generates a composite diagram based on the information it receives.

[0175] Specifically, the server uses image processing technology (e.g., OpenCV, Pillow) to generate a composite image that reflects the selected elements in the drawing.

[0176] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[0177] Output: Image data of the generated composite image

[0178] Step 10:

[0179] The server sends the generated composite diagram to the terminal.

[0180] Specifically, the composite diagram is encoded in binary format and sent to the terminal as an HTTP response.

[0181] Input: Image data of the generated composite image

[0182] Output: Binary data of the composite diagram

[0183] Step 11:

[0184] The terminal decodes the composite diagram received from the server and displays it on the user interface.

[0185] Specifically, the terminal decodes the binary data and displays a composite diagram so that the user can verify it.

[0186] Input: Binary data of the composite diagram

[0187] Output: Composite diagram displayed in the user interface

[0188] (Application Example 1)

[0189] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0190] Traditional food delivery applications lack a mechanism for users to select their desired elements and receive personalized delivery options based on those selections. As a result, the process of users selecting each element and visually confirming their desired dishes is cumbersome, leading to decreased user satisfaction.

[0191] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0192] In this invention, the server includes means for the user to select elements of an object, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for the user to reflect the selected elements onto a composite object, means for the server to generate a composite image based on the reflected composite object, means for the terminal to display the composite image, and means for the user to select a desired menu from the displayed elements and select personalized options. This makes it possible for the user to select each element while visually confirming it and to efficiently and easily select personalized delivery options that suit their individual needs.

[0193] An "object" refers to a specific element or item that a user can select and manipulate.

[0194] A "server" is a central processing unit that receives requests from users, searches a database, and provides the appropriate data.

[0195] A "database" is a data storage system in which multiple pieces of information and data are systematically stored and can be easily searched and retrieved.

[0196] "Image data" refers to data of a still image that contains visual information and is intended to be displayed to the user.

[0197] A "terminal" is a device that a user operates and uses to input information (e.g., a smartphone or tablet).

[0198] "User interface" is a general term for the operating screens and input methods that allow users to directly interact with a system.

[0199] A "composite" is a virtual composition or layout diagram generated based on the elements selected by the user.

[0200] A "composite image" is a visual composite image generated based on multiple elements selected by the user.

[0201] "Personalized options" are choices that are customized based on the user's past selection history and individual needs.

[0202] A "menu" is a collection of specific options or items that a user can choose from.

[0203] This invention provides a system that offers a series of processes in which a user selects elements of an object, a server retrieves relevant image data from a database based on that selection, and a terminal displays and manages that data. Specific embodiments are described below.

[0204] Means by which the user selects elements of an object

[0205] Users can select elements of an object using their device. For example, the device's user interface (UI) may display dropdown menus or card-style UI components, which users can use to select elements.

[0206] A means by which the server searches the database and retrieves the corresponding image data.

[0207] The device sends information about the elements selected by the user to the server in JSON format. The server parses the received selection information, searches the appropriate database, and retrieves image data related to the selected elements.

[0208] A means of displaying image data acquired by a device to the user.

[0209] Image data retrieved from the server is sent to the terminal in JSON format. The terminal parses the received data and displays the image data so that the user can visually confirm it.

[0210] A means of reflecting user-selected elements onto a composite object.

[0211] The user selects desired elements from image data displayed on their device and reflects them onto a composite object (a virtual composition or layout diagram). The elements selected by the user are then sent from the device to the server.

[0212] A server is a means for generating a composite image based on a composite that has been reflected.

[0213] The server generates a composite image based on the elements selected by the user. Image processing techniques (such as OpenCV or Pillow) are used for generation.

[0214] Means by which a terminal displays a composite image

[0215] The server generates a composite image and sends it to the terminal, which then displays it to the user. This allows the user to visually see how their selected elements will appear in the composite image.

[0216] A means for users to select personalized options

[0217] After the user reviews the composite image, they can select their desired menu from the displayed elements and request personalized options from the server. The server will then provide appropriate personalized options, taking into account the user's past selections and other information.

[0218] Hardware and software to be used

[0219] To implement this system, the following hardware and software will be used:

[0220] UI development: React Native, Flutter (registered trademark)

[0221] Data transmission and analysis: REST API, JSON format

[0222] Image data processing: Python, OpenCV, Pillow

[0223] Database: MySQL, MongoDB

[0224] Server: Node.js, Express

[0225] Cloud services: AWS (registered trademark) EC2, S3

[0226] Specific example

[0227] For example, if a user selects "steak" as their "main dish," the device sends this information to the server in JSON format. The server parses the selection, retrieves image data corresponding to "steak" from its database, and sends it back to the device. The device displays the image data, allowing the user to select their desired elements and ultimately providing a personalized delivery option.

[0228] Example of a prompt

[0229] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[0230] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0231] Step 1:

[0232] The device displays a user interface. It displays dropdown menus or card-style UI components to make it easier for the user to select elements of an object (e.g., main dish, drink, dessert). The input is a list of selectable elements of the object, and the output is the selected element.

[0233] Step 2:

[0234] The user selects elements of an object. For example, they might select "main dish" and then choose "steak" from the options. The input is the options displayed in the user interface, and the output is the specific element selected by the user.

[0235] Step 3:

[0236] The device sends the user's selection to the server. The information of the selected element is converted to JSON format and sent using a REST API. The input is the element selected by the user, and the output is the JSON data sent to the server.

[0237] Step 4:

[0238] The server searches the database based on the data it receives. It parses the selected information and generates relevant image data as a search query. The input is JSON data containing information about the elements selected by the user, and the output is image data from the database search results.

[0239] Step 5:

[0240] The server collects image data, compiles it into JSON format, and sends it to the terminal. The input is image data retrieved from the database, and the output is the JSON data sent to the terminal.

[0241] Step 6:

[0242] The terminal analyzes image data received from the server. It parses the JSON data and displays the image. The input is the JSON data sent from the server, and the output is the displayed image data.

[0243] Step 7:

[0244] The user selects a desired element from the displayed image and inputs that information into the device. The input is the displayed image data, and the output is the specific image element selected by the user.

[0245] Step 8:

[0246] The device sends the selected element to the server. The selection information is converted back to JSON format and sent using the REST API. The input is the information of the element selected by the user, and the output is the JSON data sent to the server.

[0247] Step 9:

[0248] The server generates a composite image based on the elements it receives. It combines the images of each element to create a single composite image. For example, it uses image processing techniques such as OpenCV or Pillow. The input is information on multiple elements selected by the user, and the output is the generated composite image.

[0249] Step 10:

[0250] The server generates a composite image and sends it to the terminal. The image data is encoded in binary format and sent to the terminal. The input is the generated composite image, and the output is the composite image data sent to the terminal.

[0251] Step 11:

[0252] The terminal decodes the composite image and displays it to the user. This allows the user to visually confirm the composite image. The input is the binary composite image data received from the server, and the output is the displayed composite image.

[0253] Step 12:

[0254] The user reviews the displayed composite image and selects personalized options. The input is the displayed composite image, and the output is the personalized options selected by the user.

[0255] Step 13:

[0256] The device sends personalized option information to the server. The selection information is converted to JSON format and sent using a REST API. The input is the options selected by the user, and the output is the JSON data sent to the server.

[0257] Step 14:

[0258] The server generates the final delivery information based on the personalized options it receives. The input is the information of the options selected by the user, and the output is the generated delivery information.

[0259] Step 15:

[0260] The server sends the final generated delivery information to the terminal, which then displays it. The input is the generated delivery information, and the output is the displayed delivery information.

[0261] Specific examples of operation

[0262] For example, when a user selects "steak" as their "main dish" in the app, that information is sent to the server in JSON format. The server retrieves image data of "steak" from its database based on this information and sends it to the device. The device displays the image data to the user, who selects the type of "steak" they want, and sends the information back to the server. The server generates a composite image based on the selected information and sends it to the device. The user reviews this image and finally selects their personalized delivery option.

[0263] Example of a prompt

[0264] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[0265] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0266] The present invention provides a system in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is enhanced.

[0267] A natural language explanation of the program's processing.

[0268] The user selects a building element.

[0269] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[0270] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[0271] The server searches the database.

[0272] The terminal sends the user's selection information to the server. The selection information is sent, for example, in JSON format.

[0273] The server analyzes the received information and searches an appropriate database based on that information. A search query is generated to obtain image data related to the elements of the building.

[0274] Acquisition and Transmission of Image Data

[0275] The server obtains the corresponding image data from the database. For example, it obtains the image data of "spacious floor plans".

[0276] The server collates the acquired image data in list form and sends it to the terminal. The data is sent, for example, in JSON format.

[0277] Display of Images by the Terminal

[0278] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, crosses, doors, etc. are displayed as thumbnails.

[0279] User Selection of Reflection on the Drawing

[0280] The user selects the desired elements from the displayed thumbnail images and determines their placement on the drawing. [[ID=三十三]]

[0281] The terminal sends the information of the selected elements to the server.

[0282] Generation and Display of Composite Drawings

[0283] The server generates a composite drawing based on the elements selected by the user. For this, image processing technologies (e.g., OpenCV, Pillow, etc.) are used.

[0284] The server sends the composite drawing to the terminal, and in so doing, may encode the image data in binary format.

[0285] The terminal decodes the received composite drawing and displays it to the user. This allows the user to confirm how the selected elements will look in the actual drawing.

[0286] Integration of the Emotion Engine

[0287] The emotion engine determines the emotional state based on the user's selections and reactions. For example, it recognizes emotions in real time through the analysis of the user's facial expressions and voice.

[0288] The server receives information from the emotion engine and provides means to present building elements recommended based on the user's emotional state. For example, if the user is in a "happy" state, it recommends bright-colored crosses or modern-designed doors.

[0289] Also, the emotion engine automatically adjusts the color scheme and design of the image data to be displayed according to the user's emotional state. For example, when the user is "depressed", it selects relaxing colors and designs.

[0290] Specific Example

[0291] For example, consider the case of imagining the design of a new living room. The user first selects "spacious" as the "layout", "white" as the "cross", and "wood grain" as the "door" through the UI. This selection information is sent to the server, the server retrieves the corresponding image data from the database, and sends it to the terminal. The terminal displays these images for the user to visually confirm. When the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing and sends it to the terminal. Finally, the user can view the composite drawing and confirm how their image is reflected in the actual design drawing.

[0292] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. The emotion engine automatically adjusts the color scheme and design to ensure the user has a better experience.

[0293] The system of this invention allows users to visualize in detail how their selected elements fit into the overall building design, and further enhances user satisfaction through an emotional engine. This streamlines the design process and improves user satisfaction.

[0294] The following describes the processing flow.

[0295] Step 1:

[0296] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[0297] Step 2:

[0298] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[0299] Step 3:

[0300] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[0301] Step 4:

[0302] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[0303] Step 5:

[0304] The server generates a search query and uses it to search the database. For example, perform the following search using an SQL query:

[0305] sql

[0306] SELECT image FROM Images WHERE category='Floor Plan' AND type='Wide';

[0307] SELECT image FROM Images WHERE category='Cross' AND color='White';

[0308] SELECT image FROM Images WHERE category='Door' AND style='Wood Grain';

[0309] Step 6:

[0310] The server combines the image data retrieved from the database into a list. The list includes those corresponding to each selected element.

[0311] Step 7:

[0312] The server converts this list into a data format such as JSON and sends it to the terminal.

[0313] Step 8:

[0314] The terminal analyzes the received JSON data and displays a list on the screen for each category (floor plan, cross, door, etc.).

[0315] Step 9:

[0316] The terminal displays the images in thumbnail form so that the user can intuitively select them. For example, each image is displayed in a clickable format.

[0317] Step 10:

[0318] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[0319] Step 11:

[0320] The device sends the ID and password of the image selected by the user to the server.

[0321] Step 12:

[0322] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[0323] Step 13:

[0324] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[0325] Step 14:

[0326] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[0327] Step 15:

[0328] The terminal decodes the received binary data into image data.

[0329] Step 16:

[0330] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[0331] Step 17:

[0332] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time. Specifically, it collects data using cameras and microphones and performs facial recognition and voice tone analysis.

[0333] Step 18:

[0334] The device sends data from the emotion engine to the server. This allows the user's emotional state to be transmitted to the server in real time.

[0335] Step 19:

[0336] The server analyzes the user's emotional state based on data from the emotion engine and responds appropriately. For example, if the user is "having fun," the server might suggest brighter colored wallpaper or a door with a modern design.

[0337] Step 20:

[0338] The server sends building elements recommended by the user based on their emotional state to the terminal in JSON format or similar.

[0339] Step 21:

[0340] The device displays recommended elements it has received to the user. For example, it displays appropriate options as "recommended cross-references."

[0341] Step 22:

[0342] The emotion engine adjusts the color scheme and design of the image data displayed according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[0343] Step 23:

[0344] The server adjusts the color scheme and design of the image data based on instructions from the emotion engine and generates the composite image again.

[0345] Step 24:

[0346] The device then displays a composite image, adjusted by the emotion engine, to the user again. This adjustment allows the user to see a design that matches their emotions.

[0347] Through these steps, users can visually see how their selected elements will combine and create building designs that closely match their vision. Furthermore, the emotion engine can enhance user satisfaction. This streamlines the design process and improves user satisfaction.

[0348] (Example 2)

[0349] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0350] Conventional building design systems have a cumbersome process for not only visually confirming user-selected elements but also reflecting them in actual design drawings, resulting in low usability. Furthermore, the lack of suggestions and adjustments tailored to the user's emotional state leads to decreased user satisfaction and inefficient design processes. Therefore, the present invention aims to solve these problems and provide a system that allows users to design buildings intuitively and efficiently.

[0351] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0352] In this invention, the server includes means for the user to select building elements, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for an emotion analysis engine to analyze the user's emotional state based on the user's selections and reactions and make appropriate suggestions and adjust the image data. As a result, the user can proceed with the design while intuitively visualizing the selected elements, and furthermore, suggestions and adjustments according to the emotional state can be made to improve the efficiency of the design process and user satisfaction.

[0353] A "user" refers to a person who operates the system and selects elements of a building.

[0354] A "terminal" refers to a device operated by a user (for example, a PC, smartphone, or tablet), which interacts with the system through a user interface.

[0355] A "server" is a computer device that processes requests from users and performs tasks such as searching databases and generating composite diagrams.

[0356] A "database" is a data storage system used to store and manage building elements and related image data.

[0357] "Building elements" refer to the specific parts and features necessary for building design (for example, floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0358] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[0359] A "composite drawing" refers to a single drawing constructed by combining elements of multiple buildings selected by the user, and is a drawing that is applied to actual building design.

[0360] An "emotion analysis engine" refers to software or algorithms that analyze a user's choices and reactions to determine the user's emotional state.

[0361] A "search query" refers to the instructions or conditions that a server generates to search a database.

[0362] The present invention provides a series of processes in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion analysis engine, the user experience is enhanced.

[0363] Specifically, the system uses the following hardware and software:

[0364] Hardware and software

[0365] Device: A device operated by a user, such as a PC, smartphone, or tablet.

[0366] Server: A computer device used for database searches and the generation of composite diagrams.

[0367] Database: A data storage system for storing and managing building elements and related image data.

[0368] Emotion analysis engine: Software that analyzes user choices and reactions to determine the user's emotional state (e.g., algorithms that perform facial expression analysis or voice analysis).

[0369] Detailed explanation of the program

[0370] The user selects a building element.

[0371] The device launches the user interface and displays UI components (dropdown menus, radio buttons, etc.) for selecting building elements (floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0372] Users make selections using these UI components.

[0373] The server searches the database.

[0374] The terminal organizes the user's selection information and sends it to the server as data in JSON format.

[0375] The server analyzes the received data and generates search queries based on the selected elements.

[0376] The server searches the database and retrieves the relevant image data (e.g., floor plan, wallpaper, etc.).

[0377] Acquisition and display of image data

[0378] The server sends the acquired image data to the terminal as a list-formatted JSON data.

[0379] The terminal decodes the received data and extracts a list of image data to be displayed.

[0380] The device displays each image as a thumbnail, allowing the user to visually confirm its contents.

[0381] The user selects how to apply the changes to the drawing.

[0382] The user selects the desired element from the displayed thumbnail images and places it on the drawing.

[0383] Generation and display of composite diagrams

[0384] The device then sends the information of the selected element back to the server in JSON format.

[0385] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information.

[0386] The server encodes the generated composite diagram into binary format and sends it to the terminal.

[0387] The terminal decodes the received binary data and displays the composite diagram as an image.

[0388] Integration of emotion analysis engine

[0389] The emotion analysis engine analyzes user choices and reactions (facial expressions, voice, etc.) in real time.

[0390] The server receives feedback information from the emotion engine and provides recommended elements (color, design, etc.) according to the user's emotional state.

[0391] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state.

[0392] Specific example

[0393] For example, consider the case of imagining the design of a new living room. The user first selects "spacious" as the "floor plan," "white" as the "wallpaper," and "wood grain" as the "door" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from the database and sends it to the terminal. The terminal displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing, which is then sent to the terminal. Finally, the user can view the composite drawing and see how their image is reflected in the actual design drawing.

[0394] Example of a prompt

[0395] "Design a living room. Choose a 'spacious' layout, 'white' wallpaper, and 'wood-grain' doors. If the user is satisfied with these choices, add elements that contribute to a relaxing atmosphere."

[0396] This invention allows users to intuitively visualize selected elements while proceeding with the design, and improves the efficiency of the design process and user satisfaction through suggestions and adjustments tailored to their emotional state.

[0397] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0398] Step 1:

[0399] The terminal launches a user interface and displays options for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). UI components such as dropdown menus and radio buttons are used. This allows the user to select the desired element from the options. The input for Step 1 is "User interaction," and the output is "Selected building element."

[0400] Step 2:

[0401] The user selects building elements using the displayed UI components. For example, they might select "Spacious" from the floor plan options and "White" from the wallpaper options. The input for Step 2 is the "UI components," and the output is the "elements selected by the user."

[0402] Step 3:

[0403] The terminal organizes the user's selection information and sends it to the server as JSON data. HTTP POST requests are often used for this transmission. The input for Step 3 is "user selection information," and the output is "JSON data sent to the server."

[0404] Step 4:

[0405] The server parses the JSON data received from the terminal to determine which elements have been selected. Next, it generates a search query based on the selected elements and searches the database. This process retrieves the corresponding image data. The input for step 4 is "JSON data," and the output is the "search query" and the "retrieved image data."

[0406] Step 5:

[0407] The server compiles the acquired image data into a list-formatted JSON data and sends it to the terminal. The input for Step 5 is "image data," and the output is "the JSON data sent to the terminal."

[0408] Step 6:

[0409] The terminal decodes the JSON data received from the server and extracts a list of image data to be displayed. These images are displayed in the UI as their respective thumbnail images. The input for Step 6 is "JSON data," and the output is "the displayed thumbnail images."

[0410] Step 7:

[0411] The user selects the desired element from the displayed thumbnail images and decides where to place it on the drawing. The input for Step 7 is the "thumbnail image," and the output is the "drawing placement selected by the user."

[0412] Step 8:

[0413] The terminal then sends the elements and placement information selected by the user back to the server in JSON format. The input for step 8 is "drawing placement information," and the output is "JSON data sent to the server."

[0414] Step 9:

[0415] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information. The input for step 9 is "placement information," and the output is the "generated composite image."

[0416] Step 10:

[0417] The server encodes the generated composite diagram into binary format and sends it to the terminal. The input for step 10 is the "composite diagram," and the output is the "binary data sent to the terminal."

[0418] Step 11:

[0419] The terminal decodes the received binary data and displays the composite drawing as an image. This allows the user to see how the selected elements appear in the actual drawing. The input for step 11 is "binary data," and the output is the "displayed composite drawing."

[0420] Step 12:

[0421] The emotion analysis engine analyzes the user's choices and reactions (e.g., facial expressions and voice) in real time to determine the user's emotional state. The input for step 12 is "the user's choices and reactions," and the output is "the emotional state as a result of the analysis."

[0422] Step 13:

[0423] The server receives feedback information from the emotion analysis engine and presents recommended building elements based on the user's emotional state. The input for step 13 is "emotional state feedback," and the output is "recommended elements."

[0424] Step 14:

[0425] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state. The input for step 14 is the "emotional state," and the output is the "adjusted image data."

[0426] (Application Example 2)

[0427] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0428] In modern building design, users desire to visually confirm, select, and customize building components. However, current systems struggle to provide suggestions and adjust image data while considering the user's emotional state, which can result in decreased user satisfaction. Furthermore, the visualization of how building components are reflected in composite drawings is insufficient, hindering an efficient design process. Therefore, there is a need for improved design efficiency and enhanced user experience.

[0429] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0430] In this invention, the server includes means for the user to select building components, means for the server to search a database based on the selected components and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected components on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for analyzing the user's emotional state using an emotion engine and recommending components or adjusting image data based on the analysis results. As a result, the user can select components while visually confirming them, and further improve satisfaction through suggestions and adjustments by the emotion engine.

[0431] A "user" is an individual or group that uses this system to select building components and participate in the design process.

[0432] "Building components" refer to specific parts or elements inside or outside a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0433] A "server" is a central device that receives selection information from the user, searches a database to retrieve the corresponding image data, and transmits it to the terminal.

[0434] A "database" is a storage device that stores information such as image data related to the components of a building, and allows a server to search for and retrieve that data.

[0435] A "terminal" is an electronic device (e.g., smartphone, tablet, computer) that a user operates to display image data retrieved from a server and to verify composite diagrams.

[0436] "Image data" refers to digital image information used to visually represent the components of a building.

[0437] A "composite drawing" is an overall design drawing of a building generated based on the components selected by the user.

[0438] The "emotion engine" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state in real time, and uses the results to recommend components and adjust image data.

[0439] The system of this invention provides a series of processes in which the user selects building components, the server acquires corresponding image data based on the selection, and the terminal displays and manages it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience is improved.

[0440] Hardware and software

[0441] Hardware: Smartphones, tablets, computers

[0442] Software: Python 3, Pillow library, requests library, sentiment engine, database management system, user interface (UI) framework

[0443] Data processing and data calculation

[0444] When a user selects a component, the server receives that selection information in JSON format, searches the database, and retrieves the corresponding image data. The retrieved data is then sent to the terminal in JSON format.

[0445] The terminal analyzes the received image data and displays it for the user to visually confirm. It reflects the user's selected components on the drawing and sends that information to the server. It also has a function that analyzes the user's facial expressions and voice and recognizes their emotional state in real time using an emotion engine.

[0446] The emotion engine recommends components and adjusts image data based on the analysis results.

[0447] Specific example

[0448] For example, consider a scenario where a user is imagining the design of a new living room. First, the user selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through a smartphone app. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing and sends it to the device. Finally, the user can view the composite drawing and confirm how their image has been reflected in the design.

[0449] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. In this way, the emotion engine automatically adjusts the color scheme and design, allowing users to have a better experience.

[0450] Example of a prompt

[0451] Use the following prompt:

[0452] Please select the interior elements that the user would like to install in the store. For example, "large display," "wood-grain table," "modern shelf," etc. Based on the user's selection, retrieve and display the appropriate image data. Furthermore, analyze the user's emotions and provide optimal suggestions according to their emotional state.

[0453] This concludes the detailed description of the embodiments of this invention. By using this system, users can proceed with the building design process comfortably and efficiently.

[0454] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0455] Step 1:

[0456] The user selects the building components.

[0457] Users select building components (e.g., floor plan, wallpaper, doors) through a smartphone or tablet application. This selection information is entered via a user interface (UI), and the input information is converted into JSON format.

[0458] Input: User-selected components (e.g., spacious floor plan, white wallpaper, wood-grain door)

[0459] Output: Selection information for components in JSON format (Example: {"layout":"wide", "cross":"white", "door":"wood grain"})

[0460] Step 2:

[0461] The device sends the selection information to the server.

[0462] The device sends the user-selected JSON-formatted information to the server. An HTTP POST request is used for this transmission.

[0463] Input: Selection information for components in JSON format

[0464] Output: HTTP POST request to the server

[0465] Step 3:

[0466] The server searches the database.

[0467] The server parses the received JSON-formatted information and searches the database for image data related to the building's components based on that information. The search results retrieve the corresponding image data.

[0468] Input: Selection information for components in JSON format

[0469] Output: Relevant image data (e.g., image of a spacious floor plan, image of white wallpaper, image of a wood-grain door)

[0470] Step 4:

[0471] The server sends image data to the terminal.

[0472] The server searches for and retrieves image data, compiles it into a list, encodes it in JSON format, and sends it to the terminal.

[0473] Input: Relevant image data

[0474] Output: List of image data encoded in JSON format

[0475] Step 5:

[0476] Display image data received by the device.

[0477] The terminal analyzes the JSON-formatted image data received from the server and displays thumbnail images of each component on the user interface (UI).

[0478] Input: A list of image data encoded in JSON format.

[0479] Output: Thumbnail image displayed to the user

[0480] Step 6:

[0481] The user selects how to apply the changes to the drawing.

[0482] The user selects the desired components from the displayed thumbnail images and confirms their placement on the drawing. This selection information is also compiled again in JSON format and sent from the terminal to the server.

[0483] Input: Thumbnail image selected by the user

[0484] Output: Component placement information summarized in JSON format

[0485] Step 7:

[0486] The server generates a composite image.

[0487] The server generates a composite diagram using image processing techniques (e.g., OpenCV, Pillow, etc.) based on the arrangement information of the components selected by the user.

[0488] Input: Component placement information compiled in JSON format

[0489] Output: Image data of the generated composite image

[0490] Step 8:

[0491] The server sends the composite image to the terminal.

[0492] The server encodes the generated composite diagram in JSON format and sends it to the terminal.

[0493] Input: Image data of the generated composite image

[0494] Output: Composite diagram encoded in JSON format

[0495] Step 9:

[0496] The device displays a composite image.

[0497] The terminal decodes the composite diagram in JSON format received from the server and displays it on the user interface (UI). This allows the user to see how the selected components look in the actual diagram.

[0498] Input: Composite diagram encoded in JSON format

[0499] Output: Composite diagram displayed to the user

[0500] Step 10:

[0501] The emotion engine analyzes the user's emotions.

[0502] The device inputs the user's facial expressions and voice into an emotion engine, which analyzes them in real time. As a result, the user's emotional state is determined.

[0503] Input: User facial expression data, voice data

[0504] Output: User's emotional state (e.g., satisfied, relaxed)

[0505] Step 11:

[0506] The server makes recommendations and adjustments to components based on emotional state.

[0507] The server receives emotional states from the emotion engine and, based on that, recommends appropriate components or adjusts the currently displayed image data.

[0508] Input: User's emotional state

[0509] Output: Image data of the adjusted components, additional recommended components

[0510] The above describes each processing step and its specific operation in the system of the present invention. This system allows users to comfortably and efficiently select and customize building designs.

[0511] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0512] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0513] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0514] [Second Embodiment]

[0515] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0516] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0517] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0518] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0519] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0520] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0521] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0522] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0523] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0524] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0525] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0526] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0527] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[0528] A natural language explanation of the program's processing.

[0529] The user selects a building element.

[0530] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[0531] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[0532] The server searches the database.

[0533] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[0534] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[0535] Acquisition and transmission of image data

[0536] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[0537] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[0538] Displaying images on a device

[0539] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[0540] The user selects how to apply the changes to the drawing.

[0541] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[0542] The terminal sends information about the selected element to the server.

[0543] Generation and display of composite diagrams

[0544] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[0545] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[0546] The terminal decodes the received composite image and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[0547] Specific example

[0548] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[0549] The system of this invention allows users to visualize in detail how selected elements fit into the overall building design. This streamlines the design process and improves user satisfaction.

[0550] The following describes the processing flow.

[0551] Step 1:

[0552] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[0553] Step 2:

[0554] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[0555] Step 3:

[0556] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[0557] Step 4:

[0558] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[0559] Step 5:

[0560] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[0561] SQL

[0562] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[0563] SELECT image FROM Images WHERE category='Cross' AND color='White';

[0564] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[0565] Step 6:

[0566] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[0567] Step 7:

[0568] The server converts this list into a data format such as JSON and sends it to the terminal.

[0569] Step 8:

[0570] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[0571] Step 9:

[0572] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[0573] Step 10:

[0574] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[0575] Step 11:

[0576] The device sends the ID and password of the image selected by the user to the server.

[0577] Step 12:

[0578] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[0579] Step 13:

[0580] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[0581] Step 14:

[0582] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[0583] Step 15:

[0584] The terminal decodes the received binary data into image data.

[0585] Step 16:

[0586] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[0587] Through these steps, users can visually see how their selected elements will be combined and create a building design that closely matches their vision.

[0588] (Example 1)

[0589] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0590] Existing building design systems had a problem where it was difficult for users to intuitively visualize how selected building elements would be reflected in actual drawings. Furthermore, there was a lack of systems that could generate and quickly display composite drawings based on user selections. This resulted in an inefficient design process and low user satisfaction.

[0591] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0592] In this invention, the server includes means for the user to select building elements, means for the terminal to transmit the user's selection information to the server, means for the server to generate a search query based on the selected elements, search a database and obtain corresponding image data, means for the server to transmit the obtained image data to the terminal in list format, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the terminal to transmit information about the selected elements to the server, means for the server to generate a composite drawing based on the reflected drawing, and means for the terminal to display the composite drawing to the user. As a result, the elements selected by the user are intuitively visualized, the design process is made more efficient, and user satisfaction is improved.

[0593] A "user" refers to an individual or group that uses the system to select building elements and participate in the design process.

[0594] A "terminal" refers to an electronic device (e.g., computer, smartphone, tablet) that a user uses to access and operate a system.

[0595] A "server" refers to a computer system that receives user selection information and performs processing such as database searches, image data acquisition, processing, and transmission.

[0596] "Building elements" refer to specific components that users can select in the design of a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0597] A "database" refers to a system for storing and managing information and image data related to the elements of a building.

[0598] A "search query" refers to a specific search command generated to retrieve relevant information from a database based on the user's selections.

[0599] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[0600] "List format" refers to a data format that enumerates and stores multiple pieces of data in a structured manner.

[0601] "User interface" refers to the screen display and operating means that allow a user to operate a system and visually confirm its output.

[0602] A "composite drawing" refers to a comprehensive design drawing generated by reflecting the building elements selected by the user.

[0603] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[0604] The user selects building elements.

[0605] The device provides a user interface, such as a web browser or a dedicated app. This interface displays dropdown menus and radio buttons for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). Through these interfaces, the user might, for example, select "floor plan" and then "spacious."

[0606] Send user selection information to the server

[0607] The device sends the user's selected information to the server as a JSON-formatted request. The selected information includes the "element type" and "selection options".

[0608] The server searches the database.

[0609] The server parses the received request and generates an appropriate search query. For example, if the element type is "floor plan" and the selection option is "spacious," it generates a query like "SELECT FROM images WHERE type='floor plan' AND option='spacious'." The server then executes this query in a database management system (e.g., MySQL, PostgreSQL, etc.) to retrieve the corresponding image data.

[0610] Acquire and send image data.

[0611] The server compiles the image data retrieved from the database into a list format and sends it to the terminal in JSON format.

[0612] The device displays image data.

[0613] The terminal parses the JSON data received from the server and displays the images on the user interface. Specifically, the images are displayed in a list format as thumbnails. This allows the user to visually confirm the images.

[0614] The user selects how to apply the changes to the drawing.

[0615] The user selects the desired element from the displayed thumbnail image by clicking or tapping, and then specifies the placement of that element. The device then sends this selection and placement information back to the server in JSON format.

[0616] The server generates the composite image.

[0617] The server generates a composite image based on the elements selected by the user. Image processing techniques (e.g., OpenCV, Pillow, etc.) are used in this process. After generating the composite image, the server may encode this image data into binary format.

[0618] Send and display the composite image.

[0619] The server sends the generated composite diagram to the terminal, which decodes and displays it in the user interface. This allows the user to visually confirm how the selected elements are reflected in the actual design drawing.

[0620] Specific example

[0621] For example, when designing a new living room, the user first selects "spacious" for the "layout," "white" for the "wallpaper," and "wood grain" for the "door" through the UI. The device sends this selection information to the server in JSON format. The server parses the received information, retrieves the corresponding image data from the database, and sends it to the device. The device displays these images as thumbnails, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite drawing, which is then sent to the device. Finally, the user can view the composite drawing and see how their image is reflected in the actual design.

[0622] Example of a prompt

[0623] "When designing a new room, I'd like to make the living room spacious. I also want white wallpaper and wood-grain doors. I'd like to retrieve image data that meets these conditions from a server and generate a composite drawing that incorporates these elements into the floor plan."

[0624] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0625] Step 1:

[0626] The user selects elements of the building.

[0627] Specifically, the user selects a "floor plan" through the device's user interface, for example, using a dropdown menu or radio buttons, and then selects "spacious" from the options.

[0628] Input: User selection of elements (e.g., floor plan = spacious)

[0629] Output: Selected element information (Example: {"Element Type":"Floor Plan","Selection Option":"Spacious"})

[0630] Step 2:

[0631] The device sends the information selected by the user to the server.

[0632] Specifically, the terminal packets the selection information in JSON format and sends it to the server as an HTTP request.

[0633] Input: Selected element information (Example: {"Element Type": "Floor Plan","Selection Option": "Spacious"})

[0634] Output: Sending data to the server via HTTP request

[0635] Step 3:

[0636] The server parses the received request and generates a database search query.

[0637] Specifically, the server parses the JSON and generates a query such as "SELECT FROM images WHERE type='floor plan' AND option='spacious'".

[0638] Input: Element information received from the terminal (e.g., {"Element Type":"Floor Plan", "Selection Option":"Spacious"})

[0639] Output: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[0640] Step 4:

[0641] The server executes a search query on the database management system and retrieves the corresponding image data.

[0642] Specifically, the server executes queries against database management systems such as MySQL and PostgreSQL to retrieve relevant image data.

[0643] Input: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[0644] Output: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0645] Step 5:

[0646] The server sends the acquired image data to the terminal in JSON format.

[0647] Specifically, the server compiles the acquired data into a list format, encodes it into JSON, and sends it to the terminal as an HTTP response.

[0648] Input: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0649] Output: Image data in JSON format (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0650] Step 6:

[0651] The terminal parses the JSON data received from the server and displays the image on the user interface.

[0652] Specifically, the device parses the JSON data and displays the images in a list format as thumbnails.

[0653] Input: Image data in JSON format received from the server (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[0654] Output: Thumbnail image displayed on the user interface

[0655] Step 7:

[0656] The user selects the desired element from the displayed thumbnail images and specifies its placement.

[0657] Specifically, users select a thumbnail image by clicking or tapping it, and then determine its placement by dragging and dropping it.

[0658] Input: Displayed thumbnail image and user selection and placement information

[0659] Output: Information on the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[0660] Step 8:

[0661] The terminal then sends the selected elements and their placement information back to the server in JSON format.

[0662] Specifically, the terminal packets the selected elements and their placement information and sends them to the server as an HTTP request.

[0663] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[0664] Output: Sending data to the server via HTTP request

[0665] Step 9:

[0666] The server generates a composite diagram based on the information it receives.

[0667] Specifically, the server uses image processing technology (e.g., OpenCV, Pillow) to generate a composite image that reflects the selected elements in the drawing.

[0668] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[0669] Output: Image data of the generated composite image

[0670] Step 10:

[0671] The server sends the generated composite diagram to the terminal.

[0672] Specifically, the composite diagram is encoded in binary format and sent to the terminal as an HTTP response.

[0673] Input: Image data of the generated composite image

[0674] Output: Binary data of the composite diagram

[0675] Step 11:

[0676] The terminal decodes the composite diagram received from the server and displays it on the user interface.

[0677] Specifically, the terminal decodes the binary data and displays a composite diagram so that the user can verify it.

[0678] Input: Binary data of the composite diagram

[0679] Output: Composite diagram displayed in the user interface

[0680] (Application Example 1)

[0681] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0682] Traditional food delivery applications lack a mechanism for users to select their desired elements and receive personalized delivery options based on those selections. As a result, the process of users selecting each element and visually confirming their desired dishes is cumbersome, leading to decreased user satisfaction.

[0683] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0684] In this invention, the server includes means for the user to select elements of an object, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for the user to reflect the selected elements onto a composite object, means for the server to generate a composite image based on the reflected composite object, means for the terminal to display the composite image, and means for the user to select a desired menu from the displayed elements and select personalized options. This makes it possible for the user to select each element while visually confirming it and to efficiently and easily select personalized delivery options that suit their individual needs.

[0685] An "object" refers to a specific element or item that a user can select and manipulate.

[0686] A "server" is a central processing unit that receives requests from users, searches a database, and provides the appropriate data.

[0687] A "database" is a data storage system in which multiple pieces of information and data are systematically stored and can be easily searched and retrieved.

[0688] "Image data" refers to data of a still image that contains visual information and is intended to be displayed to the user.

[0689] A "terminal" is a device that a user operates and uses to input information (e.g., a smartphone or tablet).

[0690] "User interface" is a general term for the operating screens and input methods that allow users to directly interact with a system.

[0691] A "composite" is a virtual composition or layout diagram generated based on the elements selected by the user.

[0692] A "composite image" is a visual composite image generated based on multiple elements selected by the user.

[0693] "Personalized options" are choices that are customized based on the user's past selection history and individual needs.

[0694] A "menu" is a collection of specific options or items that a user can choose from.

[0695] This invention provides a system that offers a series of processes in which a user selects elements of an object, a server retrieves relevant image data from a database based on that selection, and a terminal displays and manages that data. Specific embodiments are described below.

[0696] Means by which the user selects elements of an object

[0697] Users can select elements of an object using their device. For example, the device's user interface (UI) may display dropdown menus or card-style UI components, which users can use to select elements.

[0698] A means by which the server searches the database and retrieves the corresponding image data.

[0699] The device sends information about the elements selected by the user to the server in JSON format. The server parses the received selection information, searches the appropriate database, and retrieves image data related to the selected elements.

[0700] A means of displaying image data acquired by a device to the user.

[0701] Image data retrieved from the server is sent to the terminal in JSON format. The terminal parses the received data and displays the image data so that the user can visually confirm it.

[0702] A means of reflecting user-selected elements onto a composite object.

[0703] The user selects desired elements from image data displayed on their device and reflects them onto a composite object (a virtual composition or layout diagram). The elements selected by the user are then sent from the device to the server.

[0704] A server is a means for generating a composite image based on a composite that has been reflected.

[0705] The server generates a composite image based on the elements selected by the user. Image processing techniques (such as OpenCV or Pillow) are used for generation.

[0706] Means by which a terminal displays a composite image

[0707] The server generates a composite image and sends it to the terminal, which then displays it to the user. This allows the user to visually see how their selected elements will appear in the composite image.

[0708] A means for users to select personalized options

[0709] After the user reviews the composite image, they can select their desired menu from the displayed elements and request personalized options from the server. The server will then provide appropriate personalized options, taking into account the user's past selections and other information.

[0710] Hardware and software to be used

[0711] To implement this system, the following hardware and software will be used:

[0712] UI development: React Native, Flutter

[0713] Data transmission and analysis: REST API, JSON format

[0714] Image data processing: Python, OpenCV, Pillow

[0715] Database: MySQL, MongoDB

[0716] Server: Node.js, Express

[0717] Cloud services: AWS EC2, S3

[0718] Specific example

[0719] For example, if a user selects "steak" as their "main dish," the device sends this information to the server in JSON format. The server parses the selection, retrieves image data corresponding to "steak" from its database, and sends it back to the device. The device displays the image data, allowing the user to select their desired elements and ultimately providing a personalized delivery option.

[0720] Example of a prompt

[0721] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[0722] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0723] Step 1:

[0724] The device displays a user interface. It displays dropdown menus or card-style UI components to make it easier for the user to select elements of an object (e.g., main dish, drink, dessert). The input is a list of selectable elements of the object, and the output is the selected element.

[0725] Step 2:

[0726] The user selects elements of an object. For example, they might select "main dish" and then choose "steak" from the options. The input is the options displayed in the user interface, and the output is the specific element selected by the user.

[0727] Step 3:

[0728] The device sends the user's selection to the server. The information of the selected element is converted to JSON format and sent using a REST API. The input is the element selected by the user, and the output is the JSON data sent to the server.

[0729] Step 4:

[0730] The server searches the database based on the data it receives. It parses the selected information and generates relevant image data as a search query. The input is JSON data containing information about the elements selected by the user, and the output is image data from the database search results.

[0731] Step 5:

[0732] The server collects image data, compiles it into JSON format, and sends it to the terminal. The input is image data retrieved from the database, and the output is the JSON data sent to the terminal.

[0733] Step 6:

[0734] The terminal analyzes image data received from the server. It parses the JSON data and displays the image. The input is the JSON data sent from the server, and the output is the displayed image data.

[0735] Step 7:

[0736] The user selects a desired element from the displayed image and inputs that information into the device. The input is the displayed image data, and the output is the specific image element selected by the user.

[0737] Step 8:

[0738] The device sends the selected element to the server. The selection information is converted back to JSON format and sent using the REST API. The input is the information of the element selected by the user, and the output is the JSON data sent to the server.

[0739] Step 9:

[0740] The server generates a composite image based on the elements it receives. It combines the images of each element to create a single composite image. For example, it uses image processing techniques such as OpenCV or Pillow. The input is information on multiple elements selected by the user, and the output is the generated composite image.

[0741] Step 10:

[0742] The server generates a composite image and sends it to the terminal. The image data is encoded in binary format and sent to the terminal. The input is the generated composite image, and the output is the composite image data sent to the terminal.

[0743] Step 11:

[0744] The terminal decodes the composite image and displays it to the user. This allows the user to visually confirm the composite image. The input is the binary composite image data received from the server, and the output is the displayed composite image.

[0745] Step 12:

[0746] The user reviews the displayed composite image and selects personalized options. The input is the displayed composite image, and the output is the personalized options selected by the user.

[0747] Step 13:

[0748] The device sends personalized option information to the server. The selection information is converted to JSON format and sent using a REST API. The input is the options selected by the user, and the output is the JSON data sent to the server.

[0749] Step 14:

[0750] The server generates the final delivery information based on the personalized options it receives. The input is the information of the options selected by the user, and the output is the generated delivery information.

[0751] Step 15:

[0752] The server sends the final generated delivery information to the terminal, which then displays it. The input is the generated delivery information, and the output is the displayed delivery information.

[0753] Specific examples of operation

[0754] For example, when a user selects "steak" as their "main dish" in the app, that information is sent to the server in JSON format. The server retrieves image data of "steak" from its database based on this information and sends it to the device. The device displays the image data to the user, who selects the type of "steak" they want, and sends the information back to the server. The server generates a composite image based on the selected information and sends it to the device. The user reviews this image and finally selects their personalized delivery option.

[0755] Example of a prompt

[0756] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[0757] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0758] The present invention provides a system in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is enhanced.

[0759] A natural language explanation of the program's processing.

[0760] The user selects a building element.

[0761] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[0762] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[0763] The server searches the database.

[0764] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[0765] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[0766] Acquisition and transmission of image data

[0767] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[0768] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[0769] Displaying images on a device

[0770] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[0771] The user selects how to apply the changes to the drawing.

[0772] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[0773] The terminal sends information about the selected element to the server.

[0774] Generation and display of composite diagrams

[0775] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[0776] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[0777] The terminal decodes the received composite image and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[0778] Emotional engine integration

[0779] The emotion engine determines the user's emotional state based on their choices and responses. For example, it recognizes emotions in real time through analysis of the user's facial expressions and voice.

[0780] The server receives information from the emotion engine and provides a means to suggest building elements based on the user's emotional state. For example, if the user is in a "happy" state, it will recommend brightly colored wallpaper and modernly designed doors.

[0781] Furthermore, the emotion engine automatically adjusts the color scheme and design of the displayed image data according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[0782] Specific example

[0783] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[0784] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. The emotion engine automatically adjusts the color scheme and design to ensure the user has a better experience.

[0785] The system of this invention allows users to visualize in detail how their selected elements fit into the overall building design, and further enhances user satisfaction through an emotional engine. This streamlines the design process and improves user satisfaction.

[0786] The following describes the processing flow.

[0787] Step 1:

[0788] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[0789] Step 2:

[0790] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[0791] Step 3:

[0792] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[0793] Step 4:

[0794] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[0795] Step 5:

[0796] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[0797] SQL

[0798] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[0799] SELECT image FROM Images WHERE category='Cross' AND color='White';

[0800] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[0801] Step 6:

[0802] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[0803] Step 7:

[0804] The server converts this list into a data format such as JSON and sends it to the terminal.

[0805] Step 8:

[0806] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[0807] Step 9:

[0808] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[0809] Step 10:

[0810] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[0811] Step 11:

[0812] The device sends the ID and password of the image selected by the user to the server.

[0813] Step 12:

[0814] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[0815] Step 13:

[0816] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[0817] Step 14:

[0818] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[0819] Step 15:

[0820] The terminal decodes the received binary data into image data.

[0821] Step 16:

[0822] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[0823] Step 17:

[0824] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time. Specifically, it collects data using cameras and microphones and performs facial recognition and voice tone analysis.

[0825] Step 18:

[0826] The device sends data from the emotion engine to the server. This allows the user's emotional state to be transmitted to the server in real time.

[0827] Step 19:

[0828] The server analyzes the user's emotional state based on data from the emotion engine and responds appropriately. For example, if the user is "having fun," the server might suggest brighter colored wallpaper or a door with a modern design.

[0829] Step 20:

[0830] The server sends building elements recommended by the user based on their emotional state to the terminal in JSON format or similar.

[0831] Step 21:

[0832] The device displays recommended elements it has received to the user. For example, it displays appropriate options as "recommended cross-references."

[0833] Step 22:

[0834] The emotion engine adjusts the color scheme and design of the image data displayed according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[0835] Step 23:

[0836] The server adjusts the color scheme and design of the image data based on instructions from the emotion engine and generates the composite image again.

[0837] Step 24:

[0838] The device then displays a composite image, adjusted by the emotion engine, to the user again. This adjustment allows the user to see a design that matches their emotions.

[0839] Through these steps, users can visually see how their selected elements will combine and create building designs that closely match their vision. Furthermore, the emotion engine can enhance user satisfaction. This streamlines the design process and improves user satisfaction.

[0840] (Example 2)

[0841] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0842] Conventional building design systems have a cumbersome process for not only visually confirming user-selected elements but also reflecting them in actual design drawings, resulting in low usability. Furthermore, the lack of suggestions and adjustments tailored to the user's emotional state leads to decreased user satisfaction and inefficient design processes. Therefore, the present invention aims to solve these problems and provide a system that allows users to design buildings intuitively and efficiently.

[0843] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0844] In this invention, the server includes means for the user to select building elements, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for an emotion analysis engine to analyze the user's emotional state based on the user's selections and reactions and make appropriate suggestions and adjust the image data. As a result, the user can proceed with the design while intuitively visualizing the selected elements, and furthermore, suggestions and adjustments according to the emotional state can be made to improve the efficiency of the design process and user satisfaction.

[0845] A "user" refers to a person who operates the system and selects elements of a building.

[0846] A "terminal" refers to a device operated by a user (for example, a PC, smartphone, or tablet), which interacts with the system through a user interface.

[0847] A "server" is a computer device that processes requests from users and performs tasks such as searching databases and generating composite diagrams.

[0848] A "database" is a data storage system used to store and manage building elements and related image data.

[0849] "Building elements" refer to the specific parts and features necessary for building design (for example, floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0850] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[0851] A "composite drawing" refers to a single drawing constructed by combining elements of multiple buildings selected by the user, and is a drawing that is applied to actual building design.

[0852] An "emotion analysis engine" refers to software or algorithms that analyze a user's choices and reactions to determine the user's emotional state.

[0853] A "search query" refers to the instructions or conditions that a server generates to search a database.

[0854] The present invention provides a series of processes in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion analysis engine, the user experience is enhanced.

[0855] Specifically, the system uses the following hardware and software:

[0856] Hardware and software

[0857] Device: A device operated by a user, such as a PC, smartphone, or tablet.

[0858] Server: A computer device used for database searches and the generation of composite diagrams.

[0859] Database: A data storage system for storing and managing building elements and related image data.

[0860] Emotion analysis engine: Software that analyzes user choices and reactions to determine the user's emotional state (e.g., algorithms that perform facial expression analysis or voice analysis).

[0861] Detailed explanation of the program

[0862] The user selects a building element.

[0863] The device launches the user interface and displays UI components (dropdown menus, radio buttons, etc.) for selecting building elements (floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0864] Users make selections using these UI components.

[0865] The server searches the database.

[0866] The terminal organizes the user's selection information and sends it to the server as data in JSON format.

[0867] The server analyzes the received data and generates search queries based on the selected elements.

[0868] The server searches the database and retrieves the relevant image data (e.g., floor plan, wallpaper, etc.).

[0869] Acquisition and display of image data

[0870] The server sends the acquired image data to the terminal as a list-formatted JSON data.

[0871] The terminal decodes the received data and extracts a list of image data to be displayed.

[0872] The device displays each image as a thumbnail, allowing the user to visually confirm its contents.

[0873] The user selects how to apply the changes to the drawing.

[0874] The user selects the desired element from the displayed thumbnail images and places it on the drawing.

[0875] Generation and display of composite diagrams

[0876] The device then sends the information of the selected element back to the server in JSON format.

[0877] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information.

[0878] The server encodes the generated composite diagram into binary format and sends it to the terminal.

[0879] The terminal decodes the received binary data and displays the composite diagram as an image.

[0880] Integration of emotion analysis engine

[0881] The emotion analysis engine analyzes user choices and reactions (facial expressions, voice, etc.) in real time.

[0882] The server receives feedback information from the emotion engine and provides recommended elements (color, design, etc.) according to the user's emotional state.

[0883] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state.

[0884] Specific example

[0885] For example, consider the case of imagining the design of a new living room. The user first selects "spacious" as the "floor plan," "white" as the "wallpaper," and "wood grain" as the "door" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from the database and sends it to the terminal. The terminal displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing, which is then sent to the terminal. Finally, the user can view the composite drawing and see how their image is reflected in the actual design drawing.

[0886] Example of a prompt

[0887] "Design a living room. Choose a 'spacious' layout, 'white' wallpaper, and 'wood-grain' doors. If the user is satisfied with these choices, add elements that contribute to a relaxing atmosphere."

[0888] This invention allows users to intuitively visualize selected elements while proceeding with the design, and improves the efficiency of the design process and user satisfaction through suggestions and adjustments tailored to their emotional state.

[0889] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0890] Step 1:

[0891] The terminal launches a user interface and displays options for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). UI components such as dropdown menus and radio buttons are used. This allows the user to select the desired element from the options. The input for Step 1 is "User interaction," and the output is "Selected building element."

[0892] Step 2:

[0893] The user selects building elements using the displayed UI components. For example, they might select "Spacious" from the floor plan options and "White" from the wallpaper options. The input for Step 2 is the "UI components," and the output is the "elements selected by the user."

[0894] Step 3:

[0895] The terminal organizes the user's selection information and sends it to the server as JSON data. HTTP POST requests are often used for this transmission. The input for Step 3 is "user selection information," and the output is "JSON data sent to the server."

[0896] Step 4:

[0897] The server parses the JSON data received from the terminal to determine which elements have been selected. Next, it generates a search query based on the selected elements and searches the database. This process retrieves the corresponding image data. The input for step 4 is "JSON data," and the output is the "search query" and the "retrieved image data."

[0898] Step 5:

[0899] The server compiles the acquired image data into a list-formatted JSON data and sends it to the terminal. The input for Step 5 is "image data," and the output is "the JSON data sent to the terminal."

[0900] Step 6:

[0901] The terminal decodes the JSON data received from the server and extracts a list of image data to be displayed. These images are displayed in the UI as their respective thumbnail images. The input for Step 6 is "JSON data," and the output is "the displayed thumbnail images."

[0902] Step 7:

[0903] The user selects the desired element from the displayed thumbnail images and decides where to place it on the drawing. The input for Step 7 is the "thumbnail image," and the output is the "drawing placement selected by the user."

[0904] Step 8:

[0905] The terminal then sends the elements and placement information selected by the user back to the server in JSON format. The input for step 8 is "drawing placement information," and the output is "JSON data sent to the server."

[0906] Step 9:

[0907] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information. The input for step 9 is "placement information," and the output is the "generated composite image."

[0908] Step 10:

[0909] The server encodes the generated composite diagram into binary format and sends it to the terminal. The input for step 10 is the "composite diagram," and the output is the "binary data sent to the terminal."

[0910] Step 11:

[0911] The terminal decodes the received binary data and displays the composite drawing as an image. This allows the user to see how the selected elements appear in the actual drawing. The input for step 11 is "binary data," and the output is the "displayed composite drawing."

[0912] Step 12:

[0913] The emotion analysis engine analyzes the user's choices and reactions (e.g., facial expressions and voice) in real time to determine the user's emotional state. The input for step 12 is "the user's choices and reactions," and the output is "the emotional state as a result of the analysis."

[0914] Step 13:

[0915] The server receives feedback information from the emotion analysis engine and presents recommended building elements based on the user's emotional state. The input for step 13 is "emotional state feedback," and the output is "recommended elements."

[0916] Step 14:

[0917] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state. The input for step 14 is the "emotional state," and the output is the "adjusted image data."

[0918] (Application Example 2)

[0919] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0920] In modern building design, users desire to visually confirm, select, and customize building components. However, current systems struggle to provide suggestions and adjust image data while considering the user's emotional state, which can result in decreased user satisfaction. Furthermore, the visualization of how building components are reflected in composite drawings is insufficient, hindering an efficient design process. Therefore, there is a need for improved design efficiency and enhanced user experience.

[0921] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0922] In this invention, the server includes means for the user to select building components, means for the server to search a database based on the selected components and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected components on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for analyzing the user's emotional state using an emotion engine and recommending components or adjusting image data based on the analysis results. As a result, the user can select components while visually confirming them, and further improve satisfaction through suggestions and adjustments by the emotion engine.

[0923] A "user" is an individual or group that uses this system to select building components and participate in the design process.

[0924] "Building components" refer to specific parts or elements inside or outside a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[0925] A "server" is a central device that receives selection information from the user, searches a database to retrieve the corresponding image data, and transmits it to the terminal.

[0926] A "database" is a storage device that stores information such as image data related to the components of a building, and allows a server to search for and retrieve that data.

[0927] A "terminal" is an electronic device (e.g., smartphone, tablet, computer) that a user operates to display image data retrieved from a server and to verify composite diagrams.

[0928] "Image data" refers to digital image information used to visually represent the components of a building.

[0929] A "composite drawing" is an overall design drawing of a building generated based on the components selected by the user.

[0930] The "emotion engine" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state in real time, and uses the results to recommend components and adjust image data.

[0931] The system of this invention provides a series of processes in which the user selects building components, the server acquires corresponding image data based on the selection, and the terminal displays and manages it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience is improved.

[0932] Hardware and software

[0933] Hardware: Smartphones, tablets, computers

[0934] Software: Python 3, Pillow library, requests library, sentiment engine, database management system, user interface (UI) framework

[0935] Data processing and data calculation

[0936] When a user selects a component, the server receives that selection information in JSON format, searches the database, and retrieves the corresponding image data. The retrieved data is then sent to the terminal in JSON format.

[0937] The terminal analyzes the received image data and displays it for the user to visually confirm. It reflects the user's selected components on the drawing and sends that information to the server. It also has a function that analyzes the user's facial expressions and voice and recognizes their emotional state in real time using an emotion engine.

[0938] The emotion engine recommends components and adjusts image data based on the analysis results.

[0939] Specific example

[0940] For example, consider a scenario where a user is imagining the design of a new living room. First, the user selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through a smartphone app. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing and sends it to the device. Finally, the user can view the composite drawing and confirm how their image has been reflected in the design.

[0941] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. In this way, the emotion engine automatically adjusts the color scheme and design, allowing users to have a better experience.

[0942] Example of a prompt

[0943] Use the following prompt:

[0944] Please select the interior elements that the user would like to install in the store. For example, "large display," "wood-grain table," "modern shelf," etc. Based on the user's selection, retrieve and display the appropriate image data. Furthermore, analyze the user's emotions and provide optimal suggestions according to their emotional state.

[0945] This concludes the detailed description of the embodiments of this invention. By using this system, users can proceed with the building design process comfortably and efficiently.

[0946] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0947] Step 1:

[0948] The user selects the building components.

[0949] Users select building components (e.g., floor plan, wallpaper, doors) through a smartphone or tablet application. This selection information is entered via a user interface (UI), and the input information is converted into JSON format.

[0950] Input: User-selected components (e.g., spacious floor plan, white wallpaper, wood-grain door)

[0951] Output: Selection information for components in JSON format (Example: {"layout":"wide", "cross":"white", "door":"wood grain"})

[0952] Step 2:

[0953] The device sends the selection information to the server.

[0954] The device sends the user-selected JSON-formatted information to the server. An HTTP POST request is used for this transmission.

[0955] Input: Selection information for components in JSON format

[0956] Output: HTTP POST request to the server

[0957] Step 3:

[0958] The server searches the database.

[0959] The server parses the received JSON-formatted information and searches the database for image data related to the building's components based on that information. The search results retrieve the corresponding image data.

[0960] Input: Selection information for components in JSON format

[0961] Output: Relevant image data (e.g., image of a spacious floor plan, image of white wallpaper, image of a wood-grain door)

[0962] Step 4:

[0963] The server sends image data to the terminal.

[0964] The server searches for and retrieves image data, compiles it into a list, encodes it in JSON format, and sends it to the terminal.

[0965] Input: Relevant image data

[0966] Output: List of image data encoded in JSON format

[0967] Step 5:

[0968] Display image data received by the device.

[0969] The terminal analyzes the JSON-formatted image data received from the server and displays thumbnail images of each component on the user interface (UI).

[0970] Input: A list of image data encoded in JSON format.

[0971] Output: Thumbnail image displayed to the user

[0972] Step 6:

[0973] The user selects how to apply the changes to the drawing.

[0974] The user selects the desired components from the displayed thumbnail images and confirms their placement on the drawing. This selection information is also compiled again in JSON format and sent from the terminal to the server.

[0975] Input: Thumbnail image selected by the user

[0976] Output: Component placement information summarized in JSON format

[0977] Step 7:

[0978] The server generates a composite image.

[0979] The server generates a composite diagram using image processing techniques (e.g., OpenCV, Pillow, etc.) based on the arrangement information of the components selected by the user.

[0980] Input: Component placement information compiled in JSON format

[0981] Output: Image data of the generated composite image

[0982] Step 8:

[0983] The server sends the composite image to the terminal.

[0984] The server encodes the generated composite diagram in JSON format and sends it to the terminal.

[0985] Input: Image data of the generated composite image

[0986] Output: Composite diagram encoded in JSON format

[0987] Step 9:

[0988] The device displays a composite image.

[0989] The terminal decodes the composite diagram in JSON format received from the server and displays it on the user interface (UI). This allows the user to see how the selected components look in the actual diagram.

[0990] Input: Composite diagram encoded in JSON format

[0991] Output: Composite diagram displayed to the user

[0992] Step 10:

[0993] The emotion engine analyzes the user's emotions.

[0994] The device inputs the user's facial expressions and voice into an emotion engine, which analyzes them in real time. As a result, the user's emotional state is determined.

[0995] Input: User facial expression data, voice data

[0996] Output: User's emotional state (e.g., satisfied, relaxed)

[0997] Step 11:

[0998] The server makes recommendations and adjustments to components based on emotional state.

[0999] The server receives emotional states from the emotion engine and, based on that, recommends appropriate components or adjusts the currently displayed image data.

[1000] Input: User's emotional state

[1001] Output: Image data of the adjusted components, additional recommended components

[1002] The above describes each processing step and its specific operation in the system of the present invention. This system allows users to comfortably and efficiently select and customize building designs.

[1003] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1004] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1005] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1006] [Third Embodiment]

[1007] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1008] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1009] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1010] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1011] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1012] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1013] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1014] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1015] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1016] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1017] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1018] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1019] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[1020] A natural language explanation of the program's processing.

[1021] The user selects a building element.

[1022] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[1023] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[1024] The server searches the database.

[1025] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[1026] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[1027] Acquisition and transmission of image data

[1028] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[1029] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[1030] Displaying images on a device

[1031] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[1032] The user selects how to apply the changes to the drawing.

[1033] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[1034] The terminal sends information about the selected element to the server.

[1035] Generation and display of composite diagrams

[1036] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[1037] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[1038] The terminal decodes the received composite drawing and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[1039] Specific example

[1040] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[1041] The system of this invention allows users to visualize in detail how selected elements fit into the overall building design. This streamlines the design process and improves user satisfaction.

[1042] The following describes the processing flow.

[1043] Step 1:

[1044] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[1045] Step 2:

[1046] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[1047] Step 3:

[1048] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[1049] Step 4:

[1050] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[1051] Step 5:

[1052] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[1053] SQL

[1054] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[1055] SELECT image FROM Images WHERE category='Cross' AND color='White';

[1056] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[1057] Step 6:

[1058] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[1059] Step 7:

[1060] The server converts this list into a data format such as JSON and sends it to the terminal.

[1061] Step 8:

[1062] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[1063] Step 9:

[1064] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[1065] Step 10:

[1066] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[1067] Step 11:

[1068] The device sends the ID and password of the image selected by the user to the server.

[1069] Step 12:

[1070] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[1071] Step 13:

[1072] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[1073] Step 14:

[1074] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[1075] Step 15:

[1076] The terminal decodes the received binary data into image data.

[1077] Step 16:

[1078] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[1079] Through these steps, users can visually see how their selected elements will be combined and create a building design that closely matches their vision.

[1080] (Example 1)

[1081] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1082] Existing building design systems had a problem where it was difficult for users to intuitively visualize how selected building elements would be reflected in actual drawings. Furthermore, there was a lack of systems that could generate and quickly display composite drawings based on user selections. This resulted in an inefficient design process and low user satisfaction.

[1083] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1084] In this invention, the server includes means for the user to select building elements, means for the terminal to transmit the user's selection information to the server, means for the server to generate a search query based on the selected elements, search a database and obtain corresponding image data, means for the server to transmit the obtained image data to the terminal in list format, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the terminal to transmit information about the selected elements to the server, means for the server to generate a composite drawing based on the reflected drawing, and means for the terminal to display the composite drawing to the user. As a result, the elements selected by the user are intuitively visualized, the design process is made more efficient, and user satisfaction is improved.

[1085] A "user" refers to an individual or group that uses the system to select building elements and participate in the design process.

[1086] A "terminal" refers to an electronic device (e.g., computer, smartphone, tablet) that a user uses to access and operate a system.

[1087] A "server" refers to a computer system that receives user selection information and performs processing such as database searches, image data acquisition, processing, and transmission.

[1088] "Building elements" refer to specific components that users can select in the design of a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1089] A "database" refers to a system for storing and managing information and image data related to the elements of a building.

[1090] A "search query" refers to a specific search command generated to retrieve relevant information from a database based on the user's selections.

[1091] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[1092] "List format" refers to a data format that enumerates and stores multiple pieces of data in a structured manner.

[1093] "User interface" refers to the screen display and operating means that allow a user to operate a system and visually confirm its output.

[1094] A "composite drawing" refers to a comprehensive design drawing generated by reflecting the building elements selected by the user.

[1095] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[1096] The user selects building elements.

[1097] The device provides a user interface, such as a web browser or a dedicated app. This interface displays dropdown menus and radio buttons for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). Through these interfaces, the user might, for example, select "floor plan" and then "spacious."

[1098] Send user selection information to the server

[1099] The device sends the user's selected information to the server as a JSON request. The selected information includes the "element type" and "selection options".

[1100] The server searches the database.

[1101] The server parses the received request and generates an appropriate search query. For example, if the element type is "floor plan" and the selection option is "spacious," it generates a query like "SELECT FROM images WHERE type='floor plan' AND option='spacious'." The server then executes this query in a database management system (e.g., MySQL, PostgreSQL, etc.) to retrieve the corresponding image data.

[1102] Acquire and send image data.

[1103] The server compiles the image data retrieved from the database into a list format and sends it to the terminal in JSON format.

[1104] The device displays image data.

[1105] The terminal parses the JSON data received from the server and displays the images on the user interface. Specifically, the images are displayed in a list format as thumbnails. This allows the user to visually confirm the images.

[1106] The user selects how to apply the changes to the drawing.

[1107] The user selects the desired element from the displayed thumbnail image by clicking or tapping, and then specifies the placement of that element. The device then sends this selection and placement information back to the server in JSON format.

[1108] The server generates the composite image.

[1109] The server generates a composite image based on the elements selected by the user. Image processing techniques (e.g., OpenCV, Pillow, etc.) are used in this process. After generating the composite image, the server may encode this image data into binary format.

[1110] Send and display the composite image.

[1111] The server sends the generated composite diagram to the terminal, which decodes it and displays it in the user interface. This allows the user to visually confirm how the selected elements are reflected in the actual design drawing.

[1112] Specific example

[1113] For example, when designing a new living room, the user first selects "spacious" for the "layout," "white" for the "wallpaper," and "wood grain" for the "door" through the UI. The device sends this selection information to the server in JSON format. The server parses the received information, retrieves the corresponding image data from the database, and sends it to the device. The device displays these images as thumbnails, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite drawing, which is then sent to the device. Finally, the user can view the composite drawing and see how their image is reflected in the actual design.

[1114] Example of a prompt

[1115] "When designing a new room, I'd like to make the living room spacious. I also want white wallpaper and wood-grain doors. I'd like to retrieve image data that meets these conditions from a server and generate a composite drawing that incorporates these elements into the floor plan."

[1116] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1117] Step 1:

[1118] The user selects elements of the building.

[1119] Specifically, the user selects a "floor plan" through the device's user interface, for example, using a dropdown menu or radio buttons, and then selects "spacious" from the options.

[1120] Input: User selection of elements (e.g., floor plan = spacious)

[1121] Output: Selected element information (Example: {"Element Type":"Floor Plan","Selection Option":"Spacious"})

[1122] Step 2:

[1123] The device sends the information selected by the user to the server.

[1124] Specifically, the terminal packets the selection information in JSON format and sends it to the server as an HTTP request.

[1125] Input: Selected element information (Example: {"Element Type": "Floor Plan","Selection Option": "Spacious"})

[1126] Output: Sending data to the server via HTTP request

[1127] Step 3:

[1128] The server parses the received request and generates a database search query.

[1129] Specifically, the server parses the JSON and generates a query such as "SELECT FROM images WHERE type='floor plan' AND option='spacious'".

[1130] Input: Element information received from the terminal (e.g., {"Element Type":"Floor Plan", "Selection Option":"Spacious"})

[1131] Output: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[1132] Step 4:

[1133] The server executes a search query on the database management system and retrieves the corresponding image data.

[1134] Specifically, the server executes queries against database management systems such as MySQL and PostgreSQL to retrieve relevant image data.

[1135] Input: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[1136] Output: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1137] Step 5:

[1138] The server sends the acquired image data to the terminal in JSON format.

[1139] Specifically, the server compiles the acquired data into a list format, encodes it into JSON, and sends it to the terminal as an HTTP response.

[1140] Input: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1141] Output: Image data in JSON format (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1142] Step 6:

[1143] The terminal parses the JSON data received from the server and displays the image on the user interface.

[1144] Specifically, the device parses the JSON data and displays the images in a list format as thumbnails.

[1145] Input: Image data in JSON format received from the server (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1146] Output: Thumbnail image displayed on the user interface

[1147] Step 7:

[1148] The user selects the desired element from the displayed thumbnail images and specifies its placement.

[1149] Specifically, users select a thumbnail image by clicking or tapping it, and then determine its placement by dragging and dropping it.

[1150] Input: Displayed thumbnail image and user selection and placement information

[1151] Output: Information on the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[1152] Step 8:

[1153] The terminal then sends the selected elements and their placement information back to the server in JSON format.

[1154] Specifically, the terminal packets the selected elements and their placement information and sends them to the server as an HTTP request.

[1155] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[1156] Output: Sending data to the server via HTTP request

[1157] Step 9:

[1158] The server generates a composite diagram based on the information it receives.

[1159] Specifically, the server uses image processing technology (e.g., OpenCV, Pillow) to generate a composite image that reflects the selected elements in the drawing.

[1160] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[1161] Output: Image data of the generated composite image

[1162] Step 10:

[1163] The server sends the generated composite diagram to the terminal.

[1164] Specifically, the composite diagram is encoded in binary format and sent to the terminal as an HTTP response.

[1165] Input: Image data of the generated composite image

[1166] Output: Binary data of the composite diagram

[1167] Step 11:

[1168] The terminal decodes the composite diagram received from the server and displays it on the user interface.

[1169] Specifically, the terminal decodes the binary data and displays a composite diagram so that the user can verify it.

[1170] Input: Binary data of the composite diagram

[1171] Output: Composite diagram displayed in the user interface

[1172] (Application Example 1)

[1173] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1174] Traditional food delivery applications lack a mechanism for users to select their desired elements and receive personalized delivery options based on those selections. As a result, the process of users selecting each element and visually confirming their desired dishes is cumbersome, leading to decreased user satisfaction.

[1175] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1176] In this invention, the server includes means for the user to select elements of an object, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for the user to reflect the selected elements onto a composite object, means for the server to generate a composite image based on the reflected composite object, means for the terminal to display the composite image, and means for the user to select a desired menu from the displayed elements and select personalized options. This makes it possible for the user to select each element while visually confirming it and to efficiently and easily select personalized delivery options that suit their individual needs.

[1177] An "object" refers to a specific element or item that a user can select and manipulate.

[1178] A "server" is a central processing unit that receives requests from users, searches a database, and provides the appropriate data.

[1179] A "database" is a data storage system in which multiple pieces of information and data are systematically stored and can be easily searched and retrieved.

[1180] "Image data" refers to data of a still image that contains visual information and is intended to be displayed to the user.

[1181] A "terminal" is a device that a user operates and uses to input information (e.g., a smartphone or tablet).

[1182] "User interface" is a general term for the operating screens and input methods that allow users to directly interact with a system.

[1183] A "composite" is a virtual composition or layout diagram generated based on the elements selected by the user.

[1184] A "composite image" is a visual composite image generated based on multiple elements selected by the user.

[1185] "Personalized options" are choices that are customized based on the user's past selection history and individual needs.

[1186] A "menu" is a collection of specific options or items that a user can choose from.

[1187] This invention provides a system that offers a series of processes in which a user selects elements of an object, a server retrieves relevant image data from a database based on that selection, and a terminal displays and manages that data. Specific embodiments are described below.

[1188] Means by which the user selects elements of an object

[1189] Users can select elements of an object using their device. For example, the device's user interface (UI) may display dropdown menus or card-style UI components, which users can use to select elements.

[1190] A means by which the server searches the database and retrieves the corresponding image data.

[1191] The terminal sends information about the elements selected by the user to the server in JSON format. The server parses the received selection information, searches the appropriate database, and retrieves image data related to the selected elements.

[1192] A means of displaying image data acquired by a device to the user.

[1193] Image data retrieved from the server is sent to the terminal in JSON format. The terminal parses the received data and displays the image data so that the user can visually confirm it.

[1194] A means of reflecting user-selected elements onto a composite object.

[1195] The user selects desired elements from image data displayed on their device and reflects them onto a composite object (a virtual composition or layout diagram). The elements selected by the user are then sent from the device to the server.

[1196] A server is a means for generating a composite image based on a composite that has been reflected.

[1197] The server generates a composite image based on the elements selected by the user. Image processing techniques (such as OpenCV or Pillow) are used for generation.

[1198] Means by which a terminal displays a composite image

[1199] The server generates a composite image and sends it to the terminal, which then displays it to the user. This allows the user to visually see how their selected elements will appear in the composite image.

[1200] A means for users to select personalized options

[1201] After the user reviews the composite image, they can select their desired menu from the displayed elements and request personalized options from the server. The server will then provide appropriate personalized options, taking into account the user's past selections and other information.

[1202] Hardware and software to be used

[1203] To implement this system, the following hardware and software will be used:

[1204] UI development: React Native, Flutter

[1205] Data transmission and analysis: REST API, JSON format

[1206] Image data processing: Python, OpenCV, Pillow

[1207] Database: MySQL, MongoDB

[1208] Server: Node.js, Express

[1209] Cloud services: AWS EC2, S3

[1210] Specific example

[1211] For example, if a user selects "steak" as their "main dish," the device sends this information to the server in JSON format. The server parses the selection, retrieves image data corresponding to "steak" from its database, and sends it back to the device. The device displays the image data, allowing the user to select their desired elements and ultimately providing a personalized delivery option.

[1212] Example of a prompt

[1213] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[1214] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1215] Step 1:

[1216] The device displays a user interface. It displays dropdown menus or card-style UI components to make it easier for the user to select elements of an object (e.g., main dish, drink, dessert). The input is a list of selectable elements of the object, and the output is the selected element.

[1217] Step 2:

[1218] The user selects elements of an object. For example, they might select "main dish" and then choose "steak" from the options. The input is the options displayed in the user interface, and the output is the specific element selected by the user.

[1219] Step 3:

[1220] The device sends the user's selection to the server. The information of the selected element is converted to JSON format and sent using a REST API. The input is the element selected by the user, and the output is the JSON data sent to the server.

[1221] Step 4:

[1222] The server searches the database based on the data it receives. It parses the selected information and generates relevant image data as a search query. The input is JSON data containing information about the elements selected by the user, and the output is image data from the database search results.

[1223] Step 5:

[1224] The server collects image data, compiles it into JSON format, and sends it to the terminal. The input is image data retrieved from the database, and the output is the JSON data sent to the terminal.

[1225] Step 6:

[1226] The terminal analyzes image data received from the server. It parses the JSON data and displays the image. The input is the JSON data sent from the server, and the output is the displayed image data.

[1227] Step 7:

[1228] The user selects a desired element from the displayed image and inputs that information into the device. The input is the displayed image data, and the output is the specific image element selected by the user.

[1229] Step 8:

[1230] The device sends the selected element to the server. The selection information is converted back to JSON format and sent using the REST API. The input is the information of the element selected by the user, and the output is the JSON data sent to the server.

[1231] Step 9:

[1232] The server generates a composite image based on the elements it receives. It combines the images of each element to create a single composite image. For example, it uses image processing techniques such as OpenCV or Pillow. The input is information on multiple elements selected by the user, and the output is the generated composite image.

[1233] Step 10:

[1234] The server generates a composite image and sends it to the terminal. The image data is encoded in binary format and sent to the terminal. The input is the generated composite image, and the output is the composite image data sent to the terminal.

[1235] Step 11:

[1236] The terminal decodes the composite image and displays it to the user. This allows the user to visually confirm the composite image. The input is the binary composite image data received from the server, and the output is the displayed composite image.

[1237] Step 12:

[1238] The user reviews the displayed composite image and selects personalized options. The input is the displayed composite image, and the output is the personalized options selected by the user.

[1239] Step 13:

[1240] The device sends personalized option information to the server. The selection information is converted to JSON format and sent using a REST API. The input is the options selected by the user, and the output is the JSON data sent to the server.

[1241] Step 14:

[1242] The server generates the final delivery information based on the personalized options it receives. The input is the information of the options selected by the user, and the output is the generated delivery information.

[1243] Step 15:

[1244] The server sends the final generated delivery information to the terminal, which then displays it. The input is the generated delivery information, and the output is the displayed delivery information.

[1245] Specific examples of operation

[1246] For example, when a user selects "steak" as their "main dish" in the app, that information is sent to the server in JSON format. The server retrieves image data of "steak" from its database based on this information and sends it to the device. The device displays the image data to the user, who selects the type of "steak" they want, and sends the information back to the server. The server generates a composite image based on the selected information and sends it to the device. The user reviews this image and finally selects their personalized delivery option.

[1247] Example of a prompt

[1248] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[1249] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1250] The present invention provides a system in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is enhanced.

[1251] A natural language explanation of the program's processing.

[1252] The user selects a building element.

[1253] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[1254] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[1255] The server searches the database.

[1256] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[1257] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[1258] Acquisition and transmission of image data

[1259] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[1260] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[1261] Displaying images on a device

[1262] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[1263] The user selects how to apply the changes to the drawing.

[1264] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[1265] The terminal sends information about the selected element to the server.

[1266] Generation and display of composite diagrams

[1267] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[1268] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[1269] The terminal decodes the received composite image and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[1270] Emotional engine integration

[1271] The emotion engine determines the user's emotional state based on their choices and responses. For example, it recognizes emotions in real time through analysis of the user's facial expressions and voice.

[1272] The server receives information from the emotion engine and provides a means to suggest building elements based on the user's emotional state. For example, if the user is in a "happy" state, it will recommend brightly colored wallpaper and modernly designed doors.

[1273] Furthermore, the emotion engine automatically adjusts the color scheme and design of the displayed image data according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[1274] Specific example

[1275] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[1276] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. The emotion engine automatically adjusts the color scheme and design to ensure the user has a better experience.

[1277] The system of this invention allows users to visualize in detail how their selected elements fit into the overall building design, and further enhances user satisfaction through an emotional engine. This streamlines the design process and improves user satisfaction.

[1278] The following describes the processing flow.

[1279] Step 1:

[1280] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[1281] Step 2:

[1282] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[1283] Step 3:

[1284] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[1285] Step 4:

[1286] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[1287] Step 5:

[1288] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[1289] SQL

[1290] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[1291] SELECT image FROM Images WHERE category='Cross' AND color='White';

[1292] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[1293] Step 6:

[1294] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[1295] Step 7:

[1296] The server converts this list into a data format such as JSON and sends it to the terminal.

[1297] Step 8:

[1298] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[1299] Step 9:

[1300] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[1301] Step 10:

[1302] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[1303] Step 11:

[1304] The device sends the ID and password of the image selected by the user to the server.

[1305] Step 12:

[1306] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[1307] Step 13:

[1308] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[1309] Step 14:

[1310] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[1311] Step 15:

[1312] The terminal decodes the received binary data into image data.

[1313] Step 16:

[1314] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[1315] Step 17:

[1316] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time. Specifically, it collects data using cameras and microphones and performs facial recognition and voice tone analysis.

[1317] Step 18:

[1318] The device sends data from the emotion engine to the server. This allows the user's emotional state to be transmitted to the server in real time.

[1319] Step 19:

[1320] The server analyzes the user's emotional state based on data from the emotion engine and responds appropriately. For example, if the user is "having fun," the server might suggest brighter colored wallpaper or a door with a modern design.

[1321] Step 20:

[1322] The server sends building elements recommended by the user based on their emotional state to the terminal in JSON format or similar.

[1323] Step 21:

[1324] The device displays recommended elements it has received to the user. For example, it displays appropriate options as "recommended cross-references."

[1325] Step 22:

[1326] The emotion engine adjusts the color scheme and design of the image data displayed according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[1327] Step 23:

[1328] The server adjusts the color scheme and design of the image data based on instructions from the emotion engine and generates the composite image again.

[1329] Step 24:

[1330] The device then displays a composite image, adjusted by the emotion engine, to the user again. This adjustment allows the user to see a design that matches their emotions.

[1331] Through these steps, users can visually see how their selected elements will combine and create building designs that closely match their vision. Furthermore, the emotion engine can enhance user satisfaction. This streamlines the design process and improves user satisfaction.

[1332] (Example 2)

[1333] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1334] Conventional building design systems have a cumbersome process for not only visually confirming user-selected elements but also reflecting them in actual design drawings, resulting in low usability. Furthermore, the lack of suggestions and adjustments tailored to the user's emotional state leads to decreased user satisfaction and inefficient design processes. Therefore, the present invention aims to solve these problems and provide a system that allows users to design buildings intuitively and efficiently.

[1335] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1336] In this invention, the server includes means for the user to select building elements, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for an emotion analysis engine to analyze the user's emotional state based on the user's selections and reactions and make appropriate suggestions and adjust the image data. As a result, the user can proceed with the design while intuitively visualizing the selected elements, and furthermore, suggestions and adjustments according to the emotional state can be made to improve the efficiency of the design process and user satisfaction.

[1337] A "user" refers to a person who operates the system and selects elements of a building.

[1338] A "terminal" refers to a device operated by a user (for example, a PC, smartphone, or tablet), which interacts with the system through a user interface.

[1339] A "server" is a computer device that processes requests from users and performs tasks such as searching databases and generating composite diagrams.

[1340] A "database" is a data storage system used to store and manage building elements and related image data.

[1341] "Building elements" refer to the specific parts and features necessary for building design (for example, floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1342] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[1343] A "composite drawing" refers to a single drawing constructed by combining elements of multiple buildings selected by the user, and is a drawing that is applied to actual building design.

[1344] An "emotion analysis engine" refers to software or algorithms that analyze a user's choices and reactions to determine the user's emotional state.

[1345] A "search query" refers to the instructions or conditions that a server generates to search a database.

[1346] The present invention provides a series of processes in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion analysis engine, the user experience is enhanced.

[1347] Specifically, the system uses the following hardware and software:

[1348] Hardware and software

[1349] Device: A device operated by a user, such as a PC, smartphone, or tablet.

[1350] Server: A computer device used for database searches and the generation of composite diagrams.

[1351] Database: A data storage system for storing and managing building elements and related image data.

[1352] Emotion analysis engine: Software that analyzes user choices and reactions to determine the user's emotional state (e.g., algorithms that perform facial expression analysis or voice analysis).

[1353] Detailed explanation of the program

[1354] The user selects a building element.

[1355] The device launches the user interface and displays UI components (dropdown menus, radio buttons, etc.) for selecting building elements (floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1356] Users make selections using these UI components.

[1357] The server searches the database.

[1358] The terminal organizes the user's selection information and sends it to the server as data in JSON format.

[1359] The server analyzes the received data and generates search queries based on the selected elements.

[1360] The server searches the database and retrieves the relevant image data (e.g., floor plan, wallpaper, etc.).

[1361] Acquisition and display of image data

[1362] The server sends the acquired image data to the terminal as a list-formatted JSON data.

[1363] The terminal decodes the received data and extracts a list of image data to be displayed.

[1364] The device displays each image as a thumbnail, allowing the user to visually confirm its contents.

[1365] The user selects how to apply the changes to the drawing.

[1366] The user selects the desired element from the displayed thumbnail images and places it on the drawing.

[1367] Generation and display of composite diagrams

[1368] The device then sends the information of the selected element back to the server in JSON format.

[1369] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information.

[1370] The server encodes the generated composite diagram into binary format and sends it to the terminal.

[1371] The terminal decodes the received binary data and displays the composite diagram as an image.

[1372] Integration of emotion analysis engine

[1373] The emotion analysis engine analyzes user choices and reactions (facial expressions, voice, etc.) in real time.

[1374] The server receives feedback information from the emotion engine and provides recommended elements (color, design, etc.) according to the user's emotional state.

[1375] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state.

[1376] Specific example

[1377] For example, consider the case of imagining the design of a new living room. The user first selects "spacious" as the "floor plan," "white" as the "wallpaper," and "wood grain" as the "door" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from the database and sends it to the terminal. The terminal displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing, which is then sent to the terminal. Finally, the user can view the composite drawing and see how their image is reflected in the actual design drawing.

[1378] Example of a prompt

[1379] "Design a living room. Choose a 'spacious' layout, 'white' wallpaper, and 'wood-grain' doors. If the user is satisfied with these choices, add elements that contribute to a relaxing atmosphere."

[1380] This invention allows users to intuitively visualize selected elements while proceeding with the design, and improves the efficiency of the design process and user satisfaction through suggestions and adjustments tailored to their emotional state.

[1381] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1382] Step 1:

[1383] The terminal launches a user interface and displays options for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). UI components such as dropdown menus and radio buttons are used. This allows the user to select the desired element from the options. The input for Step 1 is "User interaction," and the output is "Selected building element."

[1384] Step 2:

[1385] The user selects building elements using the displayed UI components. For example, they might select "Spacious" from the floor plan options and "White" from the wallpaper options. The input for Step 2 is the "UI components," and the output is the "elements selected by the user."

[1386] Step 3:

[1387] The terminal organizes the user's selection information and sends it to the server as JSON data. HTTP POST requests are often used for this transmission. The input for Step 3 is "user selection information," and the output is "JSON data sent to the server."

[1388] Step 4:

[1389] The server parses the JSON data received from the terminal to determine which elements have been selected. Next, it generates a search query based on the selected elements and searches the database. This process retrieves the corresponding image data. The input for step 4 is "JSON data," and the output is the "search query" and the "retrieved image data."

[1390] Step 5:

[1391] The server compiles the acquired image data into a list-formatted JSON data and sends it to the terminal. The input for Step 5 is "image data," and the output is "the JSON data sent to the terminal."

[1392] Step 6:

[1393] The terminal decodes the JSON data received from the server and extracts a list of image data to be displayed. These images are displayed in the UI as their respective thumbnail images. The input for Step 6 is "JSON data," and the output is "the displayed thumbnail images."

[1394] Step 7:

[1395] The user selects the desired element from the displayed thumbnail images and decides where to place it on the drawing. The input for Step 7 is the "thumbnail image," and the output is the "drawing placement selected by the user."

[1396] Step 8:

[1397] The terminal then sends the elements and placement information selected by the user back to the server in JSON format. The input for step 8 is "drawing placement information," and the output is "JSON data sent to the server."

[1398] Step 9:

[1399] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information. The input for step 9 is "placement information," and the output is the "generated composite image."

[1400] Step 10:

[1401] The server encodes the generated composite diagram into binary format and sends it to the terminal. The input for step 10 is the "composite diagram," and the output is the "binary data sent to the terminal."

[1402] Step 11:

[1403] The terminal decodes the received binary data and displays the composite drawing as an image. This allows the user to see how the selected elements appear in the actual drawing. The input for step 11 is "binary data," and the output is the "displayed composite drawing."

[1404] Step 12:

[1405] The emotion analysis engine analyzes the user's choices and reactions (e.g., facial expressions and voice) in real time to determine the user's emotional state. The input for step 12 is "the user's choices and reactions," and the output is "the emotional state as a result of the analysis."

[1406] Step 13:

[1407] The server receives feedback information from the emotion analysis engine and presents recommended building elements based on the user's emotional state. The input for step 13 is "emotional state feedback," and the output is "recommended elements."

[1408] Step 14:

[1409] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state. The input for step 14 is the "emotional state," and the output is the "adjusted image data."

[1410] (Application Example 2)

[1411] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1412] In modern building design, users desire to visually confirm, select, and customize building components. However, current systems struggle to provide suggestions and adjust image data while considering the user's emotional state, which can result in decreased user satisfaction. Furthermore, the visualization of how building components are reflected in composite drawings is insufficient, hindering an efficient design process. Therefore, there is a need for improved design efficiency and enhanced user experience.

[1413] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1414] In this invention, the server includes means for the user to select building components, means for the server to search a database based on the selected components and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected components on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for analyzing the user's emotional state using an emotion engine and recommending components or adjusting image data based on the analysis results. As a result, the user can select components while visually confirming them, and further improve satisfaction through suggestions and adjustments by the emotion engine.

[1415] A "user" is an individual or group that uses this system to select building components and participate in the design process.

[1416] "Building components" refer to specific parts or elements inside or outside a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1417] A "server" is a central device that receives selection information from the user, searches a database to retrieve the corresponding image data, and transmits it to the terminal.

[1418] A "database" is a storage device that stores information such as image data related to the components of a building, and allows a server to search for and retrieve that data.

[1419] A "terminal" is an electronic device (e.g., smartphone, tablet, computer) that a user operates to display image data retrieved from a server and to verify composite diagrams.

[1420] "Image data" refers to digital image information used to visually represent the components of a building.

[1421] A "composite drawing" is an overall design drawing of a building generated based on the components selected by the user.

[1422] The "emotion engine" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state in real time, and uses the results to recommend components and adjust image data.

[1423] The system of this invention provides a series of processes in which the user selects building components, the server acquires corresponding image data based on the selection, and the terminal displays and manages it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience is improved.

[1424] Hardware and software

[1425] Hardware: Smartphones, tablets, computers

[1426] Software: Python 3, Pillow library, requests library, sentiment engine, database management system, user interface (UI) framework

[1427] Data processing and data calculation

[1428] When a user selects a component, the server receives that selection information in JSON format, searches the database, and retrieves the corresponding image data. The retrieved data is then sent to the terminal in JSON format.

[1429] The terminal analyzes the received image data and displays it for the user to visually confirm. It reflects the user's selected components on the drawing and sends that information to the server. It also has a function that analyzes the user's facial expressions and voice and recognizes their emotional state in real time using an emotion engine.

[1430] The emotion engine recommends components and adjusts image data based on the analysis results.

[1431] Specific example

[1432] For example, consider a scenario where a user is imagining the design of a new living room. First, the user selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through a smartphone app. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing and sends it to the device. Finally, the user can view the composite drawing and confirm how their image has been reflected in the design.

[1433] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. In this way, the emotion engine automatically adjusts the color scheme and design, allowing users to have a better experience.

[1434] Example of a prompt

[1435] Use the following prompt:

[1436] Please select the interior elements that the user would like to install in the store. For example, "large display," "wood-grain table," "modern shelf," etc. Based on the user's selection, retrieve and display the appropriate image data. Furthermore, analyze the user's emotions and provide optimal suggestions according to their emotional state.

[1437] This concludes the detailed description of the embodiments of this invention. By using this system, users can proceed with the building design process comfortably and efficiently.

[1438] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1439] Step 1:

[1440] The user selects the building components.

[1441] Users select building components (e.g., floor plan, wallpaper, doors) through a smartphone or tablet application. This selection information is entered via a user interface (UI), and the input information is converted into JSON format.

[1442] Input: User-selected components (e.g., spacious floor plan, white wallpaper, wood-grain door)

[1443] Output: Selection information for components in JSON format (Example: {"layout":"wide", "cross":"white", "door":"wood grain"})

[1444] Step 2:

[1445] The device sends the selection information to the server.

[1446] The device sends the user-selected JSON-formatted information to the server. An HTTP POST request is used for this transmission.

[1447] Input: Selection information for components in JSON format

[1448] Output: HTTP POST request to the server

[1449] Step 3:

[1450] The server searches the database.

[1451] The server parses the received JSON-formatted information and searches the database for image data related to the building's components based on that information. The search results retrieve the corresponding image data.

[1452] Input: Selection information for components in JSON format

[1453] Output: Relevant image data (e.g., image of a spacious floor plan, image of white wallpaper, image of a wood-grain door)

[1454] Step 4:

[1455] The server sends image data to the terminal.

[1456] The server searches for and retrieves image data, compiles it into a list, encodes it in JSON format, and sends it to the terminal.

[1457] Input: Relevant image data

[1458] Output: List of image data encoded in JSON format

[1459] Step 5:

[1460] Display image data received by the device.

[1461] The terminal analyzes the JSON-formatted image data received from the server and displays thumbnail images of each component on the user interface (UI).

[1462] Input: A list of image data encoded in JSON format.

[1463] Output: Thumbnail image displayed to the user

[1464] Step 6:

[1465] The user selects how to apply the changes to the drawing.

[1466] The user selects the desired components from the displayed thumbnail images and confirms their placement on the drawing. This selection information is also compiled again in JSON format and sent from the terminal to the server.

[1467] Input: Thumbnail image selected by the user

[1468] Output: Component placement information summarized in JSON format

[1469] Step 7:

[1470] The server generates a composite image.

[1471] The server generates a composite diagram using image processing techniques (e.g., OpenCV, Pillow, etc.) based on the arrangement information of the components selected by the user.

[1472] Input: Component placement information compiled in JSON format

[1473] Output: Image data of the generated composite image

[1474] Step 8:

[1475] The server sends the composite image to the terminal.

[1476] The server encodes the generated composite diagram in JSON format and sends it to the terminal.

[1477] Input: Image data of the generated composite image

[1478] Output: Composite diagram encoded in JSON format

[1479] Step 9:

[1480] The device displays a composite image.

[1481] The terminal decodes the composite diagram in JSON format received from the server and displays it on the user interface (UI). This allows the user to see how the selected components look in the actual diagram.

[1482] Input: Composite diagram encoded in JSON format

[1483] Output: Composite diagram displayed to the user

[1484] Step 10:

[1485] The emotion engine analyzes the user's emotions.

[1486] The device inputs the user's facial expressions and voice into an emotion engine, which analyzes them in real time. As a result, the user's emotional state is determined.

[1487] Input: User facial expression data, voice data

[1488] Output: User's emotional state (e.g., satisfied, relaxed)

[1489] Step 11:

[1490] The server makes recommendations and adjustments to components based on emotional state.

[1491] The server receives emotional states from the emotion engine and, based on that, recommends appropriate components or adjusts the currently displayed image data.

[1492] Input: User's emotional state

[1493] Output: Image data of the adjusted components, additional recommended components

[1494] The above describes each processing step and its specific operation in the system of the present invention. This system allows users to comfortably and efficiently select and customize building designs.

[1495] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1496] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1497] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1498] [Fourth Embodiment]

[1499] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1500] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1501] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1502] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1503] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1504] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1505] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1506] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1507] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1508] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1509] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1510] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1511] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1512] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[1513] A natural language explanation of the program's processing.

[1514] The user selects a building element.

[1515] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[1516] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[1517] The server searches the database.

[1518] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[1519] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[1520] Acquisition and transmission of image data

[1521] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[1522] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[1523] Displaying images on a device

[1524] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[1525] The user selects how to apply the changes to the drawing.

[1526] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[1527] The terminal sends information about the selected element to the server.

[1528] Generation and display of composite diagrams

[1529] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[1530] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[1531] The terminal decodes the received composite drawing and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[1532] Specific example

[1533] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[1534] The system of this invention allows users to visualize in detail how selected elements fit into the overall building design. This streamlines the design process and improves user satisfaction.

[1535] The following describes the processing flow.

[1536] Step 1:

[1537] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[1538] Step 2:

[1539] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[1540] Step 3:

[1541] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[1542] Step 4:

[1543] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[1544] Step 5:

[1545] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[1546] SQL

[1547] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[1548] SELECT image FROM Images WHERE category='Cross' AND color='White';

[1549] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[1550] Step 6:

[1551] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[1552] Step 7:

[1553] The server converts this list into a data format such as JSON and sends it to the terminal.

[1554] Step 8:

[1555] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[1556] Step 9:

[1557] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[1558] Step 10:

[1559] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[1560] Step 11:

[1561] The device sends the ID and password of the image selected by the user to the server.

[1562] Step 12:

[1563] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[1564] Step 13:

[1565] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[1566] Step 14:

[1567] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[1568] Step 15:

[1569] The terminal decodes the received binary data into image data.

[1570] Step 16:

[1571] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[1572] Through these steps, users can visually see how their selected elements will be combined and create a building design that closely matches their vision.

[1573] (Example 1)

[1574] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1575] Existing building design systems had a problem where it was difficult for users to intuitively visualize how selected building elements would be reflected in actual drawings. Furthermore, there was a lack of systems that could generate and quickly display composite drawings based on user selections. This resulted in an inefficient design process and low user satisfaction.

[1576] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1577] In this invention, the server includes means for the user to select building elements, means for the terminal to transmit the user's selection information to the server, means for the server to generate a search query based on the selected elements, search a database and obtain corresponding image data, means for the server to transmit the obtained image data to the terminal in list format, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the terminal to transmit information about the selected elements to the server, means for the server to generate a composite drawing based on the reflected drawing, and means for the terminal to display the composite drawing to the user. As a result, the elements selected by the user are intuitively visualized, the design process is made more efficient, and user satisfaction is improved.

[1578] A "user" refers to an individual or group that uses the system to select building elements and participate in the design process.

[1579] A "terminal" refers to an electronic device (e.g., computer, smartphone, tablet) that a user uses to access and operate a system.

[1580] A "server" refers to a computer system that receives user selection information and performs processing such as database searches, image data acquisition, processing, and transmission.

[1581] "Building elements" refer to specific components that users can select in the design of a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1582] A "database" refers to a system for storing and managing information and image data related to the elements of a building.

[1583] A "search query" refers to a specific search command generated to retrieve relevant information from a database based on the user's selections.

[1584] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[1585] "List format" refers to a data format that enumerates and stores multiple pieces of data in a structured manner.

[1586] "User interface" refers to the screen display and operating means that allow a user to operate a system and visually confirm its output.

[1587] A "composite drawing" refers to a comprehensive design drawing generated by reflecting the building elements selected by the user.

[1588] The present invention provides a system in which a user selects elements of a building, a server acquires corresponding image data based on that selection, and a terminal displays and manages that data.

[1589] The user selects building elements.

[1590] The device provides a user interface, such as a web browser or a dedicated app. This interface displays dropdown menus and radio buttons for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). Through these interfaces, the user might, for example, select "floor plan" and then "spacious."

[1591] Send user selection information to the server

[1592] The device sends the user's selected information to the server as a JSON request. The selected information includes the "element type" and "selection options".

[1593] The server searches the database.

[1594] The server parses the received request and generates an appropriate search query. For example, if the element type is "floor plan" and the selection option is "spacious," it generates a query like "SELECT FROM images WHERE type='floor plan' AND option='spacious'." The server then executes this query in a database management system (e.g., MySQL, PostgreSQL, etc.) to retrieve the corresponding image data.

[1595] Acquire and send image data.

[1596] The server compiles the image data retrieved from the database into a list format and sends it to the terminal in JSON format.

[1597] The device displays image data.

[1598] The terminal parses the JSON data received from the server and displays the images on the user interface. Specifically, the images are displayed in a list format as thumbnails. This allows the user to visually confirm the images.

[1599] The user selects how to apply the changes to the drawing.

[1600] The user selects the desired element from the displayed thumbnail image by clicking or tapping, and then specifies the placement of that element. The device then sends this selection and placement information back to the server in JSON format.

[1601] The server generates the composite image.

[1602] The server generates a composite image based on the elements selected by the user. Image processing techniques (e.g., OpenCV, Pillow, etc.) are used in this process. After generating the composite image, the server may encode this image data into binary format.

[1603] Send and display the composite image.

[1604] The server sends the generated composite diagram to the terminal, which decodes it and displays it in the user interface. This allows the user to visually confirm how the selected elements are reflected in the actual design drawing.

[1605] Specific example

[1606] For example, when designing a new living room, the user first selects "spacious" for the "layout," "white" for the "wallpaper," and "wood grain" for the "door" through the UI. The device sends this selection information to the server in JSON format. The server parses the received information, retrieves the corresponding image data from the database, and sends it to the device. The device displays these images as thumbnails, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite drawing, which is then sent to the device. Finally, the user can view the composite drawing and see how their image is reflected in the actual design.

[1607] Example of a prompt

[1608] "When designing a new room, I'd like to make the living room spacious. I also want white wallpaper and wood-grain doors. I'd like to retrieve image data that meets these conditions from a server and generate a composite drawing that incorporates these elements into the floor plan."

[1609] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1610] Step 1:

[1611] The user selects elements of the building.

[1612] Specifically, the user selects a "floor plan" through the device's user interface, for example, using a dropdown menu or radio buttons, and then selects "spacious" from the options.

[1613] Input: User selection of elements (e.g., floor plan = spacious)

[1614] Output: Selected element information (Example: {"Element Type":"Floor Plan","Selection Option":"Spacious"})

[1615] Step 2:

[1616] The device sends the information selected by the user to the server.

[1617] Specifically, the terminal packets the selection information in JSON format and sends it to the server as an HTTP request.

[1618] Input: Selected element information (Example: {"Element Type": "Floor Plan","Selection Option": "Spacious"})

[1619] Output: Sending data to the server via HTTP request

[1620] Step 3:

[1621] The server parses the received request and generates a database search query.

[1622] Specifically, the server parses the JSON and generates a query such as "SELECT FROM images WHERE type='floor plan' AND option='spacious'".

[1623] Input: Element information received from the terminal (e.g., {"Element Type":"Floor Plan", "Selection Option":"Spacious"})

[1624] Output: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[1625] Step 4:

[1626] The server executes a search query on the database management system and retrieves the corresponding image data.

[1627] Specifically, the server executes queries against database management systems such as MySQL and PostgreSQL to retrieve relevant image data.

[1628] Input: Database search query (Example: "SELECT FROM images WHERE type='floor plan' AND option='spacious'")

[1629] Output: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1630] Step 5:

[1631] The server sends the acquired image data to the terminal in JSON format.

[1632] Specifically, the server compiles the acquired data into a list format, encodes it into JSON, and sends it to the terminal as an HTTP response.

[1633] Input: Acquired image data (Example: [{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1634] Output: Image data in JSON format (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1635] Step 6:

[1636] The terminal parses the JSON data received from the server and displays the image on the user interface.

[1637] Specifically, the device parses the JSON data and displays the images in a list format as thumbnails.

[1638] Input: Image data in JSON format received from the server (Example: {"images":[{"imageID":1,"imageURL":"path / to / image1"}, {"imageID":2,"imageURL":"path / to / image2"}])

[1639] Output: Thumbnail image displayed on the user interface

[1640] Step 7:

[1641] The user selects the desired element from the displayed thumbnail images and specifies its placement.

[1642] Specifically, users select a thumbnail image by clicking or tapping it, and then determine its placement by dragging and dropping it.

[1643] Input: Displayed thumbnail image and user selection and placement information

[1644] Output: Information on the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[1645] Step 8:

[1646] The terminal then sends the selected elements and their placement information back to the server in JSON format.

[1647] Specifically, the terminal packets the selected elements and their placement information and sends them to the server as an HTTP request.

[1648] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[1649] Output: Sending data to the server via HTTP request

[1650] Step 9:

[1651] The server generates a composite diagram based on the information it receives.

[1652] Specifically, the server uses image processing technology (e.g., OpenCV, Pillow) to generate a composite image that reflects the selected elements in the drawing.

[1653] Input: Information about the selected element and its position (e.g., {"imageID":1,"position":{"x":100,"y":200}})

[1654] Output: Image data of the generated composite image

[1655] Step 10:

[1656] The server sends the generated composite diagram to the terminal.

[1657] Specifically, the composite diagram is encoded in binary format and sent to the terminal as an HTTP response.

[1658] Input: Image data of the generated composite image

[1659] Output: Binary data of the composite diagram

[1660] Step 11:

[1661] The terminal decodes the composite diagram received from the server and displays it on the user interface.

[1662] Specifically, the terminal decodes the binary data and displays a composite diagram so that the user can verify it.

[1663] Input: Binary data of the composite diagram

[1664] Output: Composite diagram displayed in the user interface

[1665] (Application Example 1)

[1666] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1667] Traditional food delivery applications lack a mechanism for users to select their desired elements and receive personalized delivery options based on those selections. As a result, the process of users selecting each element and visually confirming their desired dishes is cumbersome, leading to decreased user satisfaction.

[1668] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1669] In this invention, the server includes means for the user to select elements of an object, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for the user to reflect the selected elements onto a composite object, means for the server to generate a composite image based on the reflected composite object, means for the terminal to display the composite image, and means for the user to select a desired menu from the displayed elements and select personalized options. This makes it possible for the user to select each element while visually confirming it and to efficiently and easily select personalized delivery options that suit their individual needs.

[1670] An "object" refers to a specific element or item that a user can select and manipulate.

[1671] A "server" is a central processing unit that receives requests from users, searches a database, and provides the appropriate data.

[1672] A "database" is a data storage system in which multiple pieces of information and data are systematically stored and can be easily searched and retrieved.

[1673] "Image data" refers to data of a still image that contains visual information and is intended to be displayed to the user.

[1674] A "terminal" is a device that a user operates and uses to input information (e.g., a smartphone or tablet).

[1675] "User interface" is a general term for the operating screens and input methods that allow users to directly interact with a system.

[1676] A "composite" is a virtual composition or layout diagram generated based on the elements selected by the user.

[1677] A "composite image" is a visual composite image generated based on multiple elements selected by the user.

[1678] "Personalized options" are choices that are customized based on the user's past selection history and individual needs.

[1679] A "menu" is a collection of specific options or items that a user can choose from.

[1680] This invention provides a system that offers a series of processes in which a user selects elements of an object, a server retrieves relevant image data from a database based on that selection, and a terminal displays and manages that data. Specific embodiments are described below.

[1681] Means by which the user selects elements of an object

[1682] Users can select elements of an object using their device. For example, the device's user interface (UI) may display dropdown menus or card-style UI components, which users can use to select elements.

[1683] A means by which the server searches the database and retrieves the corresponding image data.

[1684] The terminal sends information about the elements selected by the user to the server in JSON format. The server parses the received selection information, searches the appropriate database, and retrieves image data related to the selected elements.

[1685] A means of displaying image data acquired by a device to the user.

[1686] Image data retrieved from the server is sent to the terminal in JSON format. The terminal parses the received data and displays the image data so that the user can visually confirm it.

[1687] A means of reflecting user-selected elements onto a composite object.

[1688] The user selects desired elements from image data displayed on their device and reflects them onto a composite object (a virtual composition or layout diagram). The elements selected by the user are then sent from the device to the server.

[1689] A server is a means for generating a composite image based on a composite that has been reflected.

[1690] The server generates a composite image based on the elements selected by the user. Image processing techniques (such as OpenCV or Pillow) are used for generation.

[1691] Means by which a terminal displays a composite image

[1692] The server generates a composite image and sends it to the terminal, which then displays it to the user. This allows the user to visually see how their selected elements will appear in the composite image.

[1693] A means for users to select personalized options

[1694] After the user reviews the composite image, they can select their desired menu from the displayed elements and request personalized options from the server. The server will then provide appropriate personalized options, taking into account the user's past selections and other information.

[1695] Hardware and software to be used

[1696] To implement this system, the following hardware and software will be used:

[1697] UI development: React Native, Flutter

[1698] Data transmission and analysis: REST API, JSON format

[1699] Image data processing: Python, OpenCV, Pillow

[1700] Database: MySQL, MongoDB

[1701] Server: Node.js, Express

[1702] Cloud services: AWS EC2, S3

[1703] Specific example

[1704] For example, if a user selects "steak" as their "main dish," the device sends this information to the server in JSON format. The server parses the selection, retrieves image data corresponding to "steak" from its database, and sends it back to the device. The device displays the image data, allowing the user to select their desired elements and ultimately providing a personalized delivery option.

[1705] Example of a prompt

[1706] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[1707] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1708] Step 1:

[1709] The device displays a user interface. It displays dropdown menus or card-style UI components to make it easier for the user to select elements of an object (e.g., main dish, drink, dessert). The input is a list of selectable elements of the object, and the output is the selected element.

[1710] Step 2:

[1711] The user selects an element of an object. For example, they might select "main dish" and then choose "steak" from the options. The input is the options displayed in the user interface, and the output is the specific element selected by the user.

[1712] Step 3:

[1713] The device sends the user's selection to the server. The information of the selected element is converted to JSON format and sent using a REST API. The input is the element selected by the user, and the output is the JSON data sent to the server.

[1714] Step 4:

[1715] The server searches the database based on the data it receives. It parses the selected information and generates relevant image data as a search query. The input is JSON data containing information about the elements selected by the user, and the output is image data from the database search results.

[1716] Step 5:

[1717] The server collects image data, compiles it into JSON format, and sends it to the terminal. The input is image data retrieved from the database, and the output is the JSON data sent to the terminal.

[1718] Step 6:

[1719] The terminal analyzes image data received from the server. It parses the JSON data and displays the image. The input is the JSON data sent from the server, and the output is the displayed image data.

[1720] Step 7:

[1721] The user selects a desired element from the displayed image and inputs that information into the device. The input is the displayed image data, and the output is the specific image element selected by the user.

[1722] Step 8:

[1723] The device sends the selected element to the server. The selection information is converted back to JSON format and sent using the REST API. The input is the information of the element selected by the user, and the output is the JSON data sent to the server.

[1724] Step 9:

[1725] The server generates a composite image based on the elements it receives. It combines the images of each element to create a single composite image. For example, it uses image processing techniques such as OpenCV or Pillow. The input is information on multiple elements selected by the user, and the output is the generated composite image.

[1726] Step 10:

[1727] The server generates a composite image and sends it to the terminal. The image data is encoded in binary format and sent to the terminal. The input is the generated composite image, and the output is the composite image data sent to the terminal.

[1728] Step 11:

[1729] The terminal decodes the composite image and displays it to the user. This allows the user to visually confirm the composite image. The input is the binary composite image data received from the server, and the output is the displayed composite image.

[1730] Step 12:

[1731] The user reviews the displayed composite image and selects personalized options. The input is the displayed composite image, and the output is the personalized options selected by the user.

[1732] Step 13:

[1733] The device sends personalized option information to the server. The selection information is converted to JSON format and sent using a REST API. The input is the options selected by the user, and the output is the JSON data sent to the server.

[1734] Step 14:

[1735] The server generates the final delivery information based on the personalized options it receives. The input is the information of the options selected by the user, and the output is the generated delivery information.

[1736] Step 15:

[1737] The server sends the final generated delivery information to the terminal, which then displays it. The input is the generated delivery information, and the output is the displayed delivery information.

[1738] Specific examples of operation

[1739] For example, when a user selects "steak" as their "main dish" in the app, that information is sent to the server in JSON format. The server retrieves image data of "steak" from its database based on this information and sends it to the device. The device displays the image data to the user, who selects the type of "steak" they want, and sends the information back to the server. The server generates a composite image based on the selected information and sends it to the device. The user then reviews it and finally selects their personalized delivery option.

[1740] Example of a prompt

[1741] I want to create a recipe selection application based on building elements. The user selects elements of a meal (e.g., main dish, drink, dessert) and sends this selection information to a server. The server retrieves the corresponding image data from a database and presents it to the user. The user then selects their desired meal, and the application provides personalized delivery options.

[1742] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1743] The present invention provides a series of processes in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages it. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the user experience is enhanced.

[1744] A natural language explanation of the program's processing.

[1745] The user selects a building element.

[1746] The device provides the user with options to select building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.) through a user interface. For example, UI components such as dropdown menus and radio buttons may be displayed.

[1747] Users make selections using these UI components. For example, a user might select "floor plan" and then "spacious".

[1748] The server searches the database.

[1749] The terminal sends the user's selection information to the server. This selection information is sent, for example, in JSON format.

[1750] The server analyzes the received information and searches the appropriate database based on that information. A search query is generated to retrieve image data related to the building's elements.

[1751] Acquisition and transmission of image data

[1752] The server retrieves the relevant image data from the database. For example, it retrieves image data of a "spacious floor plan."

[1753] The server compiles the acquired image data into a list format and sends it to the terminal. The data is sent in JSON format, for example.

[1754] Displaying images on a device

[1755] The terminal analyzes the data received from the server and displays it so that the user can visually confirm it. For example, images of floor plans, wallpaper, doors, etc., are displayed as thumbnails.

[1756] The user selects how to apply the changes to the drawing.

[1757] The user selects the desired element from the displayed thumbnail images and decides on its placement on the drawing.

[1758] The terminal sends information about the selected element to the server.

[1759] Generation and display of composite diagrams

[1760] The server generates a composite image based on the elements selected by the user. This uses image processing techniques (e.g., OpenCV, Pillow, etc.).

[1761] The server sends the composite image to the terminal, and in doing so, may encode the image data into binary format.

[1762] The terminal decodes the received composite drawing and displays it to the user. This allows the user to see how the selected elements will look in the actual drawing.

[1763] Emotional engine integration

[1764] The emotion engine determines the user's emotional state based on their choices and responses. For example, it recognizes emotions in real time through analysis of the user's facial expressions and voice.

[1765] The server receives information from the emotion engine and provides a means to suggest building elements based on the user's emotional state. For example, if the user is in a "happy" state, it will recommend brightly colored wallpaper and modernly designed doors.

[1766] Furthermore, the emotion engine automatically adjusts the color scheme and design of the displayed image data according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[1767] Specific example

[1768] For example, consider a scenario where a user is imagining the design of a new living room. The user first selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and incorporates them into the drawing, the server generates a composite image, which is then sent to the device. Finally, the user can view the composite image and see how their image is reflected in the actual design drawing.

[1769] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. The emotion engine automatically adjusts the color scheme and design to ensure the user has a better experience.

[1770] The system of this invention allows users to visualize in detail how their selected elements fit into the overall building design, and further enhances user satisfaction through an emotional engine. This streamlines the design process and improves user satisfaction.

[1771] The following describes the processing flow.

[1772] Step 1:

[1773] The device displays a user interface, allowing the user to select options (such as floor plans, wallpaper, fixtures, doors, and window frames). For example, it might provide dropdown menus or radio buttons.

[1774] Step 2:

[1775] The user selects specific elements from the provided UI, such as "floor plan" → "spacious," "wallpaper" → "white," and "doors" → "wood grain."

[1776] Step 3:

[1777] The device sends the user's selections to the server. These selections are sent to the server, for example, in JSON format.

[1778] Step 4:

[1779] The server analyzes the selected items received (for example, "spacious floor plan," "white wallpaper," "wood-grain door"). Based on the analysis results, it searches the database for corresponding image data.

[1780] Step 5:

[1781] The server generates a search query and uses it to search the database. For example, it performs the following search using an SQL query:

[1782] SQL

[1783] SELECT image FROM Images WHERE category='floor plan' AND type='spacious';

[1784] SELECT image FROM Images WHERE category='Cross' AND color='White';

[1785] SELECT image FROM Images WHERE category='door' AND style='wood grain';

[1786] Step 6:

[1787] The server compiles the image data retrieved from the database into a list. The list contains the corresponding elements for each selected element.

[1788] Step 7:

[1789] The server converts this list into a data format such as JSON and sends it to the terminal.

[1790] Step 8:

[1791] The terminal parses the received JSON data and displays it on the screen as a list for each category (floor plan, wallpaper, doors, etc.).

[1792] Step 9:

[1793] The device displays images in thumbnail format so that users can make intuitive selections. For example, each image is displayed in a clickable format.

[1794] Step 10:

[1795] The user selects their preferred image from the displayed thumbnail images. For example, the user clicks on "wood-grain door".

[1796] Step 11:

[1797] The device sends the ID and password of the image selected by the user to the server.

[1798] Step 12:

[1799] Based on the image ID and path received by the server, the high-resolution image of each element is retrieved again from the database.

[1800] Step 13:

[1801] The server uses image processing libraries (such as OpenCV or Pillow) to combine the selected elements. For example, it can generate a composite image by combining a "spacious floor plan," "white wallpaper," and "wood-grain door."

[1802] Step 14:

[1803] The server encodes the image data into binary format and sends it to the terminal in order to send the synthesized drawing image to the terminal.

[1804] Step 15:

[1805] The terminal decodes the received binary data into image data.

[1806] Step 16:

[1807] The terminal displays the composite drawing image to the user. For example, it can be displayed using the HTML tag or the Canvas element.

[1808] Step 17:

[1809] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time. Specifically, it collects data using cameras and microphones and performs facial recognition and voice tone analysis.

[1810] Step 18:

[1811] The device sends data from the emotion engine to the server. This allows the user's emotional state to be transmitted to the server in real time.

[1812] Step 19:

[1813] The server analyzes the user's emotional state based on data from the emotion engine and responds appropriately. For example, if the user is "having fun," the server might suggest brighter colored wallpaper or a door with a modern design.

[1814] Step 20:

[1815] The server sends building elements recommended by the user based on their emotional state to the terminal in JSON format or similar.

[1816] Step 21:

[1817] The device displays recommended elements it has received to the user. For example, it displays appropriate options as "recommended cross-references."

[1818] Step 22:

[1819] The emotion engine adjusts the color scheme and design of the image data displayed according to the user's emotional state. For example, if the user is feeling "down," it will select relaxing colors and designs.

[1820] Step 23:

[1821] The server adjusts the color scheme and design of the image data based on instructions from the emotion engine and generates the composite image again.

[1822] Step 24:

[1823] The device then displays a composite image, adjusted by the emotion engine, to the user again. This adjustment allows the user to see a design that matches their emotions.

[1824] Through these steps, users can visually see how their selected elements will combine and create building designs that closely match their vision. Furthermore, the emotion engine can enhance user satisfaction. This streamlines the design process and improves user satisfaction.

[1825] (Example 2)

[1826] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1827] Conventional building design systems have a cumbersome process for not only visually confirming user-selected elements but also reflecting them in actual design drawings, resulting in low usability. Furthermore, the lack of suggestions and adjustments tailored to the user's emotional state leads to decreased user satisfaction and inefficient design processes. Therefore, the present invention aims to solve these problems and provide a system that allows users to design buildings intuitively and efficiently.

[1828] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1829] In this invention, the server includes means for the user to select building elements, means for the server to search a database based on the selected elements and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected elements on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for an emotion analysis engine to analyze the user's emotional state based on the user's selections and reactions and make appropriate suggestions and adjust the image data. As a result, the user can proceed with the design while intuitively visualizing the selected elements, and furthermore, suggestions and adjustments according to the emotional state can be made to improve the efficiency of the design process and user satisfaction.

[1830] A "user" refers to a person who operates the system and selects elements of a building.

[1831] A "terminal" refers to a device operated by a user (for example, a PC, smartphone, or tablet), which interacts with the system through a user interface.

[1832] A "server" is a computer device that processes requests from users and performs tasks such as searching databases and generating composite diagrams.

[1833] A "database" is a data storage system used to store and manage building elements and related image data.

[1834] "Building elements" refer to the specific parts and features necessary for building design (for example, floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1835] "Image data" refers to digital image files that contain visual information related to the elements of a building.

[1836] A "composite drawing" refers to a single drawing constructed by combining elements of multiple buildings selected by the user, and is a drawing that is applied to actual building design.

[1837] An "emotion analysis engine" refers to software or algorithms that analyze a user's choices and reactions to determine the user's emotional state.

[1838] A "search query" refers to the instructions or conditions that a server generates to search a database.

[1839] The present invention provides a series of processes in which a user selects building elements, a server acquires corresponding image data based on the selection, and a terminal displays and manages the data. Furthermore, by combining this with an emotion analysis engine, the user experience is enhanced.

[1840] Specifically, the system uses the following hardware and software:

[1841] Hardware and software

[1842] Device: A device operated by a user, such as a PC, smartphone, or tablet.

[1843] Server: A computer device used for database searches and the generation of composite diagrams.

[1844] Database: A data storage system for storing and managing building elements and related image data.

[1845] Emotion analysis engine: Software that analyzes user choices and reactions to determine the user's emotional state (e.g., algorithms that perform facial expression analysis or voice analysis).

[1846] Detailed explanation of the program

[1847] The user selects a building element.

[1848] The device launches the user interface and displays UI components (dropdown menus, radio buttons, etc.) for selecting building elements (floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1849] Users make selections using these UI components.

[1850] The server searches the database.

[1851] The terminal organizes the user's selection information and sends it to the server as data in JSON format.

[1852] The server analyzes the received data and generates search queries based on the selected elements.

[1853] The server searches the database and retrieves the relevant image data (e.g., floor plan, wallpaper, etc.).

[1854] Acquisition and display of image data

[1855] The server sends the acquired image data to the terminal as a list-formatted JSON data.

[1856] The terminal decodes the received data and extracts a list of image data to be displayed.

[1857] The device displays each image as a thumbnail, allowing the user to visually confirm its contents.

[1858] The user selects how to apply the changes to the drawing.

[1859] The user selects the desired element from the displayed thumbnail images and places it on the drawing.

[1860] Generation and display of composite diagrams

[1861] The device then sends the information of the selected element back to the server in JSON format.

[1862] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information.

[1863] The server encodes the generated composite diagram into binary format and sends it to the terminal.

[1864] The terminal decodes the received binary data and displays the composite diagram as an image.

[1865] Integration of emotion analysis engine

[1866] The emotion analysis engine analyzes user choices and reactions (facial expressions, voice, etc.) in real time.

[1867] The server receives feedback information from the emotion engine and provides recommended elements (color, design, etc.) according to the user's emotional state.

[1868] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state.

[1869] Specific example

[1870] For example, consider the case of imagining the design of a new living room. The user first selects "spacious" as the "floor plan," "white" as the "wallpaper," and "wood grain" as the "door" through the UI. This selection information is sent to the server, which retrieves the corresponding image data from the database and sends it to the terminal. The terminal displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing, which is then sent to the terminal. Finally, the user can view the composite drawing and see how their image is reflected in the actual design drawing.

[1871] Example of a prompt

[1872] "Design a living room. Choose a 'spacious' layout, 'white' wallpaper, and 'wood-grain' doors. If the user is satisfied with these choices, add elements that contribute to a relaxing atmosphere."

[1873] This invention allows users to intuitively visualize selected elements while proceeding with the design, and improves the efficiency of the design process and user satisfaction through suggestions and adjustments tailored to their emotional state.

[1874] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1875] Step 1:

[1876] The terminal launches a user interface and displays options for selecting building elements (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.). UI components such as dropdown menus and radio buttons are used. This allows the user to select the desired element from the options. The input for Step 1 is "User interaction," and the output is "Selected building element."

[1877] Step 2:

[1878] The user selects building elements using the displayed UI components. For example, they might select "Spacious" from the floor plan options and "White" from the wallpaper options. The input for Step 2 is the "UI components," and the output is the "elements selected by the user."

[1879] Step 3:

[1880] The terminal organizes the user's selection information and sends it to the server as JSON data. HTTP POST requests are often used for this transmission. The input for Step 3 is "user selection information," and the output is "JSON data sent to the server."

[1881] Step 4:

[1882] The server parses the JSON data received from the terminal to determine which elements have been selected. Next, it generates a search query based on the selected elements and searches the database. This process retrieves the corresponding image data. The input for step 4 is "JSON data," and the output is the "search query" and the "retrieved image data."

[1883] Step 5:

[1884] The server compiles the acquired image data into a list-formatted JSON data and sends it to the terminal. The input for Step 5 is "image data," and the output is "the JSON data sent to the terminal."

[1885] Step 6:

[1886] The terminal decodes the JSON data received from the server and extracts a list of image data to be displayed. These images are displayed in the UI as their respective thumbnail images. The input for Step 6 is "JSON data," and the output is "the displayed thumbnail images."

[1887] Step 7:

[1888] The user selects the desired element from the displayed thumbnail images and decides where to place it on the drawing. The input for Step 7 is the "thumbnail image," and the output is the "drawing placement selected by the user."

[1889] Step 8:

[1890] The terminal then sends the elements and placement information selected by the user back to the server in JSON format. The input for step 8 is "drawing placement information," and the output is "JSON data sent to the server."

[1891] Step 9:

[1892] The server generates a composite image using image processing techniques (e.g., OpenCV or Pillow) based on the received information. The input for step 9 is "placement information," and the output is the "generated composite image."

[1893] Step 10:

[1894] The server encodes the generated composite diagram into binary format and sends it to the terminal. The input for step 10 is the "composite diagram," and the output is the "binary data sent to the terminal."

[1895] Step 11:

[1896] The terminal decodes the received binary data and displays the composite drawing as an image. This allows the user to see how the selected elements appear in the actual drawing. The input for step 11 is "binary data," and the output is the "displayed composite drawing."

[1897] Step 12:

[1898] The emotion analysis engine analyzes the user's choices and reactions (e.g., facial expressions and voice) in real time to determine the user's emotional state. The input for step 12 is "the user's choices and reactions," and the output is "the emotional state as a result of the analysis."

[1899] Step 13:

[1900] The server receives feedback information from the emotion analysis engine and presents recommended building elements based on the user's emotional state. The input for step 13 is "emotional state feedback," and the output is "recommended elements."

[1901] Step 14:

[1902] The emotion analysis engine automatically adjusts the color scheme and design of the displayed image data based on the user's emotional state. The input for step 14 is the "emotional state," and the output is the "adjusted image data."

[1903] (Application Example 2)

[1904] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1905] In modern building design, users desire to visually confirm, select, and customize building components. However, current systems struggle to provide suggestions and adjust image data while considering the user's emotional state, which can result in decreased user satisfaction. Furthermore, the visualization of how building components are reflected in composite drawings is insufficient, hindering an efficient design process. Therefore, there is a need for improved design efficiency and enhanced user experience.

[1906] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1907] In this invention, the server includes means for the user to select building components, means for the server to search a database based on the selected components and obtain corresponding image data, means for the terminal to display the obtained image data to the user, means for reflecting the user's selected components on a drawing, means for the server to generate a composite drawing based on the reflected drawing, means for the terminal to display the composite drawing, and means for analyzing the user's emotional state using an emotion engine and recommending components or adjusting image data based on the analysis results. As a result, the user can select components while visually confirming them, and further improve satisfaction through suggestions and adjustments by the emotion engine.

[1908] A "user" is an individual or group that uses this system to select building components and participate in the design process.

[1909] "Building components" refer to specific parts or elements inside or outside a building (e.g., floor plan, wallpaper, fixtures, doors, window frames, etc.).

[1910] A "server" is a central device that receives selection information from the user, searches a database to retrieve the corresponding image data, and transmits it to the terminal.

[1911] A "database" is a storage device that stores information such as image data related to the components of a building, and allows a server to search for and retrieve that data.

[1912] A "terminal" is an electronic device (e.g., smartphone, tablet, computer) that a user operates to display image data retrieved from a server and to verify composite diagrams.

[1913] "Image data" refers to digital image information used to visually represent the components of a building.

[1914] A "composite drawing" is an overall design drawing of a building generated based on the components selected by the user.

[1915] The "emotion engine" is a function that analyzes the user's facial expressions and voice, recognizes the user's emotional state in real time, and uses the results to recommend components and adjust image data.

[1916] The system of this invention provides a series of processes in which the user selects building components, the server acquires corresponding image data based on the selection, and the terminal displays and manages it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience is improved.

[1917] Hardware and software

[1918] Hardware: Smartphones, tablets, computers

[1919] Software: Python 3, Pillow library, requests library, sentiment engine, database management system, user interface (UI) framework

[1920] Data processing and data calculation

[1921] When a user selects a component, the server receives that selection information in JSON format, searches the database, and retrieves the corresponding image data. The retrieved data is then sent to the terminal in JSON format.

[1922] The terminal analyzes the received image data and displays it for the user to visually confirm. It reflects the user's selected components on the drawing and sends that information to the server. It also has a function that analyzes the user's facial expressions and voice and recognizes their emotional state in real time using an emotion engine.

[1923] The emotion engine recommends components and adjusts image data based on the analysis results.

[1924] Specific example

[1925] For example, consider a scenario where a user is imagining the design of a new living room. First, the user selects "spacious" for the "floor plan," "white" for the "wallpaper," and "wood grain" for the "doors" through a smartphone app. This selection information is sent to the server, which retrieves the corresponding image data from its database and sends it to the device. The device displays these images, allowing the user to visually confirm them. Once the user selects the desired elements and reflects them in the drawing, the server generates a composite drawing and sends it to the device. Finally, the user can view the composite drawing and confirm how their image has been reflected in the design.

[1926] Furthermore, while the user is making choices, the emotion engine analyzes the user's emotional state in real time and provides appropriate recommendations. For example, if the user is "satisfied," the server will suggest elements that help them relax further. In this way, the emotion engine automatically adjusts the color scheme and design, allowing users to have a better experience.

[1927] Example of a prompt

[1928] Use the following prompt:

[1929] Please select the interior elements that the user would like to install in the store. For example, "large display," "wood-grain table," "modern shelf," etc. Based on the user's selection, retrieve and display the appropriate image data. Furthermore, analyze the user's emotions and provide optimal suggestions according to their emotional state.

[1930] This concludes the detailed description of the embodiments of this invention. By using this system, users can proceed with the building design process comfortably and efficiently.

[1931] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1932] Step 1:

[1933] The user selects the building components.

[1934] Users select building components (e.g., floor plan, wallpaper, doors) through a smartphone or tablet application. This selection information is entered via a user interface (UI), and the input information is converted into JSON format.

[1935] Input: User-selected components (e.g., spacious floor plan, white wallpaper, wood-grain door)

[1936] Output: Selection information for components in JSON format (Example: {"layout":"wide", "cross":"white", "door":"wood grain"})

[1937] Step 2:

[1938] The device sends the selection information to the server.

[1939] The device sends the user-selected JSON-formatted information to the server. An HTTP POST request is used for this transmission.

[1940] Input: Selection information for components in JSON format

[1941] Output: HTTP POST request to the server

[1942] Step 3:

[1943] The server searches the database.

[1944] The server parses the received JSON-formatted information and searches the database for image data related to the building's components based on that information. The search results retrieve the corresponding image data.

[1945] Input: Selection information for components in JSON format

[1946] Output: Relevant image data (e.g., image of a spacious floor plan, image of white wallpaper, image of a wood-grain door)

[1947] Step 4:

[1948] The server sends image data to the terminal.

[1949] The server searches for and retrieves image data, compiles it into a list, encodes it in JSON format, and sends it to the terminal.

[1950] Input: Relevant image data

[1951] Output: List of image data encoded in JSON format

[1952] Step 5:

[1953] Display image data received by the device.

[1954] The terminal analyzes the JSON-formatted image data received from the server and displays thumbnail images of each component on the user interface (UI).

[1955] Input: A list of image data encoded in JSON format.

[1956] Output: Thumbnail image displayed to the user

[1957] Step 6:

[1958] The user selects how to apply the changes to the drawing.

[1959] The user selects the desired components from the displayed thumbnail images and confirms their placement on the drawing. This selection information is also compiled again in JSON format and sent from the terminal to the server.

[1960] Input: Thumbnail image selected by the user

[1961] Output: Component placement information summarized in JSON format

[1962] Step 7:

[1963] The server generates a composite image.

[1964] The server generates a composite diagram using image processing techniques (e.g., OpenCV, Pillow, etc.) based on the arrangement information of the components selected by the user.

[1965] Input: Component placement information compiled in JSON format

[1966] Output: Image data of the generated composite image

[1967] Step 8:

[1968] The server sends the composite image to the terminal.

[1969] The server encodes the generated composite diagram in JSON format and sends it to the terminal.

[1970] Input: Image data of the generated composite image

[1971] Output: Composite diagram encoded in JSON format

[1972] Step 9:

[1973] The device displays a composite image.

[1974] The terminal decodes the composite diagram in JSON format received from the server and displays it on the user interface (UI). This allows the user to see how the selected components look in the actual diagram.

[1975] Input: Composite diagram encoded in JSON format

[1976] Output: Composite diagram displayed to the user

[1977] Step 10:

[1978] The emotion engine analyzes the user's emotions.

[1979] The device inputs the user's facial expressions and voice into an emotion engine, which analyzes them in real time. As a result, the user's emotional state is determined.

[1980] Input: User facial expression data, voice data

[1981] Output: User's emotional state (e.g., satisfied, relaxed)

[1982] Step 11:

[1983] The server makes recommendations and adjustments to components based on emotional state.

[1984] The server receives emotional states from the emotion engine and, based on that, recommends appropriate components or adjusts the currently displayed image data.

[1985] Input: User's emotional state

[1986] Output: Image data of the adjusted components, additional recommended components

[1987] The above describes each processing step and its specific operation in the system of the present invention. This system allows users to comfortably and efficiently select and customize building designs.

[1988] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1989] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1990] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1991] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1992] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1993] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1994] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1995] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1996] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1997] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1998] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1999] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2000] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2001] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2002] Furthermore, it is not necessary to store the entirety of the specific proces...

Claims

1. A means for the user to select building elements, A means for the server to search the database based on the selected element and retrieve the corresponding image data, A means of displaying image data acquired by the terminal to the user, A means of reflecting user-selected elements onto the drawing, A means for generating a composite drawing based on a drawing reflected by a server, A means by which the terminal displays a composite diagram, A system that includes this.

2. The system according to claim 1, wherein the server has means for generating a search query based on building elements selected by the user and obtaining related image data from a database.

3. The system according to claim 1, wherein the terminal has means for transmitting information to a server that the user has selected building elements through a user interface.

Images