Reform image generation system, method, program and device.

Abstract

It is possible to obtain a highly accurate image of the post-renovation state in a shorter time. [Solution] According to one embodiment, the renovation image generation system includes a memory unit that stores multiple catalog information that associates catalog images representing interiors with category information that represents the categories of the catalog images, an image acquisition unit that acquires room images, a segmentation unit that identifies each interior area drawn in the room image and assigns interior category information to each area, a category selection unit that accepts category information selected by the user from a terminal device, an interior reception unit that accepts catalog image selection by the user from the terminal device, and an image generation unit that generates a renovation image in which interiors drawn in the room image that have been assigned category information selected by the user are replaced with interiors represented by the catalog images accepted by the interior reception unit, and displays the image on the terminal device.

Description

[Technical Field]

[0001] The present invention relates to a reform image generation system, method, program and device. [Background technology]

[0002] Conventionally, there is known a technique for simulating the post-renovation situation by replacing renovation objects such as walls, floors, curtains, or furniture with other objects from an indoor image captured by a camera, etc. As a system related to this technique, for example, a system is known in which an image range of an object such as a wall, floor, curtain, or furniture included in an original image is specified using a reference figure, and then the size of a sample image is reduced based on dimensional information assigned to the image range, and the sample image is pasted into a mask range of the object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-352259 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned conventional systems have a problem in that they lack real-time capabilities, since many processes require manual work by the user, such as adding dimensional information to the image range and scaling the size of the sample image.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to make it possible to obtain a highly accurate image of the post-renovation state in a shorter time. [Means for solving the problem]

[0006] According to one embodiment, the renovation image generation system includes a memory unit that stores multiple catalog information that associates catalog images representing interiors that decorate an indoor space with category information that represents the categories of the catalog images; an image acquisition unit that acquires room images; a segmentation unit that identifies each interior area drawn in the room image and assigns category information of the interior to each of the areas; a category selection unit that transmits the category information assigned by the segmentation unit to a terminal device and accepts a user's selection of the category information from the terminal device; an interior reception unit that transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and accepts a user's selection of the catalog image from the terminal device; and an image generation unit that generates a renovation image in which the interiors drawn in the room image that have been assigned the category information selected by the user are replaced with the interior represented by the catalog image accepted by the interior reception unit, and displays the image on the terminal device. [Effects of the Invention]

[0007] According to one embodiment, a highly accurate image of the post-renovation state can be obtained in a shorter time. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a reform image generation system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an image generating apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a terminal device. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a language model device. [Figure 6] 1 is a flowchart showing the first half of a first example of a reformed image generating method. [Figure 7]10 is a flowchart showing the second half of a first example of a reformed image generating method. [Figure 8] FIG. 10 is a diagram schematically illustrating an example of an upload screen for a room image before uploading. [Figure 9] FIG. 10 is a diagram schematically illustrating an example of a category reception screen for category information. [Figure 10] FIG. 10 is a diagram schematically illustrating an example of a selection screen when "Chair" is selected as category information. [Figure 11] FIG. 10 is a diagram schematically illustrating an example of a display screen of a generated reform image. [Figure 12] 10 is a flowchart showing the first half of a second example of a reformed image generating method. [Figure 13] 10 is a flowchart showing the second half of a second example of a reformed image generating method. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the embodiment and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] <System configuration> First, an overview of the renovation image generation system 1000 according to this embodiment will be described. The renovation image generation system 1000 is an information processing system that generates a renovation image showing one or more post-renovation room images from a room image. The renovation image generation system 1000 generates the renovation image based on the room image, various information included in catalog information, and instruction information. This information will be described later.

[0011] Users of the renovation image generation system 1000 are, for example, businesses that make renovation proposals or professional businesses such as interior coordinators. By generating renovation images using the renovation image generation system 1000, businesses that make renovation proposals or professional businesses such as interior coordinators can quickly provide customers with renovation and interior design proposals and hold meetings while showing the renovation image 125. In this case, because the renovation image 125 is generated based on the room image 121, the customer can intuitively grasp what the space will look like after renovation. Furthermore, because the renovation image 125 is generated using an interior image included in the catalog information 122, it is also possible to make proposals by correlating the proposal content with the actual catalog information 122.

[0012] A user of the renovation image generation system 1000 may be a corporation such as a real estate or construction-related business. Corporations such as real estate companies or construction-related businesses can use the renovation image generation system 1000 for the purpose of sales promotion or design support. For example, by uploading an image of a vacant room and generating a renovation image after renovation or after furniture arrangement, it is possible to introduce a property to customers with a highly visual effect.

[0013] A user of the renovation image generation system 1000 may be a resident considering renovating or redecorating their home. The resident can use the renovation image generation system 1000 to visually confirm the image of the renovated home after replacing the interior, such as furniture and wallpaper, with a design of their choice, using room images taken of their own living space. In this case, the user does not need to have specialized knowledge and can easily generate the renovation image 125 via the terminal device 2.

[0014] FIG. 1 is a diagram showing an example of the configuration of a renovation image generation system 1000. As shown in FIG. 1, the renovation image generation system 1000 includes an image generation device 1, a terminal device 2, and a language model device 3, which are communicably connected to each other via a network N. The network N is, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination of these. In the example of FIG. 1, the renovation image generation system 1000 includes one each of the image generation device 1, the terminal device 2, and the language model device 3, but may include multiple of each.

[0015] The image generation device 1 is an information processing device that generates, based on an acquired room image 121, a remodeled image 125 that shows a room image after the room image 121 has been remodeled. The image generation device 1 transmits information used to generate the remodeled image 125 to the language model device 3 via an API (Application Programming Interface). The information used to generate the remodeled image 125 includes, but is not limited to, the room image 121, various information included in catalog information, and instruction information. For example, the image generation device 1 acquires the remodeled image 125 from the language model device 3 via the API. The image generation device 1 is, for example, but is not limited to, a personal computer (PC), a smartphone, a tablet terminal, a server device, or a microcomputer. In the example of FIG. 1, the image generation device 1 is a single information processing device, but may also be realized as a system consisting of multiple information processing devices connected via a network N.

[0016] The terminal device 2 is an information processing device used by a user to transmit a room image 221 or to generate a renovation image 125. The terminal device 2 transmits the selected room image 221 and information input by the user to the image generation device 1. The terminal device 2 is, for example, but is not limited to, a PC, a smartphone, a tablet terminal, a headset, or a head-mounted display.

[0017] The language model device 3 is an information processing device that generates a response to acquired information (e.g., a prompt) and transmits the generated response to an external device. The language model device 3 receives, for example, a prompt expressed in natural language from an external device. For example, the language model device 3 receives an instruction (prompt) from the image generation device 1 and generates information in response to the instruction. In the renovation image system 1000, the language model device 3 inputs various pieces of information acquired from the image generation device 1 via an API into a machine learning model. The various pieces of information include, for example, a room image 121, various pieces of information included in catalog information 122, and instruction information 124. The language model device 3 transmits a response output from the machine learning model to the image generation device 1 via the API. The response output from the renovation image generation system 1000 is, for example, a renovation image 125. The language model device 3 is, for example, a PC, a server device, or a microcomputer, but is not limited to these. In the example of FIG. 1, the language model device 3 is a single information processing device, but may also be realized as a system consisting of multiple information processing devices connected via a network N.

[0018] <Hardware configuration of information processing device 100> Next, a description will be given of the hardware configuration of the information processing device 100. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device 100. As shown in Fig. 2, the information processing device 100 includes a processor 101, a memory 102, a storage 103, a communication I / F 104, an input device 105, an output device 106, and a drive device 107, which are connected to each other via a bus B.

[0019] The processor 101 controls each component of the information processing device 100 and realizes the functions of the information processing device 100 by loading various programs including an OS (Operating System) stored in the storage 103 into the memory 102 and executing the programs. The processor 101 is, for example, but not limited to, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a DSP (Digital Signal Processor).

[0020] The memory 102 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, but not limited to, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0021] The storage 103 stores various programs including an OS and data. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or storage class memories (SCM), but is not limited to these.

[0022] The communication I / F 104 is an interface for connecting the information processing device 100 to an external device via the network N and controlling communication. The communication I / F 104 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited to these.

[0023] The input device 105 is a device for inputting information to the information processing device 100. The input device 105 is, for example, a mouse, a keyboard, a microphone, a scanner, a photographing device (camera), various sensors, or an operation button, but is not limited to these.

[0024] The output device 106 is a device for outputting information from the information processing device 100. The output device 106 is, for example, a display device, a projector, a printer, a speaker, or a vibrator, but is not limited to these. Note that in this embodiment, the input device 105 and the output device 106 may be configured as a touch panel integrated into the display device.

[0025] The drive device 107 is a device that reads and writes data from and to the recording medium 108. The drive device 107 is, for example, but not limited to, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader. The recording medium 108 is, for example, but not limited to, a CD (Compact Disc), a DVD (Digital Versatile Disc), an FD (Floppy Disk), an MO (Magneto-Optical disk), a BD (Blu-ray (registered trademark) Disc), a USB (registered trademark) memory, or an SD card.

[0026] In this embodiment, the program may be written to the memory 102 or the storage 103 during the manufacturing stage of the information processing device 100, or may be provided to the information processing device 100 via the network N, or may be provided to the information processing device 100 via a non-transitory computer-readable recording medium such as the recording medium 108.

[0027] <Functional configuration of image generation device 1> Next, a description will be given of the functional configuration of the image generation device 1. Fig. 3 is a diagram showing an example of the functional configuration of the image generation device 1. As shown in Fig. 3, the image generation device 1 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0028] The communication unit 11 is realized by the communication I / F 104. The communication unit 11 transmits and receives information to and from the terminal device 2 or the language model device 3 via the network N.

[0029] The storage unit 12 is realized by the memory 102 and the storage 103. The storage unit 12 stores a room image 121, catalog information 122, template information 123, instruction information 124, a renovation image 125, and area change information 126.

[0030] The room image 121 is an image that is the basis for the renovation image 125. The room image 121 is an indoor image that includes one or more interiors. The room image 121 may be photographic data captured by a camera or the like. The room image 121 is an image file that can be processed by an information processing device, such as JPEG (Joint Photographic Exteriors Group), BMP (Beat Per Minute), or GIF (Graphics Interchange Format). The room image 121 is transmitted from the terminal device 2 and one or more room images 121 are stored in the memory unit 12.

[0031] The catalog information 122 includes information related to interior decorations that decorate indoor spaces. The catalog information 122 is information that associates catalog images with category information. The catalog images are images that represent interior decorations that decorate indoor spaces. The interior decorations are objects that are installed indoors and decorate the interior. Examples of the interior decorations include wallpaper, flooring, curtains, lighting, and furniture (e.g., chairs, tables, bookshelves, etc.). The category information indicates the category of the catalog image. The category is information for classifying interior decorations that have similar uses and functions. The categories may be given general names for interior decorations, such as wallpaper, flooring, curtains, lighting, chairs, tables, and bookshelves. The category information may be given in Japanese or in another language, such as English. The catalog information 122 may be stored in advance in the storage unit 12 or may be uploaded by a user from the terminal device 2. The uploaded catalog information 122 is stored in the storage unit 12. One or more pieces of catalog information 122 are stored in the storage unit 12.

[0032] The catalog information 122 may include area information indicating an interior area in the catalog image. For example, the catalog image may include areas other than the interior. In such a case, when the renovation image 125 is generated, the catalog image including the area other than the interior may be replaced with the interior of the room image 121. To prevent such an unintended replacement by the user, it is desirable that the renovation image 125 be generated based on the interior area indicated in the area information. This allows the renovation image generation system 1000 to generate a more appropriate renovation image 125.

[0033] The catalog information 122 may include interior feature information. The feature information is textual information about the features of the interior depicted in the image. The feature information may be textual information about the impression given to people inside the room, such as bright, modern, or cool. The feature information is used to narrow down or select the catalog information 122 when generating the renovation image 125. The feature information 122 may be included in the catalog information 122 in advance, or may be assigned by artificial intelligence such as a machine learning model based on the catalog image. The feature information may be assigned in Japanese or in another language such as English. Note that the image to which feature information is assigned is not limited to the catalog image, but may also be a room image 121. In this case, the feature information is textual information about the features of the interior depicted in the room image 121.

[0034] The catalog information 122 may include the cost of the interior decoration. The cost of the interior decoration is the selling price of the interior decoration shown in the catalog information. The cost of the interior decoration may also include costs related to the purchase and construction of the interior decoration, such as shipping fees and labor costs.

[0035] The catalog information 122 may include interior dimensions. The interior dimensions are numerical values ​​that specifically indicate the size of the interior. The interior dimensions are expressed in three items: width, depth, and height.

[0036] The template information 123 is information that indicates, in a predetermined language, a template of an instruction (prompt) for a predetermined machine learning model. The template information 124 is composed of a character string indicating the content of the instruction and zero or more input fields. Individual information for guiding or constraining the output of the machine learning model is input into the input field. Information input by a user via the terminal device 2 is input into the input field. The language used for the template information 124 may be a natural language other than Japanese. For example, the natural language may be any language that can be processed by a machine learning model, such as English, German, or Chinese. The storage unit 12 may store one or more types of template information 124 in advance. Predetermined information recorded in the storage unit 12 may be input into the input field. The predetermined information is, for example, information included in the catalog information 122 that is specified by the user. For example, if category information is specified by the user, the category information is input into the input field. Furthermore, interior feature information extracted from the room image 121 may be input into the input field. The template information 124 may include instructions for the machine learning model to select an interior from the catalog information 122. For example, the template information 124 may include instructions to select an interior having interior characteristics specified by the user.

[0037] The instruction information 124 is a prompt that indicates an instruction to a predetermined machine learning model. The instruction information 124 is generated by inputting information input by a user via the terminal device 2 into an input area of ​​the template information 123. The language used for the instruction information 124 may be a natural language other than Japanese. For example, the natural language may be any language that can be processed by a machine learning model, such as English, German, or Chinese. Furthermore, the instruction information 124 may be generated by inputting information pre-stored in the storage unit 12, such as information included in the catalog information 122, or interior feature information extracted from the room image 121, into the input area of ​​the template information 123.

[0038] The renovation image 125 is an image generated by the image generation unit 140. The renovation image 125 is an image in which an interior drawn in the room image 121 to which category information selected by the user has been assigned is replaced with another interior. The other interior may be an interior shown in a catalog image received by the interior reception unit 134, or may be an interior acquired by the catalog acquisition unit 140. The renovation image 125 is an image file that can be manipulated by an information processing device, such as JPEG, BMP, or GIF. The generated renovation image 125 is transmitted to the terminal device 2. One or more renovation images 125 are generated by the image generation unit 140.

[0039] The area change information 126 is information that indicates changes to the areas segmented in the room image 121. The area change information 126 is generated by the application unit 232 of the terminal device 2 through a user operation. The area change information 126 may be information for changing category information assigned to the interior of the room image 121. The area change information 126 may be a mask image in which each area of ​​the interior is color-coded.

[0040] The control unit 13 is realized by the processor 101 reading and executing a program from the memory 102 and working in cooperation with other hardware configurations. The control unit 13 controls the overall operation of the image generation device 1. The control unit 13 includes an image acquisition unit 131, a segmentation unit 132, a category selection unit 133, an interior reception unit 134, a feature extraction unit 135, a catalog acquisition unit 136, a feature selection unit 137, a recommendation unit 138, an instruction information generation unit 139, an image generation unit 140, and a change information acquisition unit 141.

[0041] The image acquisition unit 131 acquires a room image 221 from a terminal device 2 connected to the network N. The image acquisition unit 131 records the acquired room image 221 in the storage unit 12 as a room image 121. The image acquisition unit 131 may acquire a plurality of room images 221 from the terminal device 2.

[0042] The segmentation unit 132 performs segmentation processing on the room image 121. Specifically, the segmentation unit 132 identifies each interior area depicted in the room image 121 and assigns interior category information to each area. The segmentation unit 132 identifies each interior area included in the room image 121 by, for example, a known segmentation technique using a machine learning model or the like. The segmentation unit 132 classifies each pixel or pixel group in the room image 121 and assigns category information to each piece of interior, such as a wall, floor, table, chair, or lighting fixture. As a result, each interior area in the room image 121 is generated as a segmentation map associated with the category information. Note that the segmentation unit 132 may use a semantic segmentation model trained by supervised learning (e.g., U-Net, Mask R-CNN, Segment Anything Model (SAM), etc.). Furthermore, the segmentation unit 132 may receive the room image 121 as input and output a mask image representing each interior area.

[0043] Furthermore, the segmentation unit 132 may be able to change the segmented region or category information in response to a user correction operation. For example, the segmentation unit 132 changes the segmented region based on the region change information 126. The region change information 126 is acquired from the terminal device 2 by the change information acquisition unit 141. This allows the segmentation unit 132 to reflect manual corrections to the classification results obtained by the machine learning model and identify the interior region with higher accuracy. For example, if the region change information 126 is a mask image whose region has been changed on the terminal device 2, the interior region is changed according to the region of the mask image. In this way, since the segmentation unit 132 can correct the segmented region using the region change information 126, it is possible to correct misrecognitions made in the segmentation process and generate a more accurate remodeled image 125.

[0044] The category selection unit 133 accepts a selection of category information from the terminal device 2. Specifically, when the segmentation unit 132 assigns category information to the room image 121, the category selection unit 133 transmits the assigned category information to the terminal device 2. Next, the category selection unit 133 accepts a selection of category information by the user from the terminal device 2.

[0045] The interior design reception unit 134 receives a selection of catalog information 122 from the terminal device 2. Specifically, when the category selection unit 133 receives a selection of category information by the user from the terminal device 2, the interior design reception unit 134 acquires, from the storage unit 12, one or more pieces of catalog information 122 corresponding to the category information selected by the user. The interior design reception unit 134 transmits the acquired catalog information 122 to the terminal device 2. Next, the interior design reception unit 134 accepts, from the terminal device 2, the selection of catalog information 122 by the user.

[0046] The feature extraction unit 135 extracts feature information of the interior depicted in the room image 121. Specifically, when the room image 121 is acquired from the terminal device 2, the feature extraction unit 135 inputs the room image 121 into a predetermined machine learning model and extracts features for each interior depicted in the room image 121. The machine learning model may be a large-scale language model of the language model device 3, or may be another machine learning model installed in an information processing device such as the image generation device 1. In this case, the features extracted from each interior are treated as feature information. Note that the feature extraction unit 135 may extract feature information using a known method such as pattern matching instead of a machine learning model. Furthermore, when the image acquisition unit 131 acquires multiple room images 121, the feature extraction unit 135 extracts feature information from each room image 121.

[0047] The catalog acquisition unit 136 acquires a catalog image. Specifically, the catalog acquisition unit 136 acquires a catalog image of catalog information including category information selected by the user and feature information extracted from the room image 121 by the feature extraction unit 135. For example, if the category information selected by the user is "chair" and the feature information extracted by the feature information 135 is "modern," the catalog acquisition unit 136 acquires a catalog image including "chair" in the catalog information and "modern" in the feature information. Note that, if multiple catalog images are acquired, the catalog acquisition unit 136 may accept a selection of one of the catalog images from the terminal device 2, or may randomly select one catalog image. In this way, by the catalog acquisition unit 136 acquiring a catalog image using the feature information extracted from the room image 121, it is possible to easily select an interior that harmonizes with the actual room environment.

[0048] When the image acquisition unit 131 acquires multiple room images 121 and extracts multiple pieces of feature information, the catalog acquisition unit 136 acquires catalog images having feature information common to any of the images. For example, assume that the image acquisition unit 131 acquires three room images depicting chairs as interior decor, and extracts "modern" from the first image, "bright" from the second image, and "modern" from the third image. In this case, the catalog acquisition unit 136 acquires catalog images of chairs having "modern" or "bright" as feature information. Note that, because two pieces of "modern" and one piece of "bright" have been extracted as feature information, the catalog acquisition unit 136 may preferentially acquire catalog images of chairs having "modern." In this way, the catalog acquisition unit 136 can select interior decor based on features common to multiple room images 121, allowing the user to propose renovations that maintain a sense of unity throughout the home.

[0049] The feature selection unit 137 accepts the selection of feature information by the user from the terminal device 2. Specifically, when the category selection unit 133 accepts the selection of category information from the terminal device 2, the feature selection unit 137 transmits the feature information of the interior included in the catalog information including the accepted category information to the terminal device 2. Next, the feature selection unit 137 accepts the selection of feature information by the user from the terminal device 2.

[0050] The recommendation unit 138 recommends interior items that satisfy conditions specified by a user from among a plurality of catalog information stored in the storage unit 12. Specifically, the recommendation unit 138 transmits catalog information 122 that satisfies the conditions to the terminal device 2. The conditions specified by the user are conditions used by the recommendation unit 138 to narrow down the interior items to be recommended to the user of the terminal device 2. The specified conditions are, for example, characteristic information, budget, or dimensions. Note that the conditions specified by the user are not limited to these and may be any information contained in the catalog information. Note that the recommendation unit 138 may function when conditions are specified by the user. In this way, the recommendation unit 138 can narrow down the catalog information 122 according to the conditions specified by the user, thereby reducing the processing load on the image generation device 1 and the terminal device 2 and allowing the user to efficiently suggest appropriate interior items from a vast amount of catalog information.

[0051] Next, a case where feature information is specified as a condition will be described. The recommendation unit 138 recommends interior items including the category information and feature information selected by the user. Specifically, when the category selection unit 133 accepts a selection of category information from the terminal device 2 and the feature selection unit 137 accepts a selection of interior feature information from the terminal device 2, the recommendation unit 138 acquires catalog information 122 including both the accepted category information and feature information from the storage unit 12. The recommendation unit 138 transmits a catalog image included in the acquired catalog information 122 to the terminal device 2. In this way, the user selects feature information of the interior, allowing the recommendation unit 138 to make suggestions based on the user's image.

[0052] Next, a case where a budget is specified as a condition will be described. The recommendation unit 138 recommends interior items so that the costs fall within a predetermined budget. Specifically, when the category selection unit 133 accepts a selection of category information from the terminal device 2 and the recommendation unit 138 accepts information related to the budget from the terminal device 2, the recommendation unit 138 acquires catalog information 122 including the accepted category information from the storage unit 12. Next, the recommendation unit 138 compares the cost of the interior items included in the acquired catalog information 122 with the accepted budget. The recommendation unit 138 extracts catalog information 122 so that the cost falls within the accepted budget. The recommendation unit 138 transmits a catalog image included in the extracted catalog information 122 to the terminal device 2. In this way, by taking the budget into consideration, the user can propose optimal renovations within the customer's budget. Note that when multiple category information items are selected, the recommendation unit 138 may extract multiple pieces of catalog information 122 so that the total amount falls within the accepted budget.

[0053] Next, a case where dimensions are specified as conditions will be described. The recommendation unit 138 recommends interiors that meet the interior dimension conditions specified by the user. Specifically, when the category selection unit 133 accepts a selection of category information from the terminal device 2 and the recommendation unit 138 accepts information related to dimensions from the terminal device 2, the recommendation unit 138 acquires catalog information 122 including the accepted category information from the storage unit 12. Next, the recommendation unit 138 compares the dimensions of the interiors included in the acquired catalog information 122 with the accepted dimensions. The recommendation unit 138 extracts catalog information 122 of interiors having dimensions smaller than the accepted dimensions. The recommendation unit 138 transmits catalog images included in the extracted catalog information 122 to the terminal device 2. The recommendation unit 138 may exclude from the extraction target any catalog information 122 of interior items having any one of the three dimensions, width, depth, and height, larger than the received dimensions, or may exclude only the catalog information 122 of interior items having dimensions larger than any of the three dimensions, or may exclude from the extraction target any catalog information 122 of interior items having dimensions larger than the sum of the dimensions of the three dimensions. In this way, the recommendation unit 138 can efficiently select interior items that fit the spatial constraints of the room by extracting catalog information 122 based on dimensions.

[0054] If the catalog information 122 that satisfies the conditions is not stored in the storage unit 12, the recommendation unit 138 may transmit to the terminal device 2 a text message indicating that the corresponding interior item could not be extracted.

[0055] The instruction information generation unit 139 generates instruction information 124 for the machine learning model. For example, the instruction information generation unit 139 acquires a request instruction regarding replacement of interior design from the terminal device 2. Next, the instruction information generation unit 139 generates the instruction information 124 by inputting the acquired request instruction into the template information 123. For example, the request instruction represents a user request for generating the renovation image 125, such as "I want you to generate a renovation image 125 in which the interior design to be replaced is placed in the foreground of the room." The request instruction is input in text by the user via the terminal device 2.

[0056] The request instruction may include an instruction to extract catalog information 122 that includes interior features specified by the user, such as "modern." In this case, the instruction information generation unit 139 generates instruction information 124 to extract catalog information 122 that includes the specified features as feature information. Note that the instruction information generation unit 139 may also generate instruction information 124 to extract catalog information 122 that includes the feature information extracted by the feature extraction unit 135.

[0057] The image generation unit 140 generates a renovation image 125 based on various information acquired from the terminal device 2. For example, the image generation unit 140 generates a renovation image 125 by replacing an interior depicted in the room image 121 to which category information selected by the user has been assigned with an interior represented by a catalog image accepted by the interior acceptance unit 134. The image generation unit 140 displays the generated renovation image 125 on the terminal device 2. Note that the image generation unit 140 may generate a plurality of renovation images 125.

[0058] The image generation unit 140 may input information into a machine learning model to generate the renovation image 125. In this case, the image generation unit 140 inputs the catalog image, the instruction information 124, and the room image 121 into the machine learning model. The machine learning model is, for example, a large-scale language model 322 stored in the language model device 3. In this case, the image generation unit 140 transmits the catalog image, the instruction information 124, and the room image 121 to the machine learning model. The image generation unit 140 also acquires the renovation image 125 from the language model device 3. In this way, the image generation unit 140 generates the instruction information 124 and performs image generation, thereby generating diverse and natural renovation images 125 in response to user requests.

[0059] The image generation unit 140 may generate the renovation image 125 by replacing the interior depicted in the room image 121 with the interior indicated by the area information in the catalog image. In this case, the image generation unit 140 extracts the interior area based on the area information indicating the interior area in the catalog image. The image generation unit 140 uses the extracted interior area as the catalog image to generate the renovation image 125. By using the area information in the catalog image, the image generation unit 140 can prevent erroneous replacement of parts other than the interior and generate a natural and unnatural-looking renovation image 125.

[0060] Furthermore, the image generation unit 140 may generate a renovation image by further inputting the catalog information 122 into the machine learning model in addition to the catalog image. For example, if the request instruction includes interior features specified by the user and the template information 123 includes an instruction to select an interior having the specified interior features, the image generation unit 140 further inputs the catalog information 122 into the machine learning model to generate the renovation image 125. In this case, the catalog image used to generate the renovation image 125 is selected from the catalog information 122 by the machine learning model.

[0061] The modification information acquisition unit 141 acquires, from the terminal device 2, area modification information 126 for modifying an area segmented in the room image 121.

[0062] The functional configuration of the image generation device 1 is not limited to the above example. For example, the image generation device 1 may have some of the above functional configuration, with the remaining functional configuration being provided by the terminal device 2 or the language model device 3. The image generation device 1 may also have functional configurations other than those described above. Each functional configuration of the image generation device 1 may be realized by software, as described above, or by hardware such as an IC chip, an SoC, an LSI, or a microcomputer.

[0063] <Functional configuration of terminal device 2> Next, a functional configuration of the terminal device 2 will be described. Fig. 4 is a diagram showing an example of the functional configuration of the terminal device 2. Description overlapping with that of the image generating device 1 shown in Fig. 3 will be omitted as appropriate. As shown in Fig. 4, the terminal device 2 includes a communication unit 21, a storage unit 22, a control unit 23, a display unit 24, and an operation unit 25.

[0064] The communication unit 21 is realized by the communication I / F 104. The communication unit 21 transmits and receives information to and from the image generating device 1 via the network N.

[0065] The storage unit 22 is realized by the memory 102 and the storage 103. The storage unit 22 stores a room image 221 and a renovation image 222.

[0066] The room image 221 is an image that is the basis for the renovation image 125. The room image 221 is an indoor image that includes one or more interiors. The room image 221 may be photographic data captured by a camera or the like. The room image 221 is an image file that can be processed by an information processing device, such as JPEG, BMP, or GIF. One or more room images 121 are stored in advance in the storage unit 22.

[0067] The reformed image 222 is an image generated by the image generating unit 140. The reformed image 222 is transmitted from the image generating device 1. One or more reformed images 222 are stored in the storage unit 22.

[0068] The control unit 23 is realized by the processor 101 reading and executing a program from the memory 102 and working in cooperation with other hardware components. The control unit 23 controls the overall operation of the terminal device 2. The control unit 23 includes an acquisition unit 231 and an application unit 232.

[0069] The acquisition unit 231 acquires information from various devices connected to the network N. For example, the acquisition unit 231 transmits the room image 221 to the image generation device 1 and acquires a program for acquiring the renovation image 125 from the image generation device 1. The acquisition unit 231 acquires various information from the image generation device 1. The various information may be, for example, category information assigned to the interior depicted in the room image 121, information included in the catalog information 122, or the renovation image 125. The information acquired by the acquisition unit 231 is not limited to these, and may be any information transmitted and received by the renovation image generation system 1000. The acquisition unit 231 may record the information acquired from the image generation device 1 in the storage unit 22.

[0070] The application unit 232 is communicably connected to the image generation device 1 via the network N to acquire the reform image 125. Specifically, the application unit 232 requests a program from the image generation device 1 via operation of the operation unit 25. Thereafter, the program acquired by the acquisition unit 231 is loaded into the control unit 23 of the terminal device 2. The application unit 232 executes the loaded program. When the program is executed, the application unit 232 generates a screen for generating the reform image 125. The application unit 232 outputs the generated screen to the display unit 24.

[0071] The application unit 232 accepts various operations such as selection of a room image 221, selection of category information 122, selection of a catalog image, and input of a request instruction via operation of the operation unit 25. In addition to the operation unit 25, the application unit 232 may also accept various operations such as selection of a room image 221, selection of category information, selection of a catalog image, and input of a request instruction by the user's speech via a microphone built into the terminal device 2. The application unit 232 transmits results obtained by various operations such as selection of a room image 221, selection of category information, selection of a catalog image, and input of a request instruction to the image generating device 1. Furthermore, when the acquisition unit 231 acquires a remodeled image 125, the application unit 232 outputs the remodeled image 125 to the display unit 24.

[0072] The display unit 24 is, for example, a display, and is a device that outputs information as a screen of a graphical user interface that can be operated by the user. The display unit 24 may be included in the housing of the terminal device 2, or may be externally attached. Specifically, the display unit 24 may be implemented as a display device such as a CRT display, a liquid crystal display, an organic EL display, or a plasma display. It is preferable that these display devices are implemented by selectively using them depending on the type of the terminal device 2. The display unit 24 displays various screens generated by the application unit 232.

[0073] The operation unit 25 is a device that inputs information to the terminal device 2 in response to an operation by an operator. The operation unit 25 accepts an operation input made by a user. The operation input is transferred to the control unit 23 as a command signal. The operation unit 25 may be implemented as a touch panel integrated with the display unit 24. When the operation unit 25 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the operation unit 25. Instead of a touch panel, switch buttons, a mouse, a track pad, a QWERTY keyboard, etc. can be used as the operation unit 25. It is preferable that these input devices are implemented by selectively using them according to the type of terminal device 2.

[0074] The functional configuration of the terminal device 2 is not limited to the above example. For example, the terminal device 2 may have some of the above functional configuration, with the image generation device 1 or the language model device 3 having the rest. The terminal device 2 may also have functional configurations other than those described above. Each functional configuration of the terminal device 2 may be realized by software, as described above, or by hardware such as an IC chip, an SoC, an LSI, or a microcomputer.

[0075] <Functional Configuration of Language Model Device 3> Next, a description will be given of the functional configuration of the language model device 3. Fig. 5 is a diagram showing an example of the functional configuration of the language model device 3. Descriptions that overlap with those of the image generation device 1 shown in Fig. 3 will be omitted as appropriate. As shown in Fig. 5, the language model device 3 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0076] The communication unit 31 is realized by the communication I / F 104. The communication unit 31 transmits and receives information to and from the image generating device 1 via the network N.

[0077] The storage unit 32 is realized by the memory 102 and the storage 103. The storage unit 32 stores instruction information 321, a large-scale language model 322, and a reform image 323.

[0078] The instruction information 321 is a prompt to be input to a predetermined machine learning model. The instruction information 321 is the instruction information 124 sent from the image generating device 1.

[0079] The large-scale language model 322 is a machine learning model that can perform machine learning on large amounts of text data and generate natural-looking words and images. The large-scale language model 322 generates information in cooperation with each functional unit of the control unit 33. For example, when information is input from each functional unit of the control unit 33, the large-scale language model 322 generates information based on the input information. The generated information may be, for example, an image, a character string combining numbers, letters, or symbols, or statistical information such as a graph. The large-scale language model 322 outputs the generated information to each functional unit. The memory unit 32 may store one large-scale language model 322, or multiple large-scale language models 322 may be stored depending on the application. The type of large-scale language model 322 and the machine learning method are not limited.

[0080] The reformed image 323 is an image generated by the large-scale language model 322 in accordance with the instruction information 321. The reformed image 323 is generated by inputting the instruction information 321 into the large-scale language model 322.

[0081] The control unit 33 is realized by the processor 101 reading and executing a program from the memory 102 and working in cooperation with other hardware components. The control unit 33 controls the overall operation of the language model device 3. The control unit 33 includes an acquisition unit 331 and a response generation unit 332.

[0082] The acquisition unit 331 acquires information from various devices such as the image generation device 1 or the terminal device 2 connected to the network N. The acquisition unit 331 records the acquired information in the storage unit 32 and inputs it to the response generation unit 332. For example, the acquisition unit 331 acquires instruction information 124 from the image generation device 1 and records it as instruction information 321 in the storage unit 32. The acquisition unit 331 acquires a room image 121 from the image generation device 1 and inputs it to the response generation unit 332. The acquisition unit 331 acquires a catalog image from the image generation device 1 and inputs it to the response generation unit 332.

[0083] The response generation unit 332 generates a renovation image 323 based on the instruction information 321, the room image 121, and the catalog image. Specifically, the response generation unit 332 acquires the instruction information 321 from the storage unit 12. Next, the response generation unit 332 inputs the instruction information 321, the room image 121, and the catalog image into the large-scale language model 322. The response generation unit 332 acquires the renovation image 323 generated by the large-scale language model 322. The response generation unit 332 records the generated renovation image 323 in the storage unit 32. If the instruction information 321 includes an instruction to extract features of the room image 121, the response generation unit 332 may extract features of the interior depicted in the room image 121 and use them as interior feature information. If the instruction information 321 includes an instruction to determine the catalog information 122, the response generation unit 332 selects the catalog information 122 based on the extracted features. For example, the response generation unit 332 may select catalog information 122 that is common to feature information included in the catalog information 122.

[0084] The functional configuration of the language model device 3 is not limited to the above example. For example, the language model device 3 may have some of the above functional configuration, with the image generation device 1 or the terminal device 2 having the rest. The language model device 3 may also have functional configurations other than those described above. Each functional configuration of the language model device 3 may be realized by software, as described above, or by hardware such as an IC chip, an SoC, an LSI, or a microcomputer.

[0085] <Reform image generation method> The following describes a renovation image generation method executed by the renovation image generation system 1000. FIG. 6 is a flowchart showing the first half of a first example of the renovation image generation method. FIG. 7 is a flowchart showing the second half of the first example of the renovation image generation method. The renovation image generation method is executed by a terminal device 2 accessing the image generation device 1. The terminal device 2 accesses the image generation device 1 using an application such as a web browser.

[0086] (Step S101) The acquisition unit 231 of the terminal device 2 acquires the room image 221 from the storage unit 22. Specifically, the application unit 232 of the terminal device 2 requests a program from the image generation device 1 via operation of the operation unit 25. Thereafter, the program acquired by the acquisition unit 231 is loaded into the control unit 23 of the terminal device 2. The application unit 232 executes the loaded program. When the program is executed, the application unit 232 of the terminal device 2 generates an upload screen for the room image 221 for generating a renovation image. The application unit 232 outputs the generated upload screen to the display unit 24. The acquisition unit 231 acquires the room image 221 from the storage unit 22 via operation of the operation unit 25.

[0087] Fig. 8 is a diagram schematically illustrating an example of the upload screen for a room image before uploading. In Fig. 8, the application unit 232 generates an upload screen 400 and displays it on the display unit 24. The upload screen 400 has a room image display area 401, a full room button 402, a custom button 403, a remodeled image display area 404, a history display area 405, and an image generation button 406.

[0088] The room image display area 401 is an area that accepts an operation for uploading the room image 221 to the image generating device 1. The application unit 232 accepts an operation on the room image display area 401 from the user of the terminal device 2 via an operation on the operation unit 25. In response to the operation on the room image display area 401, the room image 221 is uploaded to the image generating device 1.

[0089] The full room button 402 is a button that is operated when the uploaded room image 221 is an indoor sketch image. The application unit 232 accepts selection of the full room button 402 from the user of the terminal device 2 via the operation unit 25. When the full room button 402 is selected, the full room button 402 is highlighted. If the custom button 403 is already highlighted, the highlighting of the custom button 403 is canceled in response to the highlighting of the full room button 402. The highlighting indicates that the button has been selected via the operation unit 25.

[0090] The custom button 403 is a button that is operated when the uploaded room image 221 is an image of an interior of a room that is to be renovated. The application unit 232 accepts a selection for the custom button 403 from the user of the terminal device 2 via the operation unit 25. When the custom button 403 is selected, the custom button 403 is highlighted. If the custom button 403 is already highlighted, the highlighting of the custom button 403 is canceled in response to the highlighting of the custom button 403. The highlighting indicates that the button has been selected via the operation unit 25.

[0091] The reform image display area 404 is an area that displays the generated reform image 125. The reform image display area 404 displays the reform image 125 that was generated in response to the operation of the image generation button 406. The reform image display area 404 may display the reform image 125 that was selected via the operation unit 25 from among the reform images displayed in the history display area 405. The reform image display area 404 includes a download button 404a and an edit button 404b.

[0092] The download button 404a is a button for downloading the reform image 125 displayed in the reform image display area 404 to the terminal device 2. When the download button 404a is operated via the operation unit 25, the reform image 222 is recorded in the memory unit 12. The edit button 404b is a button for editing the reform image 125 displayed in the reform image display area 404. When the edit button 404b is operated via the operation unit 25, the application unit 232 generates an editing screen for the reform image. The editing screen is displayed on the display unit 24.

[0093] The history display area 405 is an area where previously generated reform images are displayed. When a reform image 125 is generated, the reform images 125 generated up to that point are displayed in the history display area 405. When a reform image displayed in the history display area 405 is selected via the operation unit 25, the selected reform image is displayed in the reform image display area 404.

[0094] The image generation button 406 is a button for generating a remodeled image. When the image generation button 406 is selected via the operation unit 25, a remodeled image 125 is generated based on the room image 221 displayed in the room image display area 401, and is displayed in the remodeled image display area 404.

[0095] Returning to FIG. 6, the reformed image generation method will be described. (Step S102) When the acquisition unit 231 acquires the room image 221, the acquisition unit 231 transmits the room image 221 to the image generation device 1. In other words, the room image 221 is uploaded to the image generation device 1. The image acquisition unit 131 of the image generation device 1 acquires the room image 221 from the terminal device 2. The image acquisition unit 131 records the acquired room image 221 in the storage unit 12 as the room image 121.

[0096] (Step S103) The segmentation unit 132 performs a segmentation process on the room image 121. The segmentation unit 132 identifies each interior area drawn in the room image 121, and assigns interior category information to each area.

[0097] (Step S104) The segmentation unit 132 transmits the interior area and category information to the terminal device 2. The acquisition unit 231 acquires the interior area and category information from the image generation device 1.

[0098] (Step S105) The application unit 232 of the terminal device 2 accepts the selection of category information. Specifically, when the interior area and category information are acquired, the application unit 232 generates a category reception screen. The application unit 232 outputs the generated category reception screen to the display unit 24. Next, the application unit 232 accepts the selection of category information via an operation on the operation unit 25.

[0099] Fig. 9 is a diagram schematically illustrating an example of a category reception screen for category information. In Fig. 9, the application unit 232 generates a category reception screen 400a and displays it on the display unit 24. The reception screen 400 has a room image display area 401a, a full room button 402, a custom button 403, a remodeled image display area 404, a history display area 405, an image generation button 406, and an operation area 407. Note that the explanation of the symbols explained in Fig. 8 will be omitted as appropriate.

[0100] The room image display area 401a is an area for displaying a room image 221 uploaded to the image generating device 1. The application unit 232 may accept an operation to upload another room image 221 from the terminal device 2 via an operation on the operation unit 25. In this case, the room image 221 uploaded later is displayed in the room image display area 401a.

[0101] The operation area 407 is an area that displays buttons and input areas that can be operated via the operation unit 25 of the terminal device 2. For example, the operation area 407 includes a category selection area 471 and an instruction input area 472.

[0102] The category selection area 471 is an area where category information assigned to the interior drawn in the room image displayed in the room image display area 401a is displayed. The category selection area 471 displays "Walls" representing wallpaper, "Windows" representing windows, "Bed" representing a bed, "Table" representing a table, and "Floor" representing a floor. The user selects any of the category information displayed in the category selection area 471 via the operation unit 25. When category information is selected, the application unit 232 may highlight (for example, by making the border bold or changing the color) the area of ​​the interior of the room image displayed in the room image display area 401a to which the selected category information is assigned. Multiple category information may be selected. The application unit 232 transmits the selected category information to the image generating device 1. The category information displayed in the category selection area 471 is the category information assigned to the interior drawn in the room image 121.

[0103] The instruction input area 472 is an area where a request instruction is input via the operation unit 25 of the terminal device 2. The user of the terminal device 2 inputs a request instruction into the instruction input area 472 via the operation unit 25. Next, when the user of the terminal device 2 operates the image generation button 406 via the operation unit 25, the application unit 232 transmits the request instruction input into the instruction input area 472 to the image generation device 1.

[0104] Returning to FIG. 6, the reformed image generation method will be described. (Step S106) The application unit 232 transmits the received category information to the image generating device 1 . (Step S107) The interior design receiving unit 134 acquires the catalog information 122. Specifically, the interior design receiving unit 134 acquires from the storage unit 122 one or more pieces of catalog information 122 that include the received category information.

[0105] (Step S108) The interior design reception unit 134 transmits the catalog information 122 to the terminal device 2. The acquisition unit 231 acquires the catalog information 122 from the image generation device 1. The application unit 232 generates a selection screen for the catalog information 122 based on the acquired catalog information 122.

[0106] (Step S109) The application unit 232 of the terminal device 2 accepts the selection of the catalog information 122. Specifically, when the acquisition unit 231 acquires the catalog information 122, the application unit 232 generates a selection screen for the catalog information 122. The application unit 232 outputs the generated selection screen for the catalog information 122 to the display unit 24. Next, the application unit 232 accepts the selection of the catalog information 122 via an operation on the operation unit 25.

[0107] Fig. 10 is a diagram schematically illustrating an example of a selection screen when "Chair" is selected as category information. In Fig. 10, the application unit 232 generates a selection screen 500 for the catalog information 122 and displays it on the display unit 24. The selection screen 500 for the catalog information 122 may be displayed superimposed on the category reception screen 400a. The selection screen 500 for the catalog information 122 has catalog information display areas 501, 502, and 503. The acquired catalog information 122 is displayed in the catalog information display areas. Each of the catalog information display areas 501, 502, and 503 displays one piece of catalog information.

[0108] The catalog information display area 501 is an area for displaying the catalog information 122 transmitted from the image generating device 1 to the terminal device 2. The catalog information display area 501 displays various information included in the catalog information 122. For example, if the catalog information 122 includes a catalog image, category information, feature information, cost, and dimensions, the catalog information 122 is displayed as shown in the catalog information display area 501. According to the catalog information display area 501, the catalog image is an image drawn in the catalog upper table display area 501. The category information is "Chair." The feature information is "Relax." The cost is "88,800" and the dimensions are "120 x 100 x 80." The catalog information display areas 502 and 503 are displayed in the same way, so their explanations are omitted. The catalog information display areas 501, 502, and 503 accept the selection of any of the catalog information 122 via the operation unit 25 of the terminal device 2.

[0109] Returning to FIG. 6, the reformed image generation method will be described. (Step S110) When the image generation button 406 is selected via the operation unit 25, the application unit 232 receives an instruction to generate an image.

[0110] (Step S111) The application unit 232 transmits the received catalog information 122 to the image generating device 1. Note that, since the instruction input area 472 is blank in step S110, no request instruction is transmitted. If text has been input in the instruction input area 472, the application unit 232 transmits the input text to the image generating device 1 as a request instruction.

[0111] (Step S112) The instruction information generation unit 139 generates instruction information 124 for the machine learning model. Note that, since a request instruction has not been transmitted, the instruction information generation unit 139 inputs the category information selected by the user into the template information 123 to generate the instruction information 124. The instruction information 124 includes an instruction to generate a renovation image 125 in which the interior depicted in the room image 121 to which the category information selected by the user has been assigned is replaced with the interior depicted in the catalog image.

[0112] (Step S113) The image generation unit 140 generates a renovation image 125 based on the catalog information 122 and room image 121 acquired from the terminal device 2 and the generated instruction information 124. Specifically, when the image generation unit 140 accepts the selection of the catalog information 122, it acquires the catalog image included in the accepted catalog information 122. Next, the image generation unit 140 transmits the room image 121, the catalog image, and the instruction information 124 to the language model device 3. The acquisition unit 331 records the acquired instruction information 124 in the storage unit 32 as instruction information 321.

[0113] (Step S114) The response generation unit 332 generates a renovation image 323 based on various information acquired from the image generation device 1. Specifically, the response generation unit 332 acquires instruction information 321 from the memory unit 12. Next, the response generation unit 332 inputs the instruction information 321, the catalog image, and the room image 121 into the large-scale language model 322. The response generation unit 332 acquires the renovation image 323 generated by the large-scale language model 322. The response generation unit 332 records the generated renovation image 323 in the memory unit 32.

[0114] (Step S115) The response generation unit 332 transmits the generated reform image 323 to the image generation device 1. The image generation unit 140 acquires the reform image 323. The image generation unit 140 records the acquired reform image 323 in the storage unit 12 as the reform image 125.

[0115] (Step S116) The image generation unit 140 acquires the acquired reform image 125 from the storage unit 12. The image generation unit 140 transmits the reform image 125 to the terminal device 2. (Step S117) The acquisition unit 231 acquires the reform image 125. The acquisition unit 231 records the acquired reform image 231 in the storage unit 22 as a reform image 222. The application unit 232 causes the display unit 24 to display the acquired reform image 222.

[0116] Fig. 11 is a diagram schematically illustrating an example of a display screen 400b of the generated renovation image 222. In Fig. 11, the application unit 232 generates the renovation image display screen 400b based on the renovation image 222 stored in the storage unit 22 and displays it on the display unit 24. The renovation image display screen 400b has a room image display area 401a, a full room button 402, a custom button 403, a renovation image display area 404c, a history display area 405a, an image generation button 406, and an operation area 407. Note that the reference symbols explained in Figs. 8 and 9 will not be explained again as appropriate.

[0117] The history display area 405a is an area where the reform image 222 transmitted from the image generation device 1 is displayed. The history display area 405a displays multiple reform images 222. When one of the reform images 222 displayed in the history display area 405a is selected via operation of the operation unit 25, it may be displayed in the reform image display area 404c. Each time the image generation button 406 is selected, a newly generated reform image is added and displayed in the history display area 405a. When multiple reform images are generated, multiple additional reform images 222 are displayed at once in the history display area 405a.

[0118] The renovation image display area 404c is an area for displaying the renovation image 222 transmitted from the image generation device 1. For example, the renovation image display area 404c may display an image selected from the history display area 405a via operation of the operation unit 25, among the renovation images 222 transmitted from the image generation device 1. The renovation image display area 404c displays a renovation image 222 in which the word "Floor" representing the floor has been replaced from the room image 221 displayed in the room image display area 401a. In this way, the renovation image display area 404c displays the renovation image 222 corresponding to the room image 221 displayed in the room image display area 401a. Furthermore, when the download button 404a is operated, the renovation image 222 displayed in the renovation image display area 404c is downloaded to the terminal device 2. When the edit button 404b is operated, an editing screen for editing the renovation image 222 displayed in the renovation image display area 404c is displayed on the display unit 24. The edited reform image 222 may be overwritten on the reform image 222 before editing, or may be recorded in the storage unit 22 and added to the history display area 405a.

[0119] Next, a second method of reform image generation executed by the reform image generation system 1000 will be described. FIG. 12 is a flowchart showing the first half of a second example of the reform image generation method. FIG. 13 is a flowchart showing the second half of the second example of the reform image generation method. The reform image generation method is executed by the terminal device 2 accessing the image generation device 1. The terminal device 2 accesses the image generation device 1 using an application such as a web browser. Note that the contents of steps having the same reference numerals as those in the flowcharts of FIGS. 6 and 7 will be omitted as appropriate in the description.

[0120] Steps S101 to S108 are the same as those in FIG. 6, so the explanation will be omitted. (Step S201) The application unit 232 of the terminal device 2 acquires the narrowing down conditions via the operation unit 25. The narrowing down conditions are conditions for reducing the number of catalog information items displayed by adding other information to one or more pieces of displayed catalog information 122. The narrowing down conditions may be any information contained in the catalog information 122, such as characteristic information, cost, or dimensions. There may be multiple narrowing down conditions. The narrowing down conditions are displayed, for example, in the operation area 407. If cost can be set as a narrowing down condition, an input area in which the cost can be entered is displayed in the operation area 407. In Figures 12 and 13, a case where "Relax" in "characteristic information" is entered as a narrowing down condition will be described as an example.

[0121] (Step S202) The application unit 232 transmits the input narrowing-down conditions to the image generating device 1. In step S201, since "Relax" of "Feature Information" was input as a narrowing-down condition, the application unit 232 transmits "Relax" of "Feature Information" to the image generating device 1 as a narrowing-down condition.

[0122] (Step S203) The recommendation unit 138 acquires catalog information 122. Specifically, the recommendation unit 138 acquires one or more pieces of catalog information 122 including the category information received by the interior reception unit 134 and the acquired narrowing-down conditions from the storage unit 122. More specifically, the interior reception unit 134 acquires one or more pieces of catalog information 122 including the received category information and "Relax" in "characteristic information" as the acquired narrowing-down conditions from the storage unit 122.

[0123] (Step S204) The recommendation unit 138 transmits the catalog information 122 to the terminal device 2. The acquisition unit 231 acquires the catalog information 122 from the image generation device 1. The application unit 232 updates the selection screen for the catalog information 122 based on the acquired catalog information 122. Specifically, the application unit 232 updates the selection screen for the catalog information 122 so as to display the catalog information 122 including the category information accepted by the interior reception unit 134 and the information specified by the narrowing-down conditions. For example, when "Relax" is entered as the "characteristic information" as a narrowing-down condition, the application unit 232 updates the selection screen so as to display the catalog information 122 including "Relax" as the "characteristic information."

[0124] (Step S205) The application unit 232 of the terminal device 2 accepts the selection of the catalog information 122 via the operation unit 25. Specifically, when the selection screen for the catalog information 122 is updated by the application unit 232, the application unit 232 outputs the selection screen for the updated catalog information 122 to the display unit 24. Next, the application unit 232 accepts the selection of the catalog information via the operation of the operation unit 25.

[0125] (Step S206) The application unit 232 of the terminal device 2 accepts input of a request instruction via the operation unit 25. Specifically, the application unit 232 accepts text input to the instruction input area 472 via the operation unit 25. For example, by inputting an instruction related to the generation of the remodeled image 125, such as "replace only the floor at the back," into the instruction input area 472, the generated remodeled image 125 can be fine-tuned.

[0126] (Step S207) When the image generation button 406 is selected via the operation unit 25, the application unit 232 receives an instruction to generate an image.

[0127] (Step S208) The application unit 232 transmits the received catalog information 122 and request instruction to the image generating device 1. Note that, since text has been input into the instruction input area 472 in step S204, the application unit 232 transmits the input text as a request instruction to the image generating device 1.

[0128] (Step S209) The instruction information generation unit 139 generates instruction information 124 for the machine learning model. Specifically, the instruction information generation unit 139 generates instruction information 124 by inputting category information selected by the user and a request instruction transmitted from the terminal device 2 into template information 123. The subsequent steps are the same as those in FIG. 7, and therefore description thereof will be omitted.

[0129] <Summary> As described above, according to this embodiment, a renovation image generation system 1000 can be realized, which includes: a memory unit 12 that stores multiple catalog information 122 that associates catalog images representing interiors that decorate an indoor space with category information that represents the category of the catalog images; an image acquisition unit 131 that acquires room images; a segmentation unit 132 that identifies each interior area drawn in the room image and assigns interior category information to each area; a category selection unit 133 that transmits the category information assigned by the segmentation unit to a terminal device and accepts a user's selection of category information from the terminal device; an interior reception unit 134 that transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and accepts a user's selection of catalog images from the terminal device; and an image generation unit 140 that generates a renovation image in which the interiors drawn in the room image that have been assigned the category information selected by the user are replaced with the interior represented by the catalog image accepted by the interior reception unit, and displays the image on the terminal device.

[0130] The renovation image generation system 1000 performs segmentation on the room image 121, identifies interior areas depicted in the room image 121, and assigns category information, allowing the user to easily select the interior they wish to replace. This allows a renovation image 125 to be generated in which specific furniture, decorations, and other interior elements in the room image 121 are replaced with other interior elements through intuitive operations, without the need for specialized image editing techniques. Furthermore, because the user can select category information and catalog images via the terminal device 2, a variety of renovation proposals tailored to the user's preferences and purposes can be presented in a short period of time. Therefore, the renovation image generation system 1000 can realize more efficient and high-quality visual proposals in the fields of home renovation and interior proposals than conventional manual image editing and design proposals.

[0131] <Modification> In this embodiment, the large-scale language model 322 is recorded in the storage unit 32 of the language model device 3, but it may also be recorded in the storage unit 12 of the image generation device 1. In this case, the image generation unit 140 inputs the instruction information 124 to a machine learning model recorded in the storage unit 12, instead of transmitting the instruction information 124 to the language model device 3. Such a machine learning model may be an existing model, or may be a model developed by the developer of the image generation device 1.

[0132] In this embodiment, the catalog information 122 is configured to be selected by the user via the operation unit 25 of the terminal device 2, but the renovation image generation system 1000 may also be configured to select it. In this case, the user inputs text instructing the selection of the catalog image 122 into the instruction input area 472 via the operation unit 25 and presses the image generation button 406. By performing this operation, the renovation image generation system 1000 generates the renovation image 125 based on the selected category information, even if the catalog information 122 is not selected by the user. Note that when inputting into the instruction input area 472, the user may include feature information that will serve as a criterion for selecting a catalog image, such as "Please select a modern interior."

[0133] <Additional Notes> The present embodiment includes the following disclosure.

[0134] (Appendix 1) a storage unit that stores a plurality of catalog information items that associate catalog images representing interiors for decorating indoor spaces with category information that represents categories of the catalog images; an image acquisition unit that acquires a room image; a segmentation unit that identifies each interior area depicted in the room image and assigns category information of the interior to each of the areas; a category selection unit that transmits the category information assigned by the segmentation unit to a terminal device and receives a selection of the category information by a user from the terminal device; an interior design reception unit that transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and receives the selection of the catalog image by the user from the terminal device; an image generating unit that generates a renovation image by replacing the interior depicted in the room image to which the category information selected by the user is assigned with the interior represented by the catalog image received by the interior receiving unit, and displays the renovation image on the terminal device; A renovation image generation system comprising:

[0135] (Appendix 2) the storage unit stores template information of instructions to be input to the machine learning model; an instruction information generation unit that acquires a request instruction regarding replacement of the interior from the terminal device and generates instruction information for a machine learning model by inputting the request instruction into the template information; The image generation unit inputs the catalog image, the instruction information, and the room image into the machine learning model to generate the renovation image. 2. A renovation image generation system as described in appendix 1.

[0136] (Appendix 3) a modification information acquisition unit that acquires, from the terminal device, region modification information for modifying a region segmented in the room image; the segmentation unit modifies the segmented region based on the region modification information; A renovation image generation system according to appendix 1 or 2.

[0137] (Appendix 4) the catalog information includes area information indicating an interior area in a catalog image; the image generation unit replaces the interior depicted in the room image with the interior indicated by the area information to generate the renovation image. A renovation image generation system according to appendix 1 or 2.

[0138] (Appendix 5) the request instructions include interior features specified by the user; the template information includes an instruction to select an interior having the interior characteristics; The image generation unit further inputs the catalog information into the machine learning model to generate the renovation image. A renovation image generation system as described in Appendix 2.

[0139] (Appendix 6) the catalog information includes feature information of the interior; a feature extraction unit that extracts feature information of an interior from the room image; a catalog acquisition unit that acquires a catalog image of catalog information including the category information selected by the user and the feature information extracted by the feature extraction unit; A renovation image generation system according to appendix 1 or 2.

[0140] (Appendix 7) When the image acquisition unit acquires a plurality of room images, the feature extraction unit extracts the feature information from each of the room images; the catalog acquisition unit acquires a catalog image having characteristic information common to any of the extracted plurality of characteristic information. 7. A renovation image generation system as described in appendix 6.

[0141] (Appendix 8) a recommendation unit that recommends interior items that meet conditions specified by a user from among a plurality of catalog information; A renovation image generation system according to appendix 1 or 2.

[0142] (Appendix 9) the catalog information includes feature information of the interior; a feature selection unit that transmits the interior feature information included in the catalog information to the terminal device and receives a selection of the feature information by the user from the terminal device; the recommendation unit recommends interior items including category information and feature information selected by the user; 9. A renovation image generation system as described in appendix 8.

[0143] (Appendix 10) the catalog information includes the cost of the interior décor; the recommendation unit recommends the interior so that the cost falls within a predetermined budget. 9. A renovation image generation system as described in appendix 8.

[0144] (Appendix 11) the catalog information includes dimensions of the interior; the recommendation unit recommends the interior that satisfies a condition of the interior dimensions specified by the user; 9. A renovation image generation system as described in appendix 8.

[0145] (Appendix 12) The image generation unit generates a plurality of the reform images. A renovation image generation system according to appendix 1 or 2.

[0146] (Appendix 13) A method for generating a renovation image, comprising: storing a plurality of catalog information items in which catalog images representing interiors for decorating an indoor space and category information representing categories of the catalog images are associated with each other; an image acquisition step of acquiring a room image; a segmentation step of identifying each interior area depicted in the room image and assigning category information of the interior to each of the areas; a category selection step of transmitting the category information assigned by the segmentation unit to a terminal device and accepting a selection of the category information by a user from the terminal device; an interior design receiving step of transmitting one or more catalog images corresponding to the category information selected by the user to the terminal device and receiving a selection of the catalog image by the user from the terminal device; an image generating step of generating a renovation image by replacing the interior depicted in the room image to which the category information selected by the user has been assigned with the interior represented by the catalog image received by the interior reception unit, and displaying the image on the terminal device; A method for generating a renovation image.

[0147] (Appendix 14) a storage unit that stores a plurality of catalog information items that associate catalog images representing interiors for decorating indoor spaces with category information that represents categories of the catalog images; an image acquisition unit that acquires a room image; a segmentation unit that identifies each interior area depicted in the room image and assigns category information of the interior to each of the areas; a category selection unit that transmits the category information assigned by the segmentation unit to a terminal device and receives a selection of the category information by a user from the terminal device; an interior design reception unit that transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and receives the selection of the catalog image by the user from the terminal device; an image generating unit that generates a renovation image by replacing the interior depicted in the room image to which the category information selected by the user is assigned with the interior represented by the catalog image received by the interior receiving unit, and displays the renovation image on the terminal device; A reform image generating device comprising:

[0148] (Appendix 15) A renovation image generation system including a storage unit storing a plurality of catalog information items in which catalog images representing interiors for decorating indoor spaces and category information representing categories of the catalog images are associated with each other, an image acquisition step of acquiring a room image; a segmentation step of identifying each interior area depicted in the room image and assigning category information of the interior to each of the areas; a category selection step of transmitting the category information assigned by the segmentation unit to a terminal device and accepting a selection of the category information by a user from the terminal device; an interior design receiving step of transmitting one or more catalog images corresponding to the category information selected by the user to the terminal device and receiving a selection of the catalog image by the user from the terminal device; an image generating step of generating a renovation image by replacing the interior depicted in the room image to which the category information selected by the user has been assigned with the interior represented by the catalog image received by the interior reception unit, and displaying the image on the terminal device; A renovation image generation program that executes the above.

[0149] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. Furthermore, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0150] 1: Image generation device 2: Terminal device 3: Language model device 11, 21, 31: Communications Department 12,22,32: Storage section 13, 23, 33: Control unit 24: Display section 25:Operation section 131: Image acquisition unit 132: Segmentation section 133: Category selection section 134: Interior Reception Department 135: Feature extraction unit 136: Catalog Acquisition Department 137: Feature selection section 138: Recommendation Department 139: Instruction information generation unit 140: Image generation unit 141: Change information acquisition unit 231,331: Acquisition Department 232: Application section 332: Response generation unit

Claims

1. a storage unit that stores a plurality of catalog information items that associate catalog images representing interiors for decorating indoor spaces with category information that represents categories of the catalog images; an image acquisition unit that acquires a room image; a segmentation unit that identifies each interior area depicted in the room image and assigns category information of the interior to each of the areas; a category selection unit that transmits the category information assigned by the segmentation unit to a terminal device and receives a selection of the category information by a user from the terminal device; an interior design receiving unit that transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and receives the selection of the catalog image by the user from the terminal device; an image generating unit that generates a renovation image by replacing the interior depicted in the room image to which the category information selected by the user is assigned with the interior represented by the catalog image received by the interior receiving unit, and displays the renovation image on the terminal device; A renovation image generation system comprising:

2. the storage unit stores template information of instructions to be input to the machine learning model; an instruction information generation unit that acquires a request instruction regarding replacement of the interior from the terminal device and generates instruction information for a machine learning model by inputting the request instruction into the template information; The image generation unit inputs the catalog image, the instruction information, and the room image into the machine learning model to generate the renovation image. The renovation image generation system according to claim 1 .

3. a modification information acquisition unit that acquires, from the terminal device, region modification information for modifying a region segmented in the room image; the segmentation unit modifies the segmented region based on the region modification information; The reform image generation system according to claim 1 or 2.

4. the catalog information includes area information indicating an interior area in a catalog image; the image generation unit replaces the interior depicted in the room image with the interior indicated by the area information to generate the renovation image. The reform image generation system according to claim 1 or 2.

5. the request instructions include interior features specified by the user; the template information includes an instruction to select an interior having the interior characteristics; The image generation unit further inputs the catalog information into the machine learning model to generate the renovation image. The reform image generation system according to claim 2 .

6. the catalog information includes feature information of the interior; a feature extraction unit that extracts feature information of an interior from the room image; a catalog acquisition unit that acquires a catalog image of catalog information including the category information selected by the user and the feature information extracted by the feature extraction unit; The reform image generation system according to claim 1 or 2.

7. When the image acquisition unit acquires a plurality of room images, the feature extraction unit extracts the feature information from each of the room images; the catalog acquisition unit acquires a catalog image having characteristic information common to any of the extracted plurality of characteristic information. The reform image generation system according to claim 6.

8. a recommendation unit that recommends interior items that meet conditions specified by a user from among a plurality of catalog information; The reform image generation system according to claim 1 or 2.

9. the catalog information includes feature information of the interior; a feature selection unit that transmits the interior feature information included in the catalog information to the terminal device and receives a selection of the feature information by the user from the terminal device; the recommendation unit recommends interior items including category information and feature information selected by the user; The reform image generation system according to claim 8.

10. the catalog information includes the cost of the interior décor; the recommendation unit recommends the interior so that the cost falls within a predetermined budget. The reform image generation system according to claim 8.

11. the catalog information includes dimensions of the interior; the recommendation unit recommends the interior that satisfies a condition of the interior dimensions specified by the user; The reform image generation system according to claim 8.

12. The image generation unit generates a plurality of the reform images. The reform image generation system according to claim 1 or 2.

13. A renovation image generation method executed by an information processing device having a storage unit that stores a plurality of catalog information in which catalog images representing interiors that decorate indoor spaces and category information representing categories of the catalog images are associated with each other, an image acquisition step in which the information processing device acquires a room image; a segmentation step in which the information processing device identifies each interior area drawn in the room image and assigns category information of the interior to each of the areas; a category selection step in which the information processing device transmits the category information assigned in the segmentation step to a terminal device and receives a selection of the category information by a user from the terminal device; an interior design receiving step in which the information processing device transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and receives a selection of the catalog image by the user from the terminal device; an image generation step in which the information processing device generates a renovation image by replacing the interior depicted in the room image to which the category information selected by the user has been assigned with the interior represented by the catalog image received in the interior reception step, and displays the renovation image on the terminal device; A method for generating a reform image.

14. a storage unit that stores a plurality of catalog information items that associate catalog images representing interiors for decorating indoor spaces with category information that represents categories of the catalog images; an image acquisition unit that acquires a room image; a segmentation unit that identifies each interior area depicted in the room image and assigns category information of the interior to each of the areas; a category selection unit that transmits the category information assigned by the segmentation unit to a terminal device and receives a selection of the category information by a user from the terminal device; an interior design receiving unit that transmits one or more catalog images corresponding to the category information selected by the user to the terminal device and receives the selection of the catalog image by the user from the terminal device; an image generating unit that generates a renovation image by replacing the interior depicted in the room image to which the category information selected by the user is assigned with the interior represented by the catalog image received by the interior receiving unit, and displays the renovation image on the terminal device; A reform image generating device comprising:

15. A renovation image generation system including a storage unit storing a plurality of catalog information items in which catalog images representing interiors for decorating indoor spaces and category information representing categories of the catalog images are associated with each other, an image acquisition step of acquiring a room image; a segmentation step of identifying each interior area depicted in the room image and assigning category information of the interior to each of the areas; a category selection step of transmitting the category information assigned in the segmentation step to a terminal device and accepting a selection of the category information by a user from the terminal device; an interior design receiving step of transmitting one or more catalog images corresponding to the category information selected by the user to the terminal device and receiving a selection of the catalog image by the user from the terminal device; an image generating step of generating a renovation image by replacing the interior depicted in the room image to which the category information selected by the user has been assigned with the interior represented by the catalog image received in the interior receiving step, and displaying the image on the terminal device; A renovation image generation program that executes the above.

Images