Data output device, data output system, and program

Abstract

To provide a data output device and system, as well as a program, that can output an image which is obtained from an image of an office, in which the appearance of part of the interior structure in the image has been altered.SOLUTION: The data output device comprises an image acquisition unit that acquires an image of an office, and an interior structure recognition unit that recognizes interior structures, including one or more furniture, interior trims and fittings arranged in the office within the acquired image; an image generation unit that uses a machine learning-based image generation model to generate modified images with altered appearances, including one or more of the shape, pattern, and color of the interior structures, based on user requests; and an image output unit that outputs the generated modified images.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a data output device, a data output system, and a program, and specifically relates to a data output device, a data output system, and a program capable of outputting an image obtained by modifying the appearance of some spatial components in an office image from the office image.

Background Art

[0002] As a device that supports office layout design for employees and the like to work using an information processing system, for example, a layout support device disclosed in Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] When designing an office layout, the layout of the current office and the arrangement of fixtures are investigated, and a new layout is proposed while hearing requests from office employees. When proposing a layout using the prior art, layout data that can be handled by the information processing system is created using CAD (Computer Aided Design) or other software, and an office image obtained by rendering the created layout is presented. However, creating existing layout information as CAD or other data imposes a heavy burden on layout designers, and there has been a demand for more efficient support for layout proposal work.

[0005] On the other hand, when a layout is proposed, employees may request changes to the design. Furthermore, when proposing a layout, it may be necessary to present multiple variations with altered exteriors of fixtures, interior materials, and fittings. Therefore, the ability to efficiently handle changes in requests and create variations was essential.

[0006] In view of the above problems, the present invention aims to provide a data output device, a data output system, and a program that can output an image of an office from which the appearance of some spatial components in the image has been modified. [Means for solving the problem]

[0007] To solve the aforementioned problems, the data output device according to the present invention comprises: an image acquisition unit that acquires images of an office; a spatial configuration recognition unit that recognizes spatial components in the acquired images, including one or more furniture, interior components, and fixtures placed in the office; an image generation unit that uses an image generation model generated by machine learning to generate a modified image in which the appearance of the spatial components, including one or more shapes, patterns, and colors, is changed based on the user's request; and an image output unit that outputs the generated modified image. The system includes a target designation unit that receives text input from the user specifying which spatial components to be modified or which spatial components not to be modified, and specifying the degree of recognition of the spatial components. The image generation unit receives the specification of which spatial components to be modified or which spatial components not to be modified, and the specification of the degree of recognition of the spatial components, as a request from the user, and generates the modified image based on the request. It is characterized by the following.

[0008] The data output device according to the present invention, which was developed to solve the aforementioned problems, comprises: an image acquisition unit that acquires images of an office; a spatial configuration recognition unit that recognizes spatial components in the acquired images, including one or more furniture, interior components, and fixtures placed in the office; an image generation unit that uses an image generation model generated by machine learning to generate a modified image in which the appearance of the spatial components, including one or more shapes, patterns, and colors, is changed based on the user's request; an image output unit that outputs the generated modified image; and a target designation unit that accepts the user's text input to specify the degree of recognition of the spatial components, and the image generation unit is characterized in that it takes the specification of the degree of recognition of the spatial components as the user's request and generates the modified image based on that request.

[0009] The data output device according to the present invention has the target designation unit, The user's text input allows the system to specify which spatial components to be modified or which spatial components to remain unchanged. The image generation unit takes the designation received by the target designation unit as a request from the user. The modified image is generated based on the request. You may do so.

[0010] The data output device according to the present invention may further include a mask image generation unit that generates a mask image in which the spatial components whose appearance is changed or the spatial components other than those whose appearance is changed are masked in the acquired image.

[0011] The data output device according to the present invention further includes a depth map generation unit that generates a depth map showing the depth of the spatial components in the acquired image, and the image generation unit may generate the modified image based on the user's request and the depth map.

[0012] The data output device according to the present invention further includes a spatial taste specification unit that accepts the user's text input to specify a spatial taste including one or more of the color, material, or texture of the spatial components, and the image generation unit may take the specification received by the spatial taste specification unit as a request from the user and generate the modified image based on that request.

[0013] The data output device according to the present invention may further receive a specification of the degree of change to the appearance in the spatial taste specification unit, and the image generation unit may generate the modified image based on the user's request, with the spatial taste specification and the degree of change specification being the user's request.

[0014] The data output device according to the present invention may also be configured such that the spatial taste specification unit accepts an image input representing the spatial taste instead of the text input.

[0015] The data output device according to the present invention further includes a spatial configuration modification unit that changes the specified spatial configuration to a product specified by the user, wherein the target specification unit receives the specification of the spatial configuration to be changed by the user's text input, and the image generation unit changes the appearance of the specified spatial configuration to the appearance of the modified product.

[0016] A data output system according to the present invention made to solve the aforementioned problems includes a user terminal carried by a user, and a data output device comprising: an image acquisition unit that acquires an image of an office; a space component recognition unit that recognizes a space component including one or more of furniture, interior members, and fixtures arranged in the office in the acquired image; an image generation unit that generates a modified image in which the appearance including one or more of the shape, pattern, and color of the space component is changed based on a user's request, using an image generation model generated by machine learning; and an image output unit that outputs the generated modified image, wherein the image acquisition unit acquires the image of the office captured by an imaging device included in the user terminal, and the image output unit outputs the generated image to a display device included in the user terminal. Effects of the Invention

[0017] According to the configuration of the present invention, the data output device recognizes, by the space component recognition unit, a space component including one or more of furniture, interior members, and fixtures in the office image acquired by the image acquisition unit, generates, by the image generation unit, a modified image in which the appearance including one or more of the shape, pattern, and color of the recognized space component is changed based on a user's request, and outputs the modified image by the image output unit, so that it is possible to output, from the office image, an image in which the appearance of part of the space components in the image is modified. Brief Description of the Drawings

[0018] [Figure 1] It is a diagram showing the configuration of a data output system in an example of an embodiment of the present invention. [Figure 2] It is a flowchart showing the flow of data output processing in an example of the same embodiment. [Figure 3] It is a diagram showing the configuration of a data output screen in an example of the same embodiment. [Figure 4] It is a diagram showing an example of an office image in an example of the same embodiment. [Figure 5] It is a diagram showing an example of a mask image when the recognition level is specified to be low in an example of the same embodiment. [Figure 6] It is a diagram showing an example of a mask image when the recognition degree is strongly specified in an example of the same embodiment. [Figure 7] It is a diagram showing an example of a mask image for internal processing in an example of the same embodiment. [Figure 8] It is a diagram showing an example of a depth map in an example of the same embodiment. [Figure 9] It is a diagram showing a modified example of a mask image in an example of the same embodiment. [Figure 10] It is a diagram showing the configuration of a spatial taste selection dialog in an example of the same embodiment. [Figure 11] It is a diagram showing an example of an image representing a spatial taste in an example of the same embodiment. [Figure 12] It is a diagram showing an example of a modified image when the modification degree is weakly specified in an example of the same embodiment. [Figure 13] It is a diagram showing an example of a modified image when the modification degree is strongly specified in an example of the same embodiment. [Modes for Carrying Out the Invention]

[0019] Hereinafter, an example of the embodiment of the present invention will be described with reference to the drawings. Figure 1 is a diagram showing the configuration of a data output system 100 in an example of the present embodiment. As shown in Figure 1, the data output system 1 is an information processing system in which a data output device 200 and a user terminal 300 are communicably connected via a network 400.

[0020] The data output device 200 acquires an image of the office taken by a user terminal 300 (described later), and generates and outputs a modified image in which the appearance of spatial components specified by the user has been altered in the acquired image. In one example of this embodiment, the data output device 200 includes an image acquisition unit 201, a spatial configuration recognition unit 202, a spatial configuration recognition model 203, an image generation unit 204, an image generation model 205, an image output unit 206, a target designation unit 207, a mask image generation unit 208, a depth map generation unit 209, and a spatial taste designation unit 210. In this example of this embodiment, the office is a workspace where employees perform their duties, and may be, for example, one floor of a building, or a workspace enclosed by partitions.

[0021] The image acquisition unit 201 acquires images of the office. As described above, the data output system 100 in this example is an information processing system in which the data output device 200 and the user terminal 300 are connected to communicate via a network 400. In this example, as will be described later, the user terminal 300 is equipped with a camera 301, and the image acquisition unit 201 acquires images of the office taken by the camera 301 of the user terminal 300. Any well-known method may be arbitrarily selected for the specific method of acquiring images, but in this example, the user terminal 300 transmits the captured image to the data output device 200, the data output device 200 receives the transmitted image, and the image acquisition unit 201 acquires images of the office by recording them in a predetermined storage area.

[0022] The spatial configuration recognition unit 202 recognizes spatial components placed in the office in the image acquired by the aforementioned image acquisition unit 201. Spatial components are objects that make up the space of the office, and in one example of this embodiment, spatial components include one or more furniture such as chairs and desks, interior materials such as walls and floors, and building components such as doors and windows.

[0023] While any known recognition method can be arbitrarily selected for recognizing spatial components from an image of an office, in this embodiment, the data output device 200 uses a spatial configuration recognition model 203, which is a trained model generated by machine learning, to generate information indicating spatial components in the image of the office from the image of the office acquired by the aforementioned image acquisition unit 201 and the information indicating the designation received by the target designation unit 207, which will be described later.

[0024] The image generation unit 204 uses an image generation model 205, which is a trained model generated by machine learning, to generate a modified image that changes the appearance of the spatial components, including one or more of the shape, pattern, and color, based on the user's request. In one example of this embodiment, the user's request is received by the target designation unit 207 and the spatial taste designation unit 210, which will be described later.

[0025] The specific format of the user's request and the timing of its processing can be arbitrarily selected. For example, information indicating the user's request may be used as direct input data for the modified image generation process by the image generation unit 204, or the image generation unit 204 may generate the modified image using the results of intermediate processing using information indicating the user's request as input data.

[0026] Furthermore, the specific configuration of the image generation model 205 may be arbitrarily changed based on, for example, the format and timing of receiving the user request information mentioned above. In one example of this embodiment, a modified image is generated with altered appearance of spatial components from the image of the office acquired by the image acquisition unit 201, the mask image generated by the mask image generation unit 208 (described later), the spatial taste specification information received by the spatial taste specification unit 209 (described later), and the depth map generated by the depth map generation unit 209 (described later).

[0027] The image output unit 206 outputs the modified image generated by the image generation unit 204. In one example of this embodiment, as described above, the data output device 200 and the user terminal 300 are connected to each other via the network 400, and the image output unit 208 outputs the modified image to the display device 302 of the user terminal 300.

[0028] The target designation unit 207 accepts the user's text input to specify spatial components whose appearance should be changed or not changed in the image of the office acquired by the image acquisition unit 201. As described above, in this example of the embodiment, the image generation unit 204 generates a modified image with the appearance of the spatial components changed based on the user's request, but the designation of spatial components whose appearance should be changed or not changed, received by the target designation unit 207, is treated as the user's request. Whether the target designation unit 207 accepts the designation of spatial components whose appearance should be changed or not can be arbitrarily selected based on the specifications of the spatial configuration recognition unit 202, etc., and in this example of the embodiment, the target designation unit 207 accepts the designation of spatial components whose appearance should be changed.

[0029] The mask image generation unit 208 generates a mask image from the image acquired by the image acquisition unit 201, which masks the spatial components whose appearance is to be changed, or the spatial components other than those whose appearance is to be changed. In one example of this embodiment, the mask image is displayed on the display device 302 of the user terminal 300, and is also used as input data for the image generation unit 204 as information specifying the range in the office image whose appearance is to be changed.

[0030] The depth map generation unit 209 generates a depth map showing the depth of spatial components in the image acquired by the image acquisition unit 201. The generated depth map 209 is used by the image generation unit 204 to generate a modified image along with the user's request as described above. The specific method for generating the depth map can be arbitrarily selected. For example, a trained model generated by machine learning may be used, and the image of the office acquired by the image acquisition unit 201 may be used as input data. The depth map may be configured to output a bitmap image representing the relative distance between each pixel constituting the office image (the input data) and the imaging device 301 that captured the office image, expressed by color intensity.

[0031] The spatial taste specification unit 210 accepts the specification of a spatial taste by text input from the user. In this example of the embodiment, the spatial taste is information that includes one or more of the color, material, or texture of the spatial components. As described above, in this example of the embodiment, the image generation unit 204 generates a modified image in which the appearance of the spatial components has been changed based on the user's request, but the spatial taste specification received by the spatial taste specification unit 210 is treated as the user's request.

[0032] In one example of this embodiment, the data output device 100 is configured using a server computer. A program that enables the server computer to function as the data output device 100 is stored in advance on the server computer's HDD (Hard Disk Drive), SSD (Solid State Drive), or other storage device. The server computer is then loaded into memory and executed by the CPU (Central Processing Unit) to enable it to function as the data output device 100. The configuration of the data output device 100 can be arbitrarily changed; for example, the data output device 100 may be configured using two or more computers.

[0033] Furthermore, in this example, the system includes a spatial configuration recognition model 203 and an image generation model 205 as trained models generated by machine learning. However, if a separate information processing system exists that provides similar functionality to the trained models, the data output system 100 may be configured using that information processing system. For example, if a separate spatial configuration recognition system exists that provides functionality equivalent to the spatial configuration recognition model 203 in this example, the spatial configuration recognition unit 202 may be configured to use the spatial configuration recognition system as an external linkage system or as a subsystem constituting the data output system 100, instead of the spatial configuration recognition model 203. Similarly, with respect to the image generation model 205, if a separate information system exists that provides similar functionality to the image generation model 205, the image generation unit 204 may be configured to use the separate information system as an external linkage system or subsystem, instead of the image generation model 205.

[0034] The user terminal 300 is a terminal used by users of the data output system 100. In one example of this embodiment, it includes a camera 301 for photographing the office and a display or other display device 302. The user terminal 300 may be configured using a smartphone, tablet computer or other portable terminal, or a stationary personal computer or other well-known computer. Furthermore, although the user terminal 300 in this example of this embodiment is equipped with a camera 301, the user terminal 300 may be configured by combining a digital camera and a well-known computer, and the digital camera may be treated as the camera 301.

[0035] Network 400 is a computer network that connects the data output device 200 and the user terminal 300 in a communication-enabled manner. Network 400 may use any well-known computer network, such as the Internet or other wide-area networks, or a LAN (Local Area Network), or a combination of a wide-area network and a LAN to constitute Network 400. Alternatively, a VPN (Virtual Private Network) may be built on a wide-area network and used as Network 400. Furthermore, Network 400 may be a wired network, a wireless network, or a combination of a wired network and a wireless network.

[0036] The above describes the configuration of the data output system 1 in this example. Next, the data output flow in this example will be explained. Figure 2 is a flowchart showing the data output flow in this example, and Figure 3 is a diagram showing the configuration of the data output screen W1 in this example.

[0037] In one example of this embodiment, the data output device 200 displays the data output screen W1 on the display device 302 of the user terminal 300, and the user operates the user terminal 300 to generate a modified image in which the appearance of an arbitrary spatial component has been changed from an image of the office. As shown in Figure 3, the data output screen W1 includes an office image selection area W11, a target selection form W12, a recognition degree adjustment slider W13, a mask image creation button W14, a mask image area W15, a spatial taste selection form W16, a change degree adjustment slider W17, an image generation button W18, and a modified image area W19. In this example of this embodiment, the vertically oriented data output screen W1 includes the office image selection area W11 to the modified image area W19, but the screen configuration for the data output flow can be arbitrarily changed, for example, by configuring the system to perform data output while transitioning between multiple operation / viewing screens.

[0038] In this example, data output first involves the user specifying an image of the office whose appearance of the spatial components will be changed (step S1). As described above, in this example, the data output screen W1 includes an office image specification area W11. When the user presses the office image specification area W11, a dialog box (not shown) for selecting a predetermined specification method is displayed on the display device 302 of the user terminal 300. In this example, the dialog box for selecting the specification method displays options such as "Specify from photo library," "Take a new photo of the office," and "Specify from file," and the user selects an arbitrary specification method from these options to specify the office image. If "Specify from photo library" is selected, the user specifies an office image from images already taken by the user terminal 300. If "Take a new photo of the office" is selected, the user takes a new photo of the office using the shooting device 301 and specifies the captured image. If "Specify from file" is selected, the user can specify any file that is not stored in the photo library. The method for specifying the office image can be chosen arbitrarily. For example, when the user presses the office image selection area W11, the system may transition to a screen where the user terminal 300's camera 301 takes an image of the office, and the user can then specify the office image using the newly taken image.

[0039] When an office image is specified, the image acquisition unit 201 of the data output device 200 acquires the specified office image. The method and timing of acquiring the office image can be arbitrarily selected. In one example of this embodiment, when an office image is specified in the office image specification area W11, the image acquisition unit 201 of the data output device 200 records the office image specified by the user terminal 300 into a predetermined storage area of ​​the data output device 200.

[0040] Figure 4 shows an example of an office image in one example of this embodiment. In the example shown in Figure 5, the office shown in the image is furnished with furniture such as chairs and desks, interior materials such as walls and floors, and building materials such as doors and windows.

[0041] Once the office image has been specified, the user then specifies the spatial components whose appearances to change (step S2). The target specification form W12 on the data output screen W1 is a form in which the user specifies, by text input, which spatial components in the office image specified in the office image specification area W11 to change the appearance of. Below the target specification form W12, an example of text input W12a is displayed. In step S2, for example, if the user enters "chair" as text in the target specification form W12, the chair in the office image will be specified as the spatial component whose appearance will be changed. In this example, the user can also specify that the appearances of two or more spatial components will be changed by concatenating multiple spatial components with commas as text input. For example, if the user enters "chair,floor" as text in the target specification form W12, the chair and floor in the office image will be specified as the spatial components whose appearance will be changed. When a user enters text into the target designation form W12, the entered text is transmitted from the user terminal 300 to the data output device 200, and the instruction is received by the target designation unit 207 of the data output device 200. The specific method for receiving the instruction can be arbitrarily selected; for example, the target designation unit 207 may record text information indicating the instruction in the memory or other primary storage device of the data output device 200, or, if the data output screen W1 is configured as a web screen, it may be recorded on the user terminal 300 using an HTTP cookie or other method.

[0042] In one example of this embodiment, as described above, the target designation unit 207 accepts the designation of spatial components whose appearance is to be changed, as well as the designation of the degree to which the spatial components are recognized by the spatial component recognition unit 202. Regarding the designation of spatial components whose appearance is to be changed, the user specifies the degree to which the spatial components are recognized (step S3). The recognition degree adjustment slider W13 provided on the data output screen W1 specifies the degree to which spatial components are recognized in the office image specified in the image designation area W11. In one example of this embodiment, moving the slider to the left end specifies a weaker recognition degree, and moving it to the right end specifies a stronger recognition degree. When the user moves the recognition degree adjustment slider W13 left or right to specify the recognition degree, information indicating this designation is received by the target designation unit 207 of the data output device 200.

[0043] Once the office image has been specified in step S1, the spatial elements whose appearance will be changed in step S2, and the degree of recognition has been specified in step S3, the user presses the mask image creation button W14 on the data output screen W1. The data output device 200 then uses the spatial element recognition unit 202 to recognize the spatial elements in the office image (step S4), the mask image generation unit 208 to generate a mask image (step S5), and the depth map generation unit 209 to generate a depth map (step S6).

[0044] Figure 5 shows an example of a mask image in this embodiment when the recognition level is specified as weak, and Figure 6 shows an example of a mask image in this embodiment when the recognition level is specified as strong. Both Figure 5 and Figure 6 are mask images generated when the spatial components whose appearance is to be changed are specified as "chair,desk,floor,wall,ceil" by text input for the office image shown in Figure 4, and are images in which the spatial components whose appearance is to be changed are masked with predetermined colors in the office image shown in Figure 4. In the examples of Figures 6 and 7, when the recognition level is specified as weak, the plant near the center of the right edge and the storage shelf near the center of the left edge are not recognized as spatial components in Figure 5, but are recognized as spatial components in Figure 6. In this embodiment, the mask image masks the spatial components whose appearance is to be changed with predetermined colors, but how the mask is represented can be arbitrarily changed, and an image in which everything other than the spatial components whose appearance is to be changed is masked with predetermined colors may also be used. Furthermore, the specific format of the mask image can be arbitrarily selected. For example, a bitmap-format mask image may be generated, or information indicating the mask area in the office image may be generated, and the mask image may be constructed using this information and the office image.

[0045] In this embodiment, as described above, the data output device 200 includes a spatial configuration recognition unit 202 and a mask image generation unit 208. However, the combination of the spatial configuration recognition unit 202 and the mask image generation unit 208 to generate the mask image can be arbitrarily selected. For example, the spatial configuration recognition unit 202 may recognize all spatial components in the office image based on a specified office image and recognition level, and then the mask image generation unit 208 may generate a mask image based on the processing result of the spatial configuration recognition unit 202 and the specification of spatial components whose appearance has been changed. Alternatively, the spatial configuration recognition unit 202 may recognize only the spatial components whose appearance has been changed in the office image based on the image and recognition level, as well as the specification of spatial components whose appearance has been changed, and the mask image may be converted from the processing result of the spatial configuration recognition unit 202 to a predetermined mask image. Furthermore, as described above, if the spatial configuration recognition model 203 can directly output a mask image, the spatial configuration recognition unit 202 may also function as the mask image generation unit 208.

[0046] In step S5 of this embodiment, along with the aforementioned mask image, a mask image for internal processing is generated. Figure 7 shows an example of a mask image for internal processing in this embodiment. The mask images shown in Figures 5 and 6 are images displayed on the data output screen W1, as will be described later. These images mask the area in the office image where the appearance is changed, making them easy for the user to see. The mask image for internal processing, however, is an image that becomes input data for the image generation process described later. As shown in Figure 7, it is an image that shows the area in the office image where the appearance is changed and the area where the appearance is not changed as a binary bitmap. The type of image to generate as a mask image can be arbitrarily selected. For example, only images that are easy for the user to see, as shown in Figures 5 and 6, may be generated, or only mask images for internal processing, as shown in Figure 7, may be generated.

[0047] Figure 8 shows an example of a depth map in one example of this embodiment. The depth map shown in Figure 8 is a depth map generated by the depth map generation unit 209 of the data output device when the office image shown in Figure 5 is specified in the office screen specification area W11. In this example of this embodiment, the depth map represents the depth in the specified office image, that is, the relative distance of each pixel constituting the office image to the imaging device 301 that captured the office image, expressed in color intensity. In this example of this embodiment, the depth map is used as input data for the image generation model 205 as information that maintains the field of view of the office image and the positional relationships of spatial components when generating the modified image. Although this example of this embodiment uses a depth map, instead of a depth map, or in addition to a depth map, an image in which the contours of spatial components in the office image are extracted, or a segmentation map in which the office image is divided into multiple regions according to spatial components may be used. Furthermore, the format of the depth map can be arbitrarily selected. For example, the depth map may be constructed as a bitmap image that represents depth in grayscale, or it may be constructed using information that numerically indicates the depth for each pixel of the office image.

[0048] Once steps S4-6 are completed, the data output device 200 selectively displays the generated mask image and depth map in the mask screen area W15 of the data output screen W1 (step S7). A tab W15a for selecting the image to display is provided at the top of the mask image area W15, and by selecting this tab, the selected image is displayed in the mask image area W15.

[0049] In one example of this embodiment, the user can modify the mask range of the mask image displayed in the mask image area W15 by operating the user terminal 300. Figure 9 shows an example of mask image modification in one example of this embodiment, where the area represented by the diagonal lines in Figure 9 is the range in which the user modified the mask. In one example of this embodiment, as described above, the mask image generated by the mask image generation unit 208 is displayed in the mask image area W15. In this displayed state, the user can modify the mask image by operating the user terminal 300. The specific method of modification can be arbitrarily selected. For example, if the user selects an arbitrary range in the mask image displayed in the mask image area W11, a dialog box may be displayed on the display device 302 of the user terminal 300 to select whether to add or delete the mask in the selected range. If the user selects to add or delete from the displayed dialog box, the selected range may be masked or the selected range may be removed from the mask. Furthermore, in one example of this embodiment, the data processing screen W1 includes a target selection form W12, a recognition degree adjustment slider W13, a mask image creation button W13, and a mask image area W15. In addition to the above modification method, when the mask image is displayed in step S7, the user can change the spatial object whose appearance is to be modified using the target selection form W12, and / or change the degree of recognition using the recognition degree adjustment slider W13, and then press the mask image creation button W13 again, thereby repeating the recognition of the spatial object and the generation of the mask image.

[0050] In this embodiment, as described above, a mask image for internal processing is generated along with the mask image displayed on the user terminal 300. However, the timing of generating the internal processing mask image and modifying the mask range can be arbitrarily changed. For example, the mask image generation unit 208 may generate an internal processing mask image that reflects the modification each time the user modifies the mask range. Alternatively, the internal processing mask image may not be generated at the time the mask image displayed on the user terminal 300 is generated, and only after the user has finished modifying the mask image may the internal processing mask image that reflects the modification be generated.

[0051] Once the generation of the mask image and depth map is complete, the user then specifies the spatial taste (step S8). The spatial taste specification form W16 on the data output screen W1 is a form for specifying a spatial taste that includes one or more of the color, material, or texture of the spatial components by text input. Below the spatial taste specification form W16, a text input example W16a is displayed. For example, if "office, modern" is specified, the appearance of the specified spatial components in the office image will be changed to the appearance of modern office furniture. When text is entered into the spatial taste specification form W16, the spatial taste specification unit 210 of the data output device 200 receives the spatial taste instruction from the text input. The method of receiving the instruction can be arbitrarily selected; it may be recorded in the memory or other primary storage device of the data output device 200, or it may be recorded on the user terminal 300 via HTTP Cookie or the like.

[0052] Furthermore, in one example of this embodiment, the spatial taste specification unit 210 of the data output device 200 accepts a specification of the degree of change in appearance in addition to the specification of the spatial taste described above (step S9). The change degree adjustment slider W17 provided on the data output screen W1 specifies the degree to which the appearance of the spatial components specified in the target specification form W12 described above will be changed. In one example of this embodiment, moving the slider to the left end specifies a weaker change, and moving it to the right end specifies a stronger change. When the user moves the change degree adjustment slider W17 left or right to specify a change, information indicating that specification is received by the spatial taste specification unit 210 of the data output device 200.

[0053] Furthermore, the method for specifying the spatial style and the degree of change is at the user's discretion. For example, instead of, or in addition to, text input, multiple selectable spatial styles may be provided in advance, allowing the user to choose any spatial style. Figure 10 shows an example of a spatial style selection dialog W2. In the example shown in Figure 10, the spatial style selection dialog W2 displays an image W21 showing an example of a spatial structure to which the spatial style has been applied, the colors included in the spatial style W22, and the spatial style name W23. The user can select any of the spatial styles, and that selected spatial style will be treated as the specified style.

[0054] Furthermore, instead of, or in addition to, text input for specifying the spatial taste, the system may also accept input of an image that represents the spatial taste. Figure 11 shows an example of an image that represents a spatial taste. When the image shown in Figure 11 or other images are specified as images that represent a spatial taste, the spatial taste specification unit 210 of the data output device 200 may be configured to treat the specified spatial taste as including one or more of the color, material, or texture of the spatial components in the specified image. Moreover, when both text input and image-based spatial taste specification are possible, the system may be configured to allow specifying the weighting of the spatial taste specified by text input and the spatial taste specified by image, instead of, or in addition to, the change degree adjustment slider W17 on the data output screen W1. The interface for specifying the weighting may be configured similarly to the change adjustment slider W17. For example, when the slider is moved to the left end, the weighting of text input is set higher, and when it is moved to the right end, the weighting of images is set higher. Based on the specified weightings, the system may be configured to generate a modified image with the spatial taste specified by the text input and the spatial taste specified by the image as requests from the user.

[0055] Once the spatial taste specification in step S8 and the degree of change specification in step S9 are complete, when the user presses the image generation button W18 on the data output screen W1, the data output device 200 generates a modified image using the image generation unit 204 (step S10). As described above, in this example of the embodiment, the image generation unit 204 of the data output device 200 generates a modified image in which the appearance of the specified spatial components is changed using the image generation model 205. The image generation model 205 uses the image of the office acquired by the image acquisition unit 201, the mask image for internal processing generated by the mask image generation unit 208, the spatial taste specification information received by the spatial taste specification unit 209, and the depth map generated by the depth map generation unit 209 to generate a modified image in which the appearance of the spatial components in the area specified by the mask image in the office image is changed to the appearance of the color, material, or texture included in the specified spatial taste. The modified image generated by the image generation unit 204 is displayed in the modified image area W19 by the image output unit 205 (step S11).

[0056] Figure 12 shows an example of a modified image in an example of this embodiment when the degree of change is specified to be weak, and Figure 13 shows an example of a modified image in an example of this embodiment when the degree of change is specified to be strong. Both Figures 12 and 13 are modified images generated when the spatial components whose appearance is to be changed are specified as "chair,desk,floor,wall,ceil" by text input, and the spatial style is specified as "office,modern style" by text input, for the office image shown in Figure 4. In the office image, the specified spatial components are changed in one or more of the following ways: color, material, or texture, depending on the degree of change of the spatial style.

[0057] In one example of this embodiment, the data output screen W1 includes a spatial taste specification form W16, a change degree adjustment slider W17, and an image generation button W18. Once the modified image is displayed in step S11, the user can change the spatial taste using the spatial taste specification form W16 and / or change the degree of change using the change degree adjustment slider W17, and then press the image generation button W18 to repeatedly generate and display the modified image.

[0058] This concludes the description of one example of this embodiment. However, the embodiments of the present invention are not limited to those described above. For example, in this example of this embodiment, the appearance of the spatial component designated by the target designation unit 207 is changed according to the spatial taste designated by the spatial taste designation unit 209. However, instead of the spatial taste designation unit 209, or in addition to it, a spatial configuration modification unit may be provided that changes the designated spatial component to a product designated by the user, and the image generation unit 204 may be configured to change the appearance of the spatial component designated by the target designation unit 207 to the appearance of a product designated by the spatial component modification unit.

[0059] Furthermore, in one example of this embodiment, a modified image is generated with a changed appearance of a specified spatial component based on an image of the office taken by the camera 301 of the user terminal 300. However, instead of the above-mentioned image of the office, a modified image with a changed appearance of a specified spatial component may be generated from a 2D or 3D CAD (Computer Aided Design) file. In this case, the data output device 200 may be equipped with a rendering unit that renders the CAD file, and the image acquisition unit 201 may be configured to acquire the image rendered by the rendering unit as an image of the office. Alternatively, the image acquisition unit 201 may be replaced with a CAD file acquisition unit, and the spatial configuration recognition unit 202 and the mask image generation unit 208 may be configured to generate a mask image from the CAD file acquired by the CAD file acquisition unit and the specification received by the target specification unit 207.

[0060] Other specific configurations are not limited to this embodiment, and various modifications are possible without departing from the spirit of the present invention. [Explanation of Symbols]

[0061] 100 Data Output Systems 200 Data Output Devices 201 Image Acquisition Unit 202 Spatial configuration recognition unit 203 Spatial Configuration Recognition Model 204 Image generation unit 205 Image Generation Models 206 Image Output Unit 207 Designated Target Section 208 Mask Image Generation Unit 209 Depth Map Generation Unit 210 Space Taste Specification Section 300 user terminals 301 Imaging device 302 Display device 400 Networks

Claims

1. Image acquisition unit that acquires images of the office, A spatial configuration recognition unit recognizes a spatial configuration in the acquired image that includes one or more furniture, interior components, and fixtures placed in the office, An image generation unit that uses an image generation model generated by machine learning to generate a modified image in which the appearance of the spatial component is changed, including one or more of the shape, pattern, and color, based on the user's request. An image output unit that outputs the generated modified image, The system includes a target designation unit that accepts the user's text input to specify which spatial components to be modified or which spatial components not to be modified, and to specify the degree of recognition of the spatial components. The image generation unit receives the user's request to specify the spatial components to be modified or not to be modified, and the degree to which the spatial components should be recognized, and generates the modified image based on the request. Data output device.

2. An image acquisition unit that acquires images of the office, A spatial configuration recognition unit recognizes a spatial configuration in the acquired image that includes one or more furniture, interior components, and fixtures placed in the office, An image generation unit that uses an image generation model generated by machine learning to generate a modified image in which the appearance of the spatial component is changed, including one or more of the shape, pattern, and color, based on the user's request. An image output unit that outputs the generated modified image, The system includes a target designation unit that accepts the user's text input to specify the degree of recognition of the spatial components, The image generation unit takes the user's request to specify the degree of recognition of the spatial components and generates the modified image based on that request. Data output device.

3. The target designation unit receives the designation of the spatial components to be changed or the spatial components not to be changed by the user's text input, The image generation unit takes the designation received by the target designation unit as a request from the user. Based on the request, generate the modified image. The data output device according to claim 2.

4. The data output device further, The system includes a mask image generation unit that generates a mask image in which the spatial components whose appearance is changed or components other than those whose appearance is changed are masked in the acquired image. The data output device according to claim 1 or 2.

5. The data output device further, The system includes a depth map generation unit that generates a depth map showing the depth of the spatial components in the acquired image, The image generation unit generates the modified image based on the user's request and the depth map. The data output device according to claim 1 or 2.

6. The data output device further, The system includes a space taste specification unit that accepts the user's text input to specify a space taste that includes one or more of the color, material, or texture of the spatial components, The image generation unit takes the specification received by the spatial taste specification unit as a request from the user and generates the modified image based on the request. The data output device according to claim 1 or 2.

7. The aforementioned spatial taste specification unit further accepts the specification of the degree of change to the exterior appearance, The image generation unit takes the specification of the spatial taste and the specification of the degree of change as requests from the user and generates the modified image based on said requests. The data output device according to claim 6.

8. The spatial taste specification unit accepts an image representing the spatial taste as input instead of the text input. The data output device according to claim 6.

9. The data output device further includes a spatial configuration changing unit that changes the specified spatial configuration to a product specified by the user. The aforementioned target designation unit receives the designation of the spatial structure to be modified by the user's text input, The image generation unit changes the appearance of the specified spatial component to the appearance of the modified product. The data output device according to claim 1.

10. The user terminal owned by the user and the data output device described in claim 1 or 2 are provided. The image acquisition unit acquires images of the office taken by the camera device installed in the user terminal. The image output unit outputs the generated image to the display device provided by the user terminal. Data output system.

11. A program that causes a computer to function as a data output device according to claim 1 or 2.

Images