Data output device, data output system, and program

Abstract

To provide a data output device and system, as well as a program, that can output an image which is obtained from an image of an office, in which the appearance of part of the interior structure in the image has been altered.SOLUTION: The data output device comprises an image acquisition unit that acquires an image of an office, and an interior structure recognition unit that recognizes interior structures, including one or more furniture, interior trims and fittings arranged in the office within the acquired image; an image generation unit that uses a machine learning-based image generation model to generate modified images with altered appearances, including one or more of the shape, pattern, and color of the interior structures, based on user requests; and an image output unit that outputs the generated modified images.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a data output device, a data output system, and a program, and more particularly to a data output device, a data output system, and a program that can output an image of an office in which the appearance of some of the spatial components in the image has been changed. [Background technology]

[0002] BACKGROUND ART A layout support device disclosed in Patent Document 1, for example, is known as a device that uses an information processing system to support the layout design of an office where employees and the like work. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-189526 Summary of the Invention [Problem to be solved by the invention]

[0004] When designing an office layout, it is common to survey the current office layout and furniture placement, and then propose a new layout while listening to the needs of office employees. When proposing a layout using conventional technology, layout data that can be handled by an information processing system is created using CAD (Computer Aided Design) or other software, and an image of the office rendered from the created layout is presented. However, creating existing layout information as CAD or other data is a significant burden for layout designers, and there has been a demand for more efficient support for the layout proposal process.

[0005] However, when a layout is proposed, employees may change their requests. Also, when proposing a layout, it may be necessary to present multiple variations with different appearances of fixtures, interior materials, and fittings, etc., and there is a need for a system that can efficiently handle changes in requests and the creation of variations.

[0006] In view of the above problems, the present invention aims to provide a data output device, a data output system, and a program that can output an image of an office in which the appearance of some of the spatial components in the image has been changed. [Means for solving the problem]

[0007] The data output device of the present invention, which has been made to solve the above-mentioned problems, is characterized by comprising an image acquisition unit that acquires an image of an office, a spatial structure recognition unit that recognizes spatial structures in the acquired image, including one or more of fixtures, interior components, and fittings arranged in the office, an image generation unit that uses an image generation model generated by machine learning to generate a modified image that modifies the appearance of the spatial structures, including one or more of the shape, pattern, and color, based on a user's request, and an image output unit that outputs the generated modified image.

[0008] The data output device of the present invention may further include a target designation unit that accepts designation of the spatial structure to be changed or the spatial structure to be left unchanged through text input by the user, and the image generation unit may take the designation accepted by the target designation unit as a request from the user and generate the changed image based on the request.

[0009] In the data output device of the present invention, the target designation unit may further receive a designation of the degree of recognition of the spatial structure, and the image generation unit may generate the modified image in response to a request from the user, which designates the spatial structure to be changed or the structure not to be changed, and the designation of the degree of recognition.

[0010] The data output device according to the present invention may further include a mask image generation unit that generates a mask image in which the spatial structure that changes the appearance in the acquired image or a mask image other than the spatial structure that changes the appearance is masked.

[0011] The data output device of the present invention may further include a depth map generation unit that generates a depth map indicating the depth of the spatial structure in the acquired image, and the image generation unit may generate the modified image based on the user's request and the depth map.

[0012] The data output device of the present invention may further include a space taste specification unit that accepts a specification of a space taste including one or more of the color, material, or texture of the spatial component by text input from the user, and the image generation unit may take the specification accepted by the space taste specification unit as a request from the user and generate the modified image based on the request.

[0013] In the data output device of the present invention, the spatial taste designation unit may further accept a designation of the degree of change in the appearance, and the image generation unit may generate the modified image based on the user's request, with the designation of the spatial taste and the designation of the degree of change being the user's request.

[0014] In the data output device according to the present invention, the spatial taste designation unit may accept input of an image indicating the spatial taste instead of the text input.

[0015] The data output device of the present invention may further include a spatial structure modification unit that modifies the specified spatial structure to a product specified by the user, wherein the target designation unit accepts the designation of the spatial structure to be modified by text input by the user, and the image generation unit modifies the appearance of the specified spatial structure to the appearance of the modified product.

[0016] The data output system of the present invention, which has been made to solve the above-mentioned problems, comprises a data output device that includes a user terminal carried by a user, an image acquisition unit that acquires images of the office, a spatial structure recognition unit that recognizes spatial structures in the acquired images, including one or more of fixtures, interior materials, and fittings placed in the office, an image generation unit that uses an image generation model generated by machine learning to generate a modified image that modifies the appearance of the spatial structures, including one or more of the shape, pattern, and color, based on the user's request, and an image output unit that outputs the generated modified image, wherein the image acquisition unit acquires images of the office photographed by a photographing device provided in the user terminal, and the image output unit outputs the generated image to a display device provided in the user terminal. [Effects of the Invention]

[0017] According to the configuration of the present invention, the data output device recognizes spatial structures, including one or more of fixtures, interior materials, and fittings, in an image of an office acquired by an image acquisition unit using a spatial structure recognition unit, and generates a modified image of the recognized spatial structure in which the appearance, including one or more of shape, pattern, and color, is modified based on the user's request using an image generation unit, and outputs the modified image using an image output unit, so that it is possible to output an image from an office image in which the appearance of some of the spatial structures in the image has been modified. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram showing a configuration of a data output system according to an example embodiment of the present invention. [Figure 2] FIG. 10 is a flowchart showing the flow of a data output process in one example of the embodiment. [Figure 3] FIG. 10 is a diagram showing the configuration of a data output screen in an example of the same embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of an image of an office according to the embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a mask image when a weak degree of recognition is specified in the example of the embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a mask image when a strong degree of recognition is specified in the example of the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a mask image for internal processing in the example of the same embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a depth map according to the embodiment. [Figure 9] FIG. 10 is a diagram showing an example of correction of a mask image in the example of the embodiment. [Figure 10] FIG. 10 is a diagram showing the configuration of a space taste selection dialog in an example of the same embodiment. [Figure 11] FIG. 10 is a diagram showing an example of an image showing a spatial taste in the example of the embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a modified image when a weak degree of modification is specified in the example of the embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a modified image when a strong degree of modification is specified in the example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a diagram showing the configuration of a data output system 100 in this example of the present embodiment. As shown in Fig. 1, the data output system 1 is an information processing system in which a data output device 200 and a user terminal 300 are communicably connected via a network 400.

[0020] The data output device 200 is a device that acquires an image of an office taken by a user terminal 300 (described later), and generates and outputs a modified image in which the appearance of a spatial component specified by the user is modified in the acquired image. In one example of this embodiment, the data output device 200 includes an image acquisition unit 201, a spatial configuration recognition unit 202, a spatial configuration recognition model 203, an image generation unit 204, an image generation model 205, an image output unit 206, an object designation unit 207, a mask image generation unit 208, a depth map generation unit 209, and a spatial taste designation unit 210. Note that in this example of this embodiment, the office is a workspace where employees and the like perform their work, and may be, for example, one floor of a building or a workspace enclosed by partitions.

[0021] The image acquisition unit 201 acquires images of the office. As described above, the data output system 100 in one example of this embodiment is an information processing system in which the data output device 200 and the user terminal 300 are communicably connected via the network 400. In one example of this embodiment, the user terminal 300 is equipped with a photographing device 301, as will be described later, and the image acquisition unit 201 acquires images of the office captured by the photographing device 301 of the user terminal 300. While any known method may be selected as the specific method for acquiring images, in this example of this embodiment, the user terminal 300 is configured to transmit the captured images to the data output device 200, the data output device 200 receives the transmitted images, and the image acquisition unit 201 records the images in a predetermined storage area, thereby acquiring the images of the office.

[0022] The spatial configuration recognition unit 202 recognizes spatial components arranged in the office in the images acquired by the image acquisition unit 201. Spatial components are objects that make up the office space, and in one example of this embodiment, the spatial components include one or more of furniture such as chairs and desks, interior components such as walls and floors, and fittings such as doors and windows.

[0023] Any known recognition method may be selected as the method for recognizing spatial components from an image of an office, but in one example of this embodiment, the data output device 200 uses a spatial configuration recognition model 203, which is a trained model generated by machine learning, to generate information indicating spatial components in the image of the office from the image of the office acquired by the image acquisition unit 201 described above and information indicating the designation received by the target designation unit 207 described below.

[0024] The image generation unit 204 uses an image generation model 205, which is a trained model generated by machine learning, to generate a modified image in which the appearance of the spatial component, including one or more of the shape, pattern, and color, is modified based on a user's request. In one example of this embodiment, the user's request is received by a target designation unit 207 and a space taste designation unit 210, which will be described later.

[0025] The specific format of the user's request and the timing of processing may be selected arbitrarily. For example, the information indicating the user's request may be used as direct input data for the modified image generation process by the image generation unit 204, or the image generation unit 204 may generate the modified image using the results of intermediate processing using the information indicating the user's request as input data.

[0026] The specific configuration of the image generation model 205 may also be changed arbitrarily based on, for example, the format of the information indicating the user's request, the timing of receiving it, etc. In one example of this embodiment, a modified image in which the appearance of a spatial component is modified is generated from an office image acquired by the image acquisition unit 201, a mask image generated by a mask image generation unit 208 (described later), information indicating the designation of spatial taste received by a spatial taste designation unit 209 (described later), and a depth map generated by a depth map generation unit 209 (described later).

[0027] The image output unit 206 outputs the modified image generated by the image generation unit 204. In one example of this embodiment, as described above, the data output device 200 and the user terminal 300 are communicably connected via the network 400, and the image output unit 208 outputs the modified image to the display device 302 of the user terminal 300.

[0028] The target designation unit 207 accepts designation of a spatial component whose appearance is to be changed or whose appearance is not to be changed by text input by the user in the image of the office acquired by the image acquisition unit 201. As described above, in one example of this embodiment, the image generation unit 204 generates a modified image in which the appearance of the spatial component is changed based on a user request, but the designation of a spatial component whose appearance is to be changed or a spatial designated object whose appearance is not to be changed, accepted by the target designation unit 207, is treated as the user request. Whether the target designation unit 207 accepts the designation of a spatial component whose appearance is to be changed or a spatial component whose appearance is not to be changed can be arbitrarily selected based on the specifications of the spatial configuration recognition unit 202, etc., and in one example of this embodiment, the target designation unit 207 accepts the designation of a spatial component whose appearance is to be changed.

[0029] The mask image generation unit 208 generates a mask image in which the spatial components whose appearance is to be changed or the spatial components other than the spatial components whose appearance is to be changed are masked in the image acquired by the image acquisition unit 201. In one example of this embodiment, the mask image is displayed on the display device 302 of the user terminal 300, and is also used as input data for the image generation unit 204 as information specifying the range in the image of the office whose appearance is to be changed.

[0030] The depth map generation unit 209 generates a depth map indicating the depth of spatial components in the image acquired by the image acquisition unit 201. The generated depth map 209 is used by the image generation unit 204 to generate a modified image together with the above-mentioned user request. Note that a specific method for generating the depth map may be selected arbitrarily. For example, a trained model generated by machine learning may be used to output, as the depth map, a bitmap image in which the relative distance between each pixel constituting the input data office image acquired by the image acquisition unit 201 and the image capture device 301 that captured the office image is represented by color intensity, using the trained model.

[0031] The space taste designation unit 210 accepts the designation of space taste through text input by the user. In one example of this embodiment, the space taste is information including one or more of the color, material, and texture of the space components. As described above, in one example of this embodiment, the image generation unit 204 generates a modified image in which the appearance of the space components is modified based on the user's request, and the designation of space taste accepted by the space taste designation unit 210 is treated as the user's request.

[0032] In one example of this embodiment, the data output device 100 is configured using a server computer. A program that causes the server computer to function as the data output device 100 is stored in advance in a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive) of the server computer, and the program is loaded into memory and executed by a CPU (Central Processing Unit), causing the server computer to function as the data output device 100. Note that the configuration of the data output device 100 may be changed as desired, and for example, the data output device 100 may be configured using two or more computers.

[0033] Furthermore, in this example of the present embodiment, the spatial configuration recognition model 203 and the image generation model 205 are provided as trained models generated by machine learning. However, if a separate information processing system exists that provides functions similar to those of the trained models, the data output system 100 may be configured to use that information processing system. For example, if a separate spatial configuration recognition system that provides functions equivalent to the spatial configuration recognition model 203 in this example of the present embodiment exists, the spatial configuration recognition unit 202 may be configured to use the separate spatial configuration recognition system as an external collaboration system or as a subsystem that constitutes the data output system 100, instead of the spatial configuration recognition model 203. The same applies to the image generation model 205. If a separate information system that provides functions similar to those of the image generation model 205 exists, the image generation unit 204 may be configured to use the separate information system as an external collaboration system or subsystem, instead of the image generation model 205.

[0034] The user terminal 300 is a terminal used by a user of the data output system 100, and in one example of this embodiment, it is equipped with a photographing device 301 that photographs the office, and a display or other display device 302. The user terminal 300 may be configured using a smartphone, tablet computer, or other mobile terminal, or a desktop personal computer or other well-known computer. Furthermore, although the user terminal 300 in one example of this embodiment is equipped with the photographing device 301, the user terminal 300 may also be configured by combining a digital camera and a well-known computer, and the digital camera may be used as the photographing device 301.

[0035] The network 400 is a computer network that communicatively connects the data output device 200 and the user terminal 300. The network 400 may be a well-known computer network, such as the Internet or other wide area networks, or a LAN (Local Area Network), or may be a combination of a wide area network and a LAN to form the network 400. A VPN (Virtual Private Network) may also be constructed on a wide area network and used as the network 400. The network 400 may also be a wired network, a wireless network, or a combination of a wired network and a wireless network.

[0036] The above is the configuration of the data output system 1 in one example of this embodiment. Next, the flow of data output in one example of this embodiment will be described. Fig. 2 is a flow diagram showing the flow of data output in one example of this embodiment, and Fig. 3 is a diagram showing the configuration of the data output screen W1 in one example of this embodiment.

[0037] In one example of this embodiment, the data output device 200 displays a data output screen W1 on the display device 302 of the user terminal 300, and the user operates the user terminal 300 to generate a modified image by modifying the appearance of any spatial component from an image of an office. As shown in FIG. 3, the data output screen W1 includes an office image designation area W11, a target designation form W12, a recognition level adjustment slider W13, a mask image creation button W14, a mask image area W15, a space taste designation form W16, a modification level adjustment slider W17, an image creation button W18, and a modified image area W19. Note that, in this example of this embodiment, the vertically long data output screen W1 includes the office image designation area W11 to the modified image area W19, but the screen configuration for the data output flow may be changed as desired. For example, data output may be configured to transition between multiple operation and viewing screens.

[0038] In one example of this embodiment, data output begins with the user designating an image of the office for which the appearance of the spatial component is to be changed (step S1). As described above, in one example of this embodiment, the data output screen W1 includes an office image designation area W11. When the user presses the office image designation area W11, a dialog box (not shown) for selecting a predetermined designation method is displayed on the display device 302 of the user terminal 300. In one example of this embodiment, the designation method selection dialog box displays options for designating from a photo library, taking a new photo of the office, and designating from a file, and the user selects any designation method from these options to designate the office image. Selecting designate from a photo library designates an office image from images already taken with the user terminal 300. Selecting take a new photo of the office designates a new image of the office using the camera device 301 and designates the image. Furthermore, selecting designate from a file allows the user to designate any file not stored in the photo library. The method for specifying an image of the office may be selected arbitrarily. For example, when a user presses the office image specification area W11, the screen may transition to one in which an image of the office is taken using the photographing device 301 of the user terminal 300, and the newly taken image may be used to specify the image of the office.

[0039] When an image of the office is designated, the image of the designated office is acquired by the image acquisition unit 201 of the data output device 200. The method and timing for acquiring the image of the office may be selected arbitrarily, and in one example of this embodiment, the image of the office designated from the user terminal 300 is recorded in a predetermined storage area of ​​the data output device 200 by the image acquisition unit 201 of the data output device 200 at the timing when the image of the office is designated in the office image designation area W11.

[0040] Fig. 4 is a diagram showing an example of an image of an office in one example of this embodiment. In the example shown in Fig. 5, the office shown in the image is furnished with furniture such as chairs and desks, interior materials such as walls and floors, and building materials such as doors and windows.

[0041] After completing the designation of the office image, the user then designates the spatial components whose appearances are to be changed (step S2). The target designation form W12 provided on the data output screen W1 is a form for specifying, by text input, which of the spatial components appearing in the image of the office designated in the office image designation area W11 should have their appearances changed. An example of text input W12a is displayed below the target designation form W12. In step S2, for example, entering the text "chair" into the target designation form W12 designates the chair in the image of the office as the spatial component whose appearance is to be changed. In addition, in one example of this embodiment, the appearances of two or more spatial components are designated to be changed by entering text connecting multiple spatial components with commas. For example, entering the text "chair, floor" into the target designation form W12 designates the chair and floor in the image of the office as the spatial components whose appearances are to be changed. When a user inputs text into the target designation form W12, the input text is transmitted from the user terminal 300 to the data output device 200, and the instruction is accepted by the target designation unit 207 of the data output device 200. The specific method for accepting the instruction may be selected arbitrarily; for example, the target designation unit 207 may record text information indicating the instruction in the memory of the data output device 200 or other primary storage device, or, if the data output screen W1 is configured as a web screen, the text information may be recorded in the user terminal 300 by HTTP Cookie or other methods.

[0042] In one example of this embodiment, as described above, the target designation unit 207 accepts designation of the spatial component whose appearance is to be changed, as well as designation of the spatial component's recognition level by the spatial component recognition unit 202. When designating the spatial component whose appearance is to be changed, the user designates the recognition level of the spatial component (step S3). The recognition level adjustment slider W13 provided on the data output screen W1 designates the recognition level of the spatial component in the office image designated in the image designation area W11. In one example of this embodiment, moving the slider to the left end designates a weaker recognition level, and moving it to the right end designates a stronger recognition level. When the user designates the recognition level by moving the recognition level adjustment slider W13 left or right, information indicating this designation is accepted by the target designation unit 207 of the data output device 200.

[0043] After completing the specification of the office image in step S1, the specification of the spatial components that will change the appearance in step S2, and the specification of the degree of recognition in step S3, when the user presses the mask image creation button W14 on the data output screen W1, the data output device 200 recognizes the spatial components in the office image using the spatial structure recognition unit 202 (step S4), generates a mask image using the mask image generation unit 208 (step S5), and generates a depth map using the depth map generation unit 209 (step S6).

[0044] FIG. 5 is a diagram showing an example of a mask image when the recognition level is set to weak in an example of this embodiment, and FIG. 6 is a diagram showing an example of a mask image when the recognition level is set to strong in an example of this embodiment. Both FIGS. 5 and 6 are mask images generated when the spatial components whose appearance is to be changed are specified by text input as "chair, desk, floor, wall, ceil" for the image of the office shown in FIG. 4 described above. These images show the spatial components whose appearance is to be changed in the image of the office shown in FIG. 4 masked with a predetermined color. In the examples of FIGS. 6 and 7, when the recognition level is set to weak, the plant near the center right and the storage shelf near the center left are not recognized as spatial components in FIG. 5, but are recognized as spatial components in FIG. 6. Note that in the mask image of this example of this embodiment, the spatial components whose appearance is to be changed are masked with a predetermined color. However, the masking method may be changed arbitrarily, and an image may be created in which all components other than the spatial components whose appearance is to be changed are masked with a predetermined color. Furthermore, the specific format of the mask image may be selected arbitrarily; for example, a mask image in bitmap format may be generated, or information indicating the mask range in an image of an office may be generated, and the mask image may be constructed using this information and the image of the office.

[0045] In one example of this embodiment, as described above, the data output device 200 includes the spatial configuration recognition unit 202 and the mask image generation unit 208. However, the combination of the spatial configuration recognition unit 202 and the mask image generation unit 208 to generate a mask image may be selected arbitrarily. For example, the spatial configuration recognition unit 202 may recognize all spatial components appearing in a specified office image based on the image and recognition level, and then the mask image generation unit 208 may generate a mask image based on the processing results of the spatial configuration recognition unit 202 and the designation of the spatial components whose appearances have been changed. Alternatively, the spatial configuration recognition unit 202 may recognize only the spatial components whose appearances have been changed in the office image based on the image and recognition level as well as the designation of the spatial components whose appearances have been changed, and then convert the processing results of the spatial configuration recognition unit 202 into a predetermined mask image. Furthermore, as described above, in cases where the spatial configuration recognition model 203 can directly output a mask image, the spatial configuration recognition unit 202 may also function as the mask image generation unit 208.

[0046] In step S5 of this embodiment, a mask image for internal processing is generated along with the mask image described above. FIG. 7 is a diagram showing an example of a mask image for internal processing in this embodiment. The mask images shown in FIGS. 5 and 6 are images displayed on the data output screen W1, as described below, and are images in which the area in the office image whose appearance is to be changed is masked to make it easier for the user to see. The mask image for internal processing is an image that serves as input data for the image generation process, as described below, and is an image that shows, as shown in FIG. 7, the areas in the office image whose appearance is to be changed and the areas whose appearance is not to be changed as a binary bitmap. Note that the type of image to be generated as the mask image may be selected arbitrarily. It is possible to generate only images that are easy for the user to see, as shown in FIGS. 5 and 6, or to generate only a mask image for internal processing, as shown in FIG. 7.

[0047] FIG. 8 is a diagram illustrating an example of a depth map in an example of this embodiment. The depth map illustrated in FIG. 8 is generated by the depth map generation unit 209 of the data output device when the office image of FIG. 5 described above is specified in the office screen specification area W11. The depth map in this example of this embodiment is an image that represents the depth in the specified office image, i.e., the relative distance from the camera 301 that captured the office image, as color intensity, for each pixel constituting the office image. In this example of this embodiment, the depth map is used as input data for the image generation model 205 as information retaining the angle of view of the office image and the positional relationship of spatial components when generating a modified image. Note that, although a depth map is used in this example of this embodiment, it is also possible to use an image in which the contours of spatial components, etc., in the office image are extracted, or a segmentation map in which the office image is divided into multiple regions for each spatial component, etc., instead of or in addition to the depth map. The format of the depth map may be chosen arbitrarily; for example, the depth map may be constructed as a bitmap image that represents depth in grayscale, or the depth map may be constructed with numerical depth information for each pixel in the office image.

[0048] When steps S4 to S6 are completed, the data output device 200 selectively displays the generated mask image and depth map in the mask screen area W15 of the data output screen W1 (step S7). A tab W15a for selecting an image to be displayed is provided above the mask image area W15, and by selecting the tab, the selected image is displayed in the mask image area W15.

[0049] In one example of this embodiment, a user can operate the user terminal 300 to modify a mask area of ​​the mask image displayed in the mask image area W15. FIG. 9 is a diagram showing an example of mask image modification in one example of this embodiment, where the shaded area in FIG. 9 is the area modified by the user. In one example of this embodiment, as described above, a mask image generated by the mask image generation unit 208 is displayed in the mask image area W15. In this displayed state, the user can modify the mask image by operating the user terminal 300. The specific method of modification may be selected arbitrarily. For example, when a user selects an arbitrary area in the mask image displayed in the mask image area W11, a dialog box for selecting whether to add or delete a mask from the selected area may be displayed on the display device 302 of the user terminal 300. When the user selects add or delete from the displayed dialog, the selected area is masked or the selected area is deleted from the mask. Furthermore, in one example of this embodiment, the data processing screen W1 is provided with a target designation form W12, a recognition degree adjustment slider W13, a mask image creation button W13, and a mask image area W15, and in addition to the above-described correction method, when the mask image is displayed in step S7, the user can change the spatial component whose appearance is to be changed using the target designation form W12, and / or change the degree of recognition using the recognition degree adjustment slider W13 and press the mask image creation button W13 again, thereby repeating the recognition of the spatial component and the generation of the mask image.

[0050] In one example of this embodiment, as described above, a mask image for internal processing is generated together with a mask image to be displayed on the user terminal 300, but the timing of generating the mask image for internal processing and correcting the mask range may be changed as desired. For example, the mask image generating unit 208 may generate a mask image for internal processing that reflects the correction each time the user corrects the mask range, or alternatively, the mask image for internal processing may not be generated when the mask image to be displayed on the user terminal 300 is generated, but may generate a mask image for internal processing that reflects the correction when the user has completed correcting the mask image.

[0051] Once the mask image and depth map are generated, the user then specifies the space taste (step S8). The space taste specification form W16 on the data output screen W1 is a form for specifying the space taste, including one or more of the color, material, and texture of the space components, by text input. A text input example W16a is displayed below the space taste specification form W16. For example, specifying "office, modern" allows the user to specify that the appearance of the specified space components in the office image be changed to that of modern office furniture. When text is entered into the space taste specification form W16, the space taste specification unit 210 of the data output device 200 accepts the space taste specification via the text input. The method for accepting the specification may be selected arbitrarily, and may be recorded in the memory or other primary storage device of the data output device 200, or may be recorded in the user terminal 300 via an HTTP Cookie or the like.

[0052] In addition, in one example of this embodiment, the space taste designation unit 210 of the data output device 200 accepts a designation of the degree of change of the appearance in addition to the designation of the above-mentioned space taste (step S9). The change degree adjustment slider W17 provided on the data output screen W1 designates the degree of change of the appearance of the space component designated in the above-mentioned target designation form W12. In one example of this embodiment, moving the slider to the left end designates a weaker degree of change, and moving it to the right end designates a stronger degree of change. When the user designates the degree of change by moving the change degree adjustment slider W17 left or right, information indicating this designation is accepted by the space taste designation unit 210 of the data output device 200.

[0053] Note that the method for specifying the space taste and the degree of change may be selected arbitrarily. For example, instead of or in addition to text input, multiple selectable space tastes may be provided in advance, and the user may select any space taste. FIG. 10 is a diagram showing an example of a space taste selection dialog W2. In the example shown in FIG. 10, the space taste selection dialog W2 displays an image W21 showing an example of a space component to which a space taste has been applied, a color W22 included in the space taste, and a space taste name W23. When the user selects one of the space tastes, the selected space taste may be treated as being specified.

[0054] Furthermore, instead of or in addition to text input, the designation of spatial taste may be made by accepting the input of an image representing the spatial taste. FIG. 11 is a diagram showing an example of an image representing a spatial taste. When the image shown in FIG. 11 or another image is designated as an image representing a spatial taste, the spatial taste designation unit 210 of the data output device 200 may be configured to treat the spatial taste as having been designated, including one or more of the color, material, or texture of the spatial components in the designated image. Furthermore, when both text input and image input of spatial taste are possible, instead of or in addition to the change degree adjustment slider W17 on the data output screen W1, the designation may be configured to allow the user to specify the weight of the spatial taste designated by text input and the spatial taste designated by image. The interface for specifying the weighting may be configured in the same way as the change degree adjustment slider W17, and may be configured so that, for example, when the slider is moved to the left end, the weighting of the text input is increased, and when it is moved to the right end, the weighting of the image is increased, and a modified image is generated based on the specified weighting, taking into account the spatial taste specified by the text input and the spatial taste specified by the image as requested by the user.

[0055] When the user presses the image generation button W18 on the data output screen W1 after completing the designation of the spatial taste in step S8 and the designation of the degree of change in step S9, the data output device 200 causes the image generation unit 204 to generate a modified image (step S10). As described above, in one example of this embodiment, the image generation unit 204 of the data output device 200 generates a modified image in which the appearance of the specified spatial component is changed using the image generation model 205. The image generation model 205 generates a modified image in which the appearance of the spatial component in the area designated by the mask image in the office image is changed to the appearance of the color, material, or texture included in the specified spatial taste, based on the office image acquired by the image acquisition unit 201, the mask image for internal processing generated by the mask image generation unit 208, information indicating the designation of the spatial taste accepted by the spatial taste designation unit 209, and the depth map generated by the depth map generation unit 209. The modified image generated by the image generating unit 204 is displayed in the modified image area W19 by the image output unit 205 (step S11).

[0056] Fig. 12 is a diagram showing an example of a modified image when a weak degree of change is specified in one example of this embodiment, and Fig. 13 is a diagram showing an example of a modified image when a strong degree of change is specified in one example of this embodiment. Both Fig. 12 and Fig. 13 are modified images generated when the spatial components to change the appearance of the office image shown in Fig. 4 are specified by text input as "chair, desk, floor, wall, ceiling" and the spatial style is specified by text input as "office, modern style." In the office image, the specified spatial components have one or more of their color, material, or texture changed depending on the degree of change in the spatial style.

[0057] In one example of this embodiment, the data output screen W1 has a space taste designation form W16, a change degree adjustment slider W17, and an image generation button W18. When the modified image is displayed in step S11, the user can change the space taste using the space taste designation form W16 and / or change the degree of change using the change degree adjustment slider W17 and press the image generation button W18, thereby repeatedly generating and displaying the modified image.

[0058] This completes the description of one example of this embodiment. However, the embodiment of the present invention is not limited to the above. For example, in this example of this embodiment, the appearance of the spatial component designated by the target designation unit 207 is changed according to the spatial taste designated by the spatial taste designation unit 209. However, instead of or in addition to the spatial taste designation unit 209, a spatial composition change unit that changes the designated spatial component to a product designated by the user may be provided, and the image generation unit 204 may be configured to change the appearance of the spatial component designated by the target designation unit 207 to the appearance of the product designated by the spatial component change unit.

[0059] In one example of this embodiment, a modified image in which the appearance of a specified spatial component is modified is generated based on an image of the office captured by the image capturing device 301 of the user terminal 300. However, instead of the office image, a modified image in which the appearance of a specified spatial component is modified may be generated from a 2D or 3D CAD (Computer Aided Design) file. In this case, the data output device 200 may be configured to include a rendering unit that renders the CAD file, and the image acquisition unit 201 may be configured to acquire the image rendered by the rendering unit as the office image. Alternatively, the image acquisition unit 201 may be configured to include a CAD file acquisition unit instead of the image acquisition unit 201, and the spatial component recognition unit 202 and the mask image generation unit 208 may be configured to generate a mask image based on the CAD file acquired by the CAD file acquisition unit and the specification accepted by the target designation unit 207.

[0060] Other specific configurations are not limited to those of the present embodiment, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0061] 100 Data Output System 200 Data output device 201 Image acquisition unit 202 Spatial configuration recognition unit 203 Spatial Configuration Recognition Model 204 Image Generation Unit 205 Image Generation Model 206 Image output unit 207 Target Designation Section 208 Mask image generation unit 209 Depth map generation unit 210 Space taste specification section 300 User terminal 301 Imaging equipment 302 Display device 400 Network

Claims

1. an image acquisition unit that acquires an image of the office; a spatial configuration recognition unit that recognizes spatial configurations including one or more of furniture, interior components, and fittings arranged in the office in the acquired image; an image generation unit that generates a modified image by changing the appearance of the spatial configuration, including one or more of the shape, pattern, and color, based on a user's request, using an image generation model generated by machine learning; an image output unit that outputs the generated modified image; A data output device comprising:

2. The data output device further includes: a target designation unit that accepts designation of the spatial component to be changed or the spatial component not to be changed by text input by the user; 2. The data output device according to claim 1, wherein the image generation section takes the designation received by the target designation section as a request from the user and generates the modified image based on the request.

3. the target designation unit further receives a designation of a recognition degree of the spatial component; the image generation unit generates the changed image in response to a request from the user, which is a designation of the spatial component to be changed or the component not to be changed, and a designation of the recognition degree.

3. The data output device according to claim 2.

4. The data output device further includes: a mask image generating unit that generates a mask image in which the spatial component that changes the appearance or a component other than the spatial component that changes the appearance is masked in the acquired image; 2. The data output device according to claim 1.

5. The data output device further includes: a depth map generating unit that generates a depth map indicating a depth of the spatial component in the acquired image; the image generator generates the modified image based on the user request and the depth map.

2. The data output device according to claim 1.

6. The data output device further includes: a space taste designation unit that accepts designation of a space taste including one or more of the color, material, and texture of the space component by text input by the user; the image generation unit generates the modified image based on the request of the user, taking the request received by the space taste designation unit as the request of the user.

2. The data output device according to claim 1.

7. The space taste designation unit further receives a designation of a degree of change of the appearance, the image generation unit generates the modified image based on the request of the user, which is a designation of the spatial taste and a designation of the degree of modification; 7. The data output device according to claim 6.

8. The space taste designation unit accepts input of an image indicating the space taste instead of the text input.

7. The data output device according to claim 6.

9. the data output device further includes a spatial component change unit that changes the designated spatial component to a product designated by the user; the target designation unit accepts designation of the spatial component to be changed by text input by the user, The image generation unit changes the appearance of the specified spatial configuration to the appearance of the changed product.

3. The data output device according to claim 2.

10. A data output system comprising a user terminal carried by the user and the data output device according to any one of claims 1 to 9, The image acquisition unit acquires an image of the office taken by a photographing device provided in the user terminal, the image output unit outputs the generated image to a display device provided in the user terminal. Data output system.

11. A program that causes a computer to function as the data output device according to any one of claims 1 to 9.

Images