Information processing system, information processing method and information processing program

Abstract

To generate an image obtained by appropriately performing a design corresponding to a theme to an image displaying a three-dimensional model of a building.SOLUTION: An information processing system related to one example provided with at least one processor. The at least one processor generates a design image about a three-dimensional model by an image generation model on the basis of the image generation model for newly generating an image on the basis of an image an a prompt, a model image displaying the three-dimensional model of a building, theme information showing an index of a design, and type information showing attributes of the building shown by the model image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present disclosure relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there are known techniques for performing simulations related to three-dimensional models. For example, Patent Document 1 discloses a technique for integrating a three-dimensional shape model of a room and a three-dimensional shape model of fittings, and generating and outputting an image of a three-dimensional space corresponding to the position of a viewpoint. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-043647 Summary of the Invention [Problem to be solved by the invention]

[0004] One aspect of the present disclosure aims to generate an image that is appropriately designed according to a theme for an image displaying a 3D model of a building. [Means for solving the problem]

[0005] An information processing system according to one embodiment of the present disclosure includes at least one processor, and the at least one processor generates a design image for the three-dimensional model using an image generation model that generates a new image based on an image and a prompt, a model image that displays a three-dimensional model of a building, theme information that indicates design indicators, and type information that indicates attributes of the building shown by the model image.

[0006] An information processing method according to one embodiment of the present disclosure is an information processing method executed by an information processing system having at least one processor, and includes a step of generating a design image for the three-dimensional model using the image generation model based on an image generation model that generates a new image based on an image and a prompt, a model image that displays a three-dimensional model of a building, theme information that indicates design indicators, and type information that indicates attributes of the building shown by the model image.

[0007] An information processing program according to one embodiment of the present disclosure causes a computer to execute a step of generating a design image relating to a three-dimensional model using an image generation model based on an image and a prompt, a model image displaying a three-dimensional model of a building, theme information indicating design indicators, and type information indicating attributes of the building indicated by the model image.

[0008] According to this embodiment, a design image of a 3D model of a building is generated based on the model image, the theme information, and the type information. Therefore, an image in which a design according to the theme is appropriately applied to an image displaying a 3D model of a building can be generated. Effect of the Invention

[0009] According to one aspect of the present disclosure, it is possible to generate an image in which a design appropriate to a theme is applied to an image displaying a three-dimensional model of a building. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of an overall configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system according to an embodiment. [Diagram 3] FIG. 13 is a diagram showing an example of a model screen display. [Figure 4] FIG. 13 is a diagram showing an example of a model screen display. [Diagram 5]FIG. 13 is a diagram showing an example of a display of a capture screen. [Figure 6] FIG. 13 is a diagram showing an example of a display of a capture screen. [Figure 7] FIG. 13 is a diagram showing an example of a display of a design screen. [Figure 8] 4 is a flowchart illustrating an example of an information processing method according to the embodiment. [Figure 9] 4 is a flowchart illustrating an example of an information processing method according to the embodiment. [Figure 10] 4 is a flowchart illustrating an example of an information processing method according to the embodiment. [Figure 11] FIG. 13 is a diagram showing an example of a display of a design screen according to a modified example. [Figure 12] FIG. 13 is a diagram illustrating an example of an overall configuration of an information processing system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicated description will be omitted.

[0012] [System configuration] The information processing system according to the embodiment is a computer system for providing an image to a user viewing information on a space such as a building. Specifically, the information processing system is a computer system for providing an image that displays a space such as a building and that is designed according to a theme.

[0013] A building refers to a structure that is fixed to a specific land or space. A building is, for example, a structure having a roof and pillars or walls. The building according to the present embodiment may include a civil engineering structure. A building is, for example, a detached house, an apartment building, an office building, or a public facility. A building may be, for example, a facility that serves as an event venue, such as a baseball field, an athletic field, a horse racing track, an arena, or a concert hall. The type of building is not limited. The space of a building is not limited to the space inside the building, but also includes the exterior of the building and the surrounding environment of the building. The space includes objects that form the space other than the building.

[0014] Viewing information about a space such as a building refers to the act of visually checking the target building and objects associated with the building when, for example, designing, constructing, inspecting, managing, viewing, viewing, or using the building. In the following, as an example of viewing information about a space such as a building, viewing information about the building that is necessary when viewing inside the building is shown. Viewing (viewing) inside a building is the act of visually checking furniture, fixtures, wallpaper, etc. that are displayed as examples of when a user actually lives in the building. The purpose of viewing a building is not limited. In the following, viewing inside a detached house is shown as an example of viewing a building.

[0015] A user who views information when viewing the inside of a building refers to an entity that directly or indirectly participates in the viewing inside the building. For example, the user includes at least one of an entity involved in the construction of the building to be viewed, an entity that holds a preview, and an entity that views the building. The user may include, for example, a customer (owner) considering purchasing or renting the building, a designer and a design company that designs the building, a real estate company and a developer that buys and sells the building, a builder that constructs the building, and the like. The user may also include an administrator of the information processing system.

[0016] FIG. 1 is a diagram showing an example of the overall configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes a management device 5. The management device 5 can access at least one user terminal 2, a three-dimensional model database 3, a server 4, and an image generation model database 6 via a communication network 7. In the information processing system 1, a model image displaying a three-dimensional model is generated in the management device 5 based on viewpoint information input by the user terminal 2 and information related to the three-dimensional model displayed by the three-dimensional model database 3. In addition, the information processing system 1 of this embodiment generates a design image related to the three-dimensional model based on theme information indicating design indicators, type information indicating attributes of the building indicated by the model image, and the model image. Furthermore, in the information processing system 1 of this embodiment, at least one of the model image and the design image is displayed on the screen of the user terminal 2. Details of the information processing system 1 will be described below.

[0017] The user terminal 2 is a computer used by a user. There is no limitation on the type of computer used as the user terminal 2. For example, the user terminal 2 may be a mobile terminal such as a high-function mobile phone (smartphone), a mobile phone, a personal digital assistant (PDA), a tablet terminal, a wearable terminal, or a personal computer. There is no limitation on the number of user terminals 2 in the information processing system 1. The user terminal 2 includes input devices such as a keyboard, a mouse, and a touch panel, and a display device such as a monitor and a touch panel.

[0018] The user terminal 2 accepts the user's input operation and transmits information related to the input operation to the management device 5 via the communication network 7. In response to the display control of the management device 5, the user terminal 2 displays a screen for the user to view the inside of the building. In the present disclosure, the screen is also referred to as a "model screen." The model screen displays a main element as an element that allows the user to check objects in the building. In the model screen, the main element is an element that draws a three-dimensional model of the building in the imaging area of ​​the camera object, which is visible from a viewpoint that can be operated by the user. The main element represents, for example, a two-dimensional HTML element for drawing the three-dimensional model on the screen. The main element in the model screen is updated in response to the user's input operation.

[0019] Furthermore, in response to the display control of the management device 5, the user terminal 2 displays a screen for acquiring a model image that has acquired at least a part of the main element. In the present disclosure, the screen is also referred to as a "capture screen." The model image is an image that records at least a part of the main element displayed on the user terminal 2. The model image is recorded by capturing an image of the main element displayed on the user terminal 2 at any timing in response to an input operation by the user. The model image is, for example, a screenshot of the main element.

[0020] Furthermore, in response to the display control of the management device 5, the user terminal 2 displays a screen for viewing a design image obtained by changing the design of the model image from which at least a part of the main element has been acquired. In the present disclosure, this screen is also referred to as a "design screen." The design screen displays a design image in which a design according to a theme has been applied to the model image. In the design screen, the design image is an image in which a design has been applied to a three-dimensional model of a building in the imaging area of ​​the camera object, as seen from the same viewpoint as the main element shown in the model image. "Applying a design" refers to making the content based on annotation information, user information, theme information, and type information, which will be described later.

[0021] The 3D model database 3 is a database that stores property information used to display a 3D model of a building on the user terminal 2 and to generate a model image. In one example, data items of the property information include a building ID that uniquely identifies the building and a 3D model of the building. The 3D model is electronic data for showing the building in a three-dimensional manner. In one example, the property information is stored in the 3D model database 3 by the designer of the building or the operator of the management device 5.

[0022] In one example, the three-dimensional model is created by converting a design drawing created by CAD (Computer Aided Design) using a conversion system. The three-dimensional model may be created using, for example, 3DCAD (three-dimensional Computer Aided Design), BIM (Building Information Modeling), CIM (Construction Information Modeling / Management), etc. The three-dimensional model may include models of the exterior and interior of a building, models of furniture that can be installed in a building, etc.

[0023] The three-dimensional model may include setting information, which is a feature that a user can set for the three-dimensional model. In one example, the setting information may include a model that expresses the exterior condition, the interior condition, the sunlight condition, the time of day, the condition of a human model virtually standing in a space, the installation condition of a structure, etc. The configuration of the property information and the three-dimensional model is not limited.

[0024] The server 4 stores annotation information, user information, theme information, and type information that are registered in advance in the management device 5, as well as model images and design images generated in the management device 5, and the like.

[0025] The annotation information refers to information associated with the three-dimensional model. The annotation information includes at least one of content information and viewpoint information. The content information includes information associated with the three-dimensional model related to the object of the building. In one example, the data items of the content information include a content ID and detailed information.

[0026] Content information is information that expresses a set of one or more pieces of detailed information corresponding to a certain object as one group (one unit). A content ID is an identifier that uniquely identifies the detailed information assigned to a specified object. An object of a building may be a tangible object related to the building to which content information is assigned. An object may be a component (equipment) of a building, such as a pillar, wall, beam, window, door, storage, interior, exterior, etc., or an item installed in a building, such as furniture, electrical appliances, etc. Content information may include information associated with the space and layout formed by the components of a 3D model of the building.

[0027] The detailed information is a collection of one or more pieces of information (hereinafter referred to as related information) associated with at least one of an object and a space. Each record of the detailed information may include a related information ID that uniquely identifies the related information, the related information, and a display order. Each piece of related information may be expressed by at least one of characters, figures, sounds, images, and videos. Each piece of related information may include, for example, the name, shape, function, position, size, range, etc., of the object and space of the building. The content information may be displayed on the model screen as a pop-up. In this case, the detailed information of the content information may include the display position on the model screen.

[0028] The viewpoint information includes information about a viewpoint indicating a position from which the building in the three-dimensional model is viewed. The viewpoint can be said to be how the three-dimensional model displayed on the user terminal 2 appears. The viewpoint may be located outside the building or inside the building. In one example, the viewpoint is located inside the building from which the interior of the building can be viewed. The viewpoint information includes information about at least one viewpoint. The viewpoint information includes a name of the viewpoint, position information regarding the position of the viewpoint relative to the three-dimensional model, and angle of view information regarding the angle of view relative to the three-dimensional model. The name of the viewpoint is character information that can be distinguished from other viewpoints. The information position information includes each of the horizontal and height coordinates of the viewpoint relative to the three-dimensional model. The position information is information indicating a position to which a camera object described later moves. The angle of view information is information indicating a range captured by a camera object described later. The angle of view information will be described in detail later. The viewpoint information may include a thumbnail image as an example of a three-dimensional model of a building seen from the viewpoint.

[0029] The user information is information on the characteristics of a user related to a building in a three-dimensional model. The user related to the building may be the same as the user who views the above-mentioned information. The user related to the building refers to, for example, a customer who is considering purchasing or renting a building. The user information may include a user ID and persona information. The user ID is an identifier that uniquely identifies the persona information registered by the user. The user ID is set, for example, for each user who views the main element on the user terminal 2. The persona information includes a typical user image that uses the three-dimensional model viewing service or the service related to the information processing system 1. The persona information includes information on at least one of age, sex, place of residence, occupation, position, annual income, hobbies, special skills, values, family structure, upbringing, how to spend holidays, lifestyle, favorite color, and surrounding environment. The user information is information that is input in advance by the user to the registration screen displayed on the user terminal 2 by the management device 5, and is stored in the server 4.

[0030] Theme information is information that indicates design indicators. Theme information is information that expresses an idea, etc., to be expressed for the entire building or at least one object of the building as the basis of the design image. Theme information may include information provided by the user. Theme information may include theme explanatory information and a theme name. The theme explanatory information is information that explains an idea, etc., to be expressed for the entire building or one or more objects of the building. The theme explanatory information and the theme name may be expressed by at least one of characters, figures, sounds, images, and videos. The theme name is information that expresses the theme explanatory information with a small amount of information. The theme name may be expressed by at least one of characters and figures.

[0031] The theme name included in the theme information may be expressed by a general name such as "modern", "minimalist", "professional", "tropical", "vintage", etc. The theme information may include multiple design indexes provided by a company in charge of at least one of the design (planning) and construction of a building. The theme information may include, for example, one or more themes that are standardly used by a manufacturer (company) that designs or constructs a building. In this case, the theme name included in the theme information may be, for example, "Japanese modern", "California style", "authentic modern", etc. If the theme name is "Japanese modern", the theme explanation information includes text information such as "It actively incorporates solid wood, white walls, and diatomaceous earth that evoke Japanese tradition, as well as Japanese-style fittings and furniture such as tatami mats, shoji screens, and sliding doors, and fuses Western modern style. It is a style that pursues warmth while also pursuing tranquility and freshness."

[0032] The type information is information indicating attributes of the building shown in the model image. The attributes of the building refer to the properties of the space of the building that occupies a large proportion of the model image. The properties of the space include the intended use of the space by the user. The type information may include a type name. The type name is the name of the space. The type name includes the names of the spaces that occupy the building and the surroundings of the building, such as the living room, kitchen, bathroom, washroom, corridor, stairs, balcony, garden, etc. The type name may be expressed by at least one of characters and figures.

[0033] The image generation model database 6 is a database that stores an image generation model that generates a new image based on an image and a prompt. The image generation model generates a new design image based on a model image as the image, and theme information and type information as the prompt. The prompt may include annotation information and user information. The image generation model may accept other information in addition to the model image, annotation information, user information, theme information, and type information. The other information may be, for example, property information. The image generation model will be described in detail later. In one example, the image generation model is stored in the image generation model database 6 by a machine learning unit 9 of the management device 5 described later.

[0034] The management device 5 is a computer that controls the contents of the model screen, capture screen, and design screen to be displayed on the user terminal 2. The management device 5 has, as functional components, a machine learning unit 9, a model control unit 10, a capture control unit 20, and a design control unit 30. First, the model control unit 10 will be described. The model control unit 10 includes a first input receiving unit 11, a setting unit 12, an image drawing unit 13, and a first display control unit 14. These functional components work together to control the model screen on the user terminal 2.

[0035] The first input receiving unit 11 is a functional element that receives information input in the user terminal 2. The first input receiving unit 11 receives query information for querying the three-dimensional model database 3 for property information. The first input receiving unit 11 receives viewpoint information. The first input receiving unit 11 outputs an operation signal indicating the received information to each functional element of the model control unit 10 of the management device 5.

[0036] The setting unit 12 is a functional element that generates a camera object. The camera object is used to draw a three-dimensional model in its imaging area as a main element in a canvas area. The main element is an element used to generate a model image. The camera object moves to a specified viewpoint in response to a user's input operation, and captures an image of the three-dimensional model. The camera object is used when displaying at least a model screen in the user terminal 2.

[0037] The setting unit 12 sets an imaging area for the camera object. The imaging area is an area captured by the camera object. The setting unit 12 accesses the server 4 to acquire viewpoint information based on an operation signal input from the first input receiving unit 11, and sets the imaging area of ​​the camera object based on the acquired viewpoint information.

[0038] The imaging area is set based on the angle of view information included in the viewpoint information input in the user terminal 2. For example, the angle of view information includes the angle of view, the aspect ratio, the close distance, the far distance, and the like. The angle of view includes the horizontal and height directions at which a three-dimensional model can be viewed with the viewpoint as the center. The height direction angle includes the elevation angle (depression angle). The aspect ratio indicates the aspect ratio of the imaging result. The close distance is the near clip distance. An area closer than the close distance to a camera object located at a certain viewpoint is not included in the imaging result. The far distance is the far clip distance. An area farther than the far distance to a camera object located at a certain viewpoint is not included in the imaging result. As described above, the imaging area includes the area of ​​the quadrangular pyramid between the close distance and the far distance among the quadrangular pyramids that can be defined by the angle of view and the aspect ratio.

[0039] Furthermore, the setting unit 12 is a functional element that generates a canvas area for a viewpoint where a three-dimensional model captured by a camera object can be drawn. The canvas area is an area where a three-dimensional model to be displayed in the main element can be drawn. The canvas area is an area where two-dimensional image information can be drawn when information on a three-dimensional model is changed to two-dimensional image information by capturing an image by a camera object. The size of the canvas area is, for example, set in advance and stored in the server 4.

[0040] The image drawing unit 13 is a functional element that moves the camera object to a certain viewpoint based on viewpoint information and draws a three-dimensional model within the imaging area of ​​the camera object in the canvas area. The viewpoint to which the camera object moves is set to a viewpoint specified by a user's input operation, for example. Note that the viewpoint to which the camera object moves may be any of a plurality of viewpoints. Hereinafter, the viewpoint may be referred to as a drawing viewpoint.

[0041] The image drawing unit 13 moves the camera object to a viewpoint in the three-dimensional model based on viewpoint information corresponding to the drawing viewpoint. The image drawing unit 13 sets the orientation of the camera object to a predetermined direction, for example. The image drawing unit 13 draws the three-dimensional model within the imaging area of ​​the camera object in the canvas area. In response to the user's input operation accepted by the first input accepting unit 11, the position and orientation of the camera object can be changed each time.

[0042] The first display control unit 14 is a functional element that controls the model screen on the user terminal 2. The first display control unit 14 accesses at least one of the three-dimensional model database 3 and the server 4 based on an operation signal input from the first input receiving unit 11 to acquire at least one of property information and viewpoint information, and controls the model screen based on the acquired information. In one example, the first display control unit 14 applies the acquired information to the setting unit 12 and the image drawing unit 13. The first display control unit 14 acquires a three-dimensional model of the canvas area drawn by the image drawing unit 13, and controls the model screen based on the acquired information.

[0043] Specifically, when a user desires to view a specific building, the user inputs query information for querying property information corresponding to the specific building into the user terminal 2. When query information is input from a certain user terminal 2, the first display control unit 14 acquires property information corresponding to the query information from the 3D model database 3. The first display control unit 14 sends a 3D model of the building indicated by the property information to the image rendering unit 13. The first display control unit 14 acquires viewpoint information associated with the property information from the server 4. The first display control unit 14 sends the viewpoint information to the setting unit 12 and the image rendering unit 13.

[0044] The first display control unit 14 reads out viewpoint information corresponding to the property information of the building from the server 4. The first display control unit 14 displays a three-dimensional model based on the viewpoint information recorded in the server 4. That is, the first display control unit 14 displays the three-dimensional model drawn in the canvas area by the image drawing unit 13 on the screen of the user terminal 2. The element displayed on the screen is the main element. With this display method, a three-dimensional model from the same viewpoint as one drawing viewpoint included in the viewpoint information is displayed on the user terminal 2 as the main element.

[0045] Furthermore, the first display control unit 14 displays a viewpoint display element as viewpoint information on the model screen based on the viewpoint information. The viewpoint display element displays information related to at least one viewpoint included in the viewpoint information. The first display control unit 14 acquires viewpoint information corresponding to the building from the server 4, and displays a list of viewpoints included in the viewpoint information on the screen of the user terminal 2. The first display control unit 14 displays the viewpoint display elements in the list so that they are smaller than the main element. For example, the first display control unit 14 displays thumbnail images of viewpoints close to the viewpoint indicating the main element in order in the list. Furthermore, the first display control unit 14 controls so that the same image as the main element is not displayed in the list of viewpoint information in the viewpoint display element.

[0046] Furthermore, the first display control unit 14 reads out content information corresponding to the property information of the building from the server 4. The first display control unit 14 displays content display elements as content information on the model screen based on the content information. At least one detailed information included in the content information is displayed in the content display elements. The first display control unit 14 acquires content information corresponding to the building from the server 4, and displays detailed information on at least one object captured in the main element of the content information on the screen of the user terminal 2. The first display control unit 14 displays the content display elements in the list so that they are smaller than the main element. The first display control unit 14 may also control so that viewpoint display elements related to viewpoint information are displayed on the same screen together with content display elements related to the content information.

[0047] Next, a description will be given of the capture control unit 20. The capture control unit 20 includes a second input receiving unit 21, a model image generating unit 22, and a second display control unit 23. These functional components work together to generate a model image and control the capture screen on the user terminal 2.

[0048] The second input receiving unit 21 is a functional element that receives information input in the user terminal 2. For example, the second input receiving unit 21 receives an input operation signal from a user to acquire a model image for a main element displayed on the user terminal 2. The second input receiving unit 21 outputs an operation signal indicating the received information to each functional element of the capture control unit 20 of the management device 5.

[0049] The model image generating unit 22 is a functional element that generates a model image that displays a three-dimensional model drawn in a canvas area in response to a user's input operation. The model image generating unit 22 generates a model image that includes a three-dimensional model of an object identical to a main element displayed on the model screen at the time the user's input operation is performed. That is, the model image generating unit 22 generates one model image for one user's input operation. One model image is generated for one viewpoint. One model image displays a three-dimensional model that can be seen from one corresponding viewpoint. The image generating unit 45 stores the generated model image in the server 4.

[0050] The second display control unit 23 displays a model image display element including the model image generated by the model image generation unit 22 on the capture screen. The model image and a functional user interface (hereinafter, referred to as functional UI) for the model image are displayed in the model image display element. In this manner, the second input receiving unit 21 receives a user's input operation signal on the model screen, and the second display control unit 23 displays the model image display element on the screen of the user terminal 2, whereby the second display control unit 23 transitions the screen on the user terminal 2 from the model screen to the capture screen.

[0051] The functional UI includes, for example, a user interface for re-capturing the model image, a user interface for saving the model image, and a user interface for instructing generation of a design image. The second display control unit 23 acquires the model image from the model image generation unit 22, and displays the functional UI together with the model image on the screen of the user terminal 2. The second display control unit 23 displays the model image display element so that it is smaller than the main element. The second display control unit 23 may display the model image display element instead of the viewpoint display element and the content display element.

[0052] When the user inputs an operation to generate a design image in the model image display element, the second display control unit 23 displays a capture screen display element on the capture screen. The capture screen display element displays a user interface that allows the user to input theme information and type information as at least a part of a prompt regarding the generation of the design image. The capture screen display element displays a user interface that instructs the user to generate an image that can be operated after the theme information and type information are input. In this way, the second input receiving unit 21 receives an input operation signal from the user, and the second display control unit 23 displays the capture screen display element on the screen of the user terminal 2, allowing the user to give detailed instructions regarding the generation of the design image.

[0053] Next, the design control unit 30 will be described. The design control unit 30 includes a third input receiving unit 31, a design image generating unit 32, and a third display control unit 33. These functional components work together to generate a design image and control the design screen on the user terminal 2.

[0054] The third input receiving unit 31 is a functional element that receives information input in the user terminal 2. For example, the third input receiving unit 31 receives an input operation signal from a user to acquire a design image for a capture screen display element of a capture screen displayed on the user terminal 2. The third input receiving unit 31 outputs an operation signal indicating the received information to each functional element of the design control unit 30 of the management device 5.

[0055] The design image generating unit 32 is a functional element that generates a design image in response to an input operation by a user. The design image generating unit 32 acquires a model image, annotation information, user information, theme information, and type information from the server 4. The design image generating unit 32 acquires an image generation model from the image generation model database 6. The design image generating unit 32 generates a design image by inputting a model image as an image to the image generation model, and inputting annotation information, user information, theme information, and type information as prompts.

[0056] Here, the image generation model will be described in detail. In one example, the image generation model may include a neural network that accepts inputs of a model image, annotation information, user information, theme information, and type information, and outputs a design image. In this case, the image generation model is a trained model. An example of a neural network includes, but is not limited to, a recurrent neural network (RNN).

[0057] The image generation model is, for example, a model in which machine learning is performed in the machine learning unit 9. In machine learning for obtaining an image generation model, input information including at least one of theme information and type information, as well as a model image, and a teacher image corresponding to the input information are prepared in advance. The input information may include at least one of annotation information, user information, theme information, and type information. The annotation information, user information, theme information, and type information are used as captions associated with the model image. The teacher image is a correct image after design is applied based on at least one of annotation information, user information, theme information, and type information, as well as the model image. The teacher image is, for example, generated in advance by a user. In this way, the machine learning unit 9 trains the image generation model using a data set that includes a model image, at least one of annotation information, user information, theme information, and type information, and a teacher image.

[0058] The machine learning unit 9 shown in FIG. 1 generates an image generation model using input information and a teacher image. The machine learning unit 9 generates the image generation model at least before the design image generation unit 32 generates a design image. In one example, the machine learning unit 9 executes the following process for each record of the input information and the teacher image. That is, the machine learning unit 9 inputs the input information indicated by the record into the image generation model, and obtains an estimated image output from the neural network. Note that each data of the input information and the teacher image in the machine learning does not have to be data related to a building.

[0059] The machine learning unit 9 updates the parameters in the image generation model using a method such as backpropagation based on the error between the estimated image and a teacher image (correct answer image) corresponding to the input image indicated by the record. The error refers to the degree of agreement of each parameter, such as the type, size, position, range, and color, between the object included in the estimated image and the object included in the teacher image. For example, the machine learning unit 9 updates the weights of the neural network. The machine learning unit 9 continues machine learning until a given termination condition is satisfied. For example, the machine learning unit 9 may evaluate the performance of the image generation model using verification data and terminate the machine learning when the evaluation satisfies a given criterion. Alternatively, the termination condition may be set based on the error, or may be set based on the number of records to be processed, i.e., the number of times of learning. The machine learning unit 9 stores the image generation model at the time when the machine learning is terminated in the image generation model database 6.

[0060] The image generation model thus trained has a function of extracting a composition from a model image so that a design image has substantially the same composition as the model image, and also has a function of generating a realistic image from a line drawing so as to ensure the reality of the image.

[0061] In the design image generating unit 32, the image generation model is regulated to generate an image according to the information contained in the input prompt by referring to the information contained in the input prompt. "Along with the information contained in the prompt" refers to reflecting the information contained in the input information and the teacher image according to the degree of matching, by referring to the input information used when learning the image generation model and the teacher image associated with the input information, which has a high degree of matching with the information contained in the prompt. The image generation model extracts character information contained in at least one of the annotation information, user information, theme information, and type information input as the prompt, for example. The image generation model compares the degree of matching between all words contained in the character information and the information contained in the input information and the teacher image learned by the image generation model, and selects the teacher image associated with the input information with the highest degree of matching as a reference image.

[0062] The image generation model extracts characteristic information contained in the reference image as reference information, and applies a design to the model image by adding the reference information to the model image, thereby generating a design image. The reference information includes at least one of annotation information, user information, theme information, and type information, for example. Adding reference information to the model image refers to, for example, the image generation model extracting an object contained in the model image, selecting reference information that can be added to the extracted object, and converting at least a portion of the information about the object into reference information. The at least a portion of the information about the object and the reference information include, for example, the color, brightness, shape, size, position, range, etc. of the object.

[0063] Adding reference information to a model image may include, for example, deleting or modifying an object included in a model image based on the reference information of the reference image. The image generation model may, for example, not delete architectural structures from the 3D model included in the model image. The image generation model may, for example, be adjusted so that only furniture objects can be deleted from the 3D model included in the model image. In this case, the image generation model may distinguish between structures and furniture included in the model image based on annotation information. The structures here refer to configurations such as walls, pillars, ceilings, and floors that are necessary to define the floor plan.

[0064] In this way, the design image generating unit 32 generates an image including a 3D model whose design has been changed according to the information included in the prompt while maintaining many of the objects and the composition of the objects of the building displayed in the model image. That is, the model image generating unit 22 generates one design image for one user input operation and one model image. One design image is generated for one viewpoint. One design image displays a 3D model whose design has been changed and which can be viewed from a corresponding viewpoint. The design image generating unit 32 stores the generated design image in the server 4.

[0065] The third display control unit 33 displays on the design screen a design image display element including the design image generated by the design image generation unit 32. The third input acceptance unit 31 receives an input operation signal from the user, and the third display control unit 33 displays the design image display element on the screen of the user terminal 2, thereby transitioning the screen on the user terminal 2 from the capture screen to the design screen.

[0066] At least a design image is displayed in the design image display element. The third display control unit 33 acquires the design image from the design image generation unit 32 and displays the design image on the screen of the user terminal 2. The third display control unit 33 may display the design image display element so as to be larger than the main element. The third display control unit 33 may display the design image display element instead of the main element, the viewpoint display element, and the model image display element.

[0067] Fig. 2 is a diagram showing a general hardware configuration of the management device 5. As shown in Fig. 2, for example, the management device 5 includes, as physical components, a processor 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106.

[0068] The processor 101 executes an operating system and application programs. The main memory 102 is composed of, for example, a ROM and a RAM. The auxiliary memory 103 is composed of, for example, a hard disk or a flash memory, and generally stores a larger amount of data than the main memory 102. The auxiliary memory 103 stores an information processing program P for realizing each of the above-mentioned functional elements in the management device 5. The communication control unit 104 is composed of, for example, a wireless communication module. The input device 105 is composed of, for example, a keyboard, a mouse, or a touch panel. The output device 106 is composed of, for example, a monitor and a speaker.

[0069] Each functional element of the management device 5 is realized by loading an information processing program P onto the processor 101 or the main memory unit 102 and having the processor 101 execute the information processing program P. The information processing program P includes codes for realizing the machine learning unit 9, the model control unit 10, the capture control unit 20, and the design control unit 30. The processor 101 operates the communication control unit 104, the input device 105, or the output device 106 in accordance with the information processing program P, and reads and writes data in the main memory unit 102 or the auxiliary memory unit 103. Each functional element of the user terminal 2 is realized by this processing. Data or a database required for processing may be stored in the main memory unit 102 or the auxiliary memory unit 103.

[0070] The information processing program P may be provided in a state where it is fixedly recorded on a tangible recording medium such as a semiconductor memory. Alternatively, the information processing program P may be provided via a communication network as a data signal superimposed on a carrier wave. The provided information processing program P is stored in the auxiliary storage unit 103.

[0071] The management device 5 may be a computer that stores property information in the three-dimensional model database 3. For example, the management device 5 accesses the three-dimensional model database 3 via the communication network 7 and stores the latest property information in the three-dimensional model database 3, thereby newly registering or updating the property information.

[0072] The model screen, capture screen, and design screen displayed on the user terminal 2 will be described in order. First, the model screen displayed on the user terminal 2 will be described. As described above, the model screen is a screen for displaying a main element and annotation information. The model screen includes, for example, a main element that depicts a three-dimensional model indicated by property information corresponding to the inquiry information, and at least one of content information and viewpoint information.

[0073] Fig. 3 is a diagram showing an example of a model screen. The model screen 40 includes a content display element 61 as content information 60 of the annotation information. In the example shown in Fig. 3, the model screen 40 includes a main element 41. The first display control unit 14 displays the main element 41 drawn by the image drawing unit 13 on the model screen 40.

[0074] The main element 41 is a user interface capable of displaying a three-dimensional model 50 seen from a drawing viewpoint and displaying information about an object 51. The main element 41 is an element that draws the three-dimensional model 50 captured by a camera object moved to a preset drawing viewpoint in the image drawing unit 13 and drawn in a canvas area. The first display control unit 14 changes the user's drawing viewpoint for the three-dimensional model 50 in response to an input operation such as dragging or scrolling in the user terminal 2. At this time, the image drawing unit 13 captures the three-dimensional model 50 by a camera object in response to an input operation such as dragging or scrolling in the user terminal 2, and draws the three-dimensional model 50 in the canvas area.

[0075] The first display control unit 14 displays an image of the three-dimensional model 50 newly drawn by the image drawing unit 13 on the model screen 40 as a main element 41. This function allows the user to virtually move within the three-dimensional model 50 and visually check any part of the building from any position and angle. In the example shown in FIG. 3, an object 51 (a garbage disposer) of the building represented by the three-dimensional model 50 is displayed in the main element 41 so as to be located at the bottom center of the model screen 40.

[0076] The content display element 61 is a user interface that displays thumbnail images of objects and is movable to each viewpoint. The content display element 61 displays an object name field 62, at least one thumbnail display frame 63, and an object description field 64 as content information 60 corresponding to the viewpoint. In response to an input operation such as clicking, dragging, or scrolling on the user terminal 2, the first display control unit 14 displays the content information 60 of at least one object visible from the viewpoint. For example, the content display element 61 is displayed when the user clicks on the object 51 included in the main element 41 or its surroundings.

[0077] The name of the object is displayed in the object name column 62. The thumbnail display frames 63 are provided in the same number as the thumbnail images related to the prerecorded object, and are columns for displaying the thumbnail images. The object description column 64 displays text information about the object.

[0078] 4 is a diagram showing an example of a model screen. The model screen 40 includes a viewpoint display element 71 as viewpoint information 70 of the annotation information. In the example shown in FIG. 4, the model screen 40 includes a main element 41. The first display control unit 14 displays the main element 41 drawn by the image drawing unit 13 on the model screen 40. The method of displaying the main element 41 is the same as the example shown in FIG. 3.

[0079] The viewpoint display element 71 is a user interface that displays thumbnail images of each viewpoint and allows movement to each viewpoint. The viewpoint display element 71 displays a name 72 to clearly indicate that the viewpoint display element 71 is being displayed. The viewpoint display element 71 displays a list of thumbnail display frames 73 including an explanation for each viewpoint. In the example of FIG. 4, the thumbnail display frame 73 displays viewpoint information including a viewpoint name field 74 and a thumbnail image 75 corresponding to each of the multiple viewpoints. The first display control unit 14 displays the viewpoint information for each viewpoint in response to an input operation such as clicking, dragging, or scrolling on the user terminal 2.

[0080] The thumbnail display frames 73 are provided in the number of viewpoints, and are columns for displaying thumbnail images 75 stored in advance in the server 4. The thumbnail display frames 73 are also a user interface for changing the user's viewpoint with respect to the 3D model 50 to a viewpoint at which a thumbnail image 75 is displayed. In response to a user's operation of the thumbnail display frame 73 corresponding to a certain viewpoint, the user's viewpoint is moved to that viewpoint. As the user's viewpoint moves, the 3D model 50 as seen from the viewpoint after the movement is displayed on the model screen 40.

[0081] FIG. 5 is a diagram showing an example of a capture screen. The capture screen 42 is a screen that displays input information 80 including a model image, theme information, and type information. The capture screen 42 includes a model image display element 81 for acquiring a model image. In the example shown in FIG. 5, the capture screen includes a main element 41. The second display control unit 23 displays the main element 41 drawn by the image drawing unit 13 on the model screen 40. The method of displaying the main element 41 is the same as the example shown in FIG. 3.

[0082] The model image display element 81 is a user interface that displays a model image 82 and buttons 83 to 85 as functional UIs, and allows the user to operate to obtain a model image and set a design image. The model image display element 81 displays a model image 82 captured from a certain viewpoint through an input operation by the user. If the model image 82 has not yet been obtained in the 3D model 50 of the building currently being viewed by the user, the area of ​​the model image 82 shown in FIG. 5 is displayed as a blank space. At this time, the second display control unit 23 displays the model image display element 81 in response to an input operation such as clicking, dragging, or scrolling on the user terminal 2.

[0083] 5, the model image display element 81 displays, as functional UIs, for example, an acquisition button 83 for acquiring a model image, a save button 84 for saving the model image, and a first image generation button 85 for instructing generation of a design image. When the user clicks the acquisition button 83, a model image including a 3D model 50 of the same object as the main element 41 is generated by the model image generation unit 22. When the user clicks the save button 84, the generated model image is stored in the server 4. When the user clicks the first image generation button 85, a capture screen display element is displayed.

[0084] 6 is a diagram showing an example of a capture screen. The capture screen 42 includes a capture screen display element 86 for acquiring a design image. In the example shown in FIG. 6, the capture screen includes a main element 41 and a model image display element 81. The second display control unit 23 displays the main element 41 drawn by the image drawing unit 13 on the model screen 40. The method of displaying the main element 41 is the same as the example shown in FIG. 3.

[0085] The capture screen display element 86 may be displayed superimposed on the main element 41 and the model image display element 81. The capture screen display element 86 is a user interface that displays a model image 82, a theme information input field 87, a type information input field 88, and a second image generation button 89, and allows an operation to start generation of a design image. The capture screen display element 86 displays a model image 82 captured from a certain viewpoint by a user's input operation.

[0086] When the user clicks on the theme information input field 87, a list of theme names is displayed, and the user can select theme information according to his / her preference. When the user clicks on the type information input field 88, a list of type names is displayed, and the user can select the type information indicated by the model image 82. When the user clicks on the second image generation button 89, generation of a design image is started.

[0087] FIG. 7 is a diagram showing an example of a design screen. The design screen 43 includes a design image 44. The third display control unit 33 displays the design image 44 generated by the design image generation unit 32 on the design screen 43. For example, when the design image 44 is generated, the third display control unit 33 switches from the capture screen 42 to the design screen 43 and displays it. The design image 44 shown in FIG. 7 is an example of a design image generated by the design image generation unit 32 based on the model image 82 when the theme information is set to "Japanese modern" and the type information is set to "living room" in the capture screen display element 86 of the capture screen 42. The first display control unit 14, the second display control unit 23, and the third display control unit 33 switch and display the model screen 40, the capture screen 42, and the design screen 43 by a predetermined user input operation.

[0088] [System Operation] The operation of the management device 5 will be described with reference to Fig. 8, Fig. 9, and Fig. 10, and an information processing method according to this embodiment will be described. Fig. 8, Fig. 9, and Fig. 10 are flowcharts showing an example of the information processing method according to the embodiment. First, with reference to Fig. 8, control of a model screen on one user terminal 2 will be described. The control of the model screen shown in Fig. 8 is display control of a main element 41 of a model screen 40.

[0089] In step S11, the first display control unit 14 acquires the three-dimensional model 50 and viewpoint information of a viewpoint. In response to the first input receiving unit 11 receiving the inquiry information from the user terminal 2, the first display control unit 14 reads out property information corresponding to the inquiry information from the three-dimensional model database 3, and acquires the three-dimensional model 50 indicated by the property information. The first display control unit 14 also reads out viewpoint information of a viewpoint corresponding to the acquired property information from the server 4, and acquires the viewpoint information.

[0090] In step S13, the setting unit 12 generates a camera object. The setting unit 12 sets the imaging area of ​​the camera object based on the angle of view information of the viewpoint information acquired by the first display control unit .

[0091] In step S15, image rendering unit 13 moves the camera object to the viewpoint. Image rendering unit 13 moves the camera object to the position of the viewpoint in three-dimensional model 50 based on the viewpoint information of the viewpoint acquired by first display control unit 14.

[0092] In step S17, the setting unit 12 generates a canvas area. In one example, the setting unit 12 sets the canvas area based on the size of the canvas area stored in the server 4.

[0093] In step S19, the image drawing unit 13 draws the three-dimensional model 50 within the imaging area of ​​the camera object in the canvas area.

[0094] In step S21, the first display control unit 14 displays the three-dimensional model 50 drawn in the canvas area as a main element 41 on the model screen 40. In one example, the first display control unit 14 displays the main element 41 on the model screen 40 on the user terminal 2. The first display control unit 14 may display at least one of annotation information and a model image display element 81 according to the three-dimensional model 50 included in the main element 41.

[0095] As shown in step S23, the management device 5 repeats the processes from step S21 onwards until the model screen 40 is terminated (for example, until the model screen is closed on the user terminal 2, or until an operation to transition to a design screen is performed). For example, in response to a user's operation, the management device 5 performs the above-mentioned control of the model screen when moving to a new viewpoint.

[0096] Next, control of the capture screen 42 on one user terminal 2 will be described with reference to Fig. 9. The control of the capture screen 42 shown in Fig. 9 is display control of the model image display element 81 and the capture screen display element 86. The control of the model image 82 of the capture screen 42 shown in Fig. 9 can be performed simultaneously with the control of the main element 41.

[0097] In step S31, the second display control unit 23 acquires a model image. In response to the second input receiving unit 21 receiving an operation signal indicating that the acquire button 83 for the model image display element 81 has been pressed from the user terminal 2, the second display control unit 23 acquires the three-dimensional model 50 drawn in the canvas area as a model image. In one example, the second display control unit 23 acquires a model image including the same object as the main element 41 displayed on the model screen 40 on the user terminal 2.

[0098] In step S33, the second display control unit 23 displays the acquired model image in the model image display element 81 of the model screen 40 on the user terminal 2.

[0099] In step S35, the second display control unit 23 displays a capture screen display element 86 into which theme information and type information can be input. In one example, in response to the second input receiving unit 21 receiving from the user terminal 2 an operation signal indicating that the first image generation button 85 in the model image display element 81 has been pressed, the second display control unit 23 displays the capture screen display element 86 of the capture screen 42 on the user terminal 2. The management device 5 ends the capture screen 42 at the timing when, for example, generation of a design image is started and the design screen 43 is displayed.

[0100] Next, control of the design screen 43 on one of the user terminals 2 will be described with reference to Fig. 10. The control of the design screen 43 shown in Fig. 10 is control of generation of a design image and control of display of the design image.

[0101] In step S41, the design image generation unit 32 acquires a model image. The third input acceptance unit 31 receives an operation signal indicating that the second image generation button 89 has been pressed while the model image 82 is displayed in the capture screen display element 86 from the user terminal 2. In response to receiving the operation signal, the design image generation unit 32 acquires the model image as an image to be input to the image generation model.

[0102] In step S43, the design image generating unit 32 acquires theme information and type information as a prompt. The third input receiving unit 31 receives an operation signal from the user terminal 2 indicating that the second image generating button 89 has been pressed with each piece of information input into the theme information input field 87 and the type information input field 88 in the capture screen display element 86. In response to receiving the operation signal, the design image generating unit 32 acquires the theme information and type information.

[0103] In step S45, the design image generating unit 32 acquires annotation information as a prompt. The second input receiving unit 21 receives an operation signal indicating that the second image generation button 89 has been pressed from the user terminal 2. In response to receiving the operation signal, the design image generating unit 32 acquires annotation information associated with the object included in the acquired model image from the server 4. In one example, the design image generating unit 32 acquires annotation information associated with the object included in the model image from the server 4 based on at least one of the main element 41, the three-dimensional model 50, and the type information on the model screen 40 or the capture screen 42.

[0104] In step S47, the design image generation unit 32 acquires user information as a prompt. The second input acceptance unit 21 receives an operation signal indicating that the second image generation button 89 has been pressed from the user terminal 2. In response to receiving the operation signal, the design image generation unit 32 acquires user information corresponding to the user operating the user terminal 2 from the server 4. In one example, based on login information for the information processing system 1 input by the user to the user terminal 2 when viewing the main element 41, the design image generation unit 32 acquires user information including a user ID corresponding to the login information from the server 4.

[0105] In step S49, the design image generating unit 32 generates a design image by inputting a model image as an image to the image generation model, and inputting annotation information, user information, theme information, and type information as prompts. Note that, at least before step S49 is started, the design image generating unit 32 acquires the image generation model learned by the machine learning unit 9 and stored in the image generation model database 6 from the image generation model database 6.

[0106] In step S51, the third display control unit 33 outputs the design image 44. In one example, the third display control unit 33 displays the design image 44 on the design screen 43. In another example, the third display control unit 33 causes the server 4 to store the design image 44.

[0107] In addition, in the control of the design screen 43 shown in Fig. 10, at least one of the annotation information acquisition process (step S45) and the user information acquisition process (step S47) may not be executed. In the control of the design screen 43 shown in Fig. 10, the design image generation unit 32 may generate a design image by inputting a model image as an image to the image generation model, and inputting at least one of annotation information, user information, theme information, and type information as a prompt.

[0108] [effect] As described above, an information processing system according to one embodiment of the present disclosure includes at least one processor, and the at least one processor generates a design image for the three-dimensional model using an image generation model that generates a new image based on an image and a prompt, a model image that displays a three-dimensional model of a building, theme information that indicates design indicators, and type information that indicates attributes of the building shown by the model image.

[0109] An information processing method according to one embodiment of the present disclosure is an information processing method executed by an information processing system having at least one processor, and includes a step of generating a design image for the three-dimensional model using the image generation model based on an image generation model that generates a new image based on an image and a prompt, a model image that displays a three-dimensional model of a building, theme information that indicates design indicators, and type information that indicates attributes of the building shown by the model image.

[0110] An information processing program according to one embodiment of the present disclosure causes a computer to execute a step of generating a design image relating to a three-dimensional model using an image generation model based on an image and a prompt, a model image displaying a three-dimensional model of a building, theme information indicating design indicators, and type information indicating attributes of the building indicated by the model image.

[0111] According to this embodiment, the model image 82 is input as an input image, and the theme information and type information are input as prompts to the image generation model. This generates a design image 44 related to the 3D model 50 of the building based on the model image 82, the theme information, and the type information. This makes it possible to generate an image in which a design appropriate to the theme is applied to an image displaying the 3D model of the building.

[0112] When viewing (viewing) the inside of a building, it may be necessary to know the appearance (conditions) of the building from a viewpoint inside the building. For example, in the conventional technology described in Patent Document 1, only a 3D model seen from one viewpoint set by the user is displayed on the screen, so it is difficult to confirm the appearance of a 3D model with various designs or the appearance of a 3D model with a design desired by the user. In addition, in the conventional technology, when generating an image in which a 3D model is subjected to various designs, it is necessary to prepare design data applicable to the 3D model in advance, which may require a lot of man-hours.

[0113] In addition, a model that automatically generates an image with a design applied to an original input image is known as a conventional technology. When considering the design of the exterior or interior of a building, an image captured of an existing building may be used as the input image. For this reason, a model that automatically generates an image with a design applied cannot be applied to a building that has not yet been completed, and there is a possibility that the design of the exterior or interior of the building cannot be fully considered before the building is handed over to the user, such as during the planning or construction of the building.

[0114] Furthermore, depending on the above-mentioned model, a design not desired by the user or a design not suitable for the exterior or interior of the building may be automatically applied to the input image. In addition, even if a 3D model image of a building can be applied to the above-mentioned model, the model may alter parts of the building that cannot be changed, such as the structure, fixtures, and layout of the building. Therefore, it may not be possible to obtain an image with an appropriate design that conforms to the structure of the building to be completed.

[0115] On the other hand, the information processing system 1 acquires an image of a 3D model of a building as an input image, acquires theme information and type information as prompts, and applies these data to an image generation model, thereby enabling the generation of an image in which a design appropriate to the theme is applied to an image displaying the 3D model of a building, regardless of whether the target building is completed or not.

[0116] In another embodiment of the information processing system, at least one processor may obtain content information of a three-dimensional model related to an object of the building shown in the model image, and generate a design image by an image generation model based on the content information. In this way, by utilizing the content information related to the object of the building shown in the model image when generating a design image, the image generation model can more accurately recognize the object in the model image and apply a more appropriate design. For example, compared to directly recognizing the object from the graphic information of the model image, additional information such as character information included in the content information can also be used to generate the design image, making it possible to more accurately estimate the object shown in the model image. Furthermore, for example, after accurately recognizing multiple objects shown in the model image, an appropriate design can be selected for the multiple recognized objects.

[0117] In another embodiment of the information processing system, at least one processor may obtain viewpoint information on a viewpoint indicating a position from which to view the building in the three-dimensional model, and generate a design image by an image generation model based on the viewpoint information for the model image. In this way, by utilizing the viewpoint information for the model image when generating a design image, the image generation model can more accurately recognize objects in the model image and apply a more appropriate design. For example, by using additional information such as the position and angle of view for the building included in the viewpoint information in addition to the theme information in generating the design image, the object of the building shown in the model image can be more accurately recognized.

[0118] In an information processing system according to another embodiment, at least one processor may generate a model image for a viewpoint based on viewpoint information related to a viewpoint indicating a position from which a building in a three-dimensional model is viewed, and generate a design image by an image generation model based on the generated model image. According to such an embodiment, since a model image can be generated according to a viewpoint, various types of design images can be generated for a three-dimensional model of a building. For example, when a user is viewing a three-dimensional model, a model image can be generated at an arbitrary viewpoint according to an input operation of the user, so that the information processing system 1 can generate a design image according to a model image related to an arbitrary viewpoint.

[0119] Also, for example, a user viewing a three-dimensional model from a certain viewpoint can display objects necessary for the design, or display the three-dimensional model with unnecessary objects hidden. This allows a model image to be generated based on a three-dimensional model including objects desired to be displayed in the design image, thereby increasing the possibility that a design image that meets the needs of the user will be generated, and increasing user satisfaction.

[0120] In an information processing system according to another embodiment, at least one processor may generate a camera object having an imaging area set therein and a canvas area in which a three-dimensional model can be drawn, move the camera object to a viewpoint, draw the three-dimensional model in the imaging area of ​​the camera object in the canvas area, and generate an image displaying the three-dimensional model drawn in the canvas area as a model image. In this way, the three-dimensional model drawn in the canvas area can be generated as a model image.

[0121] In another embodiment of the information processing system, the theme information may include multiple design indicators provided by a contractor in charge of at least one of the design and construction of the building. In this way, not only general themes but also theme information in accordance with the building related to the model image is input as a prompt to the image generation model, so that a realistic design image can be provided to the user. For example, a standard specification theme (style) coordinated by a manufacturer (company) may be predefined. A user of the building selects a favorite theme from the standard specification themes, and construction of the building is started based on the selected theme. In this embodiment of the information processing system 1, a design image is generated using the theme information as one of the prompts, so that a user can imagine the building that will actually be constructed in more detail by checking the design image, shortening the communication between the user, the designer, and the contractor, and shortening the construction period.

[0122] In another embodiment of the information processing system, at least one processor may acquire user information indicating user characteristics related to the building in the three-dimensional model, and generate a design image by the image generation model based on the user information. In this way, by utilizing the user information of the user who wishes to acquire the design image when generating the design image, the image generation model can apply a more appropriate design to the model image according to the user's preferences, tastes, etc. For example, by using additional information, namely persona information, included in the user information, in the design image generation, the design characteristics of the building, such as color, brightness, layout, and object placement, can be made to be the user's preferred characteristics.

[0123] In an information processing system according to another embodiment, the image generation model may be a model constructed by machine learning based on input information including a model image, theme information, and type information, and a teacher image corresponding to the input information. Since the image generation model is trained in advance by machine learning in this way, a design image designed according to the information included in the prompt can be appropriately obtained.

[0124] In the information processing system according to another embodiment, the at least one processor may display the design image on a screen of the display device. In this manner, the design image is displayed on the display device, thereby enabling the user to view the design image.

[0125] [Variations] The present disclosure has been described in detail above based on the embodiments. However, the present disclosure is not limited to the above embodiments. The present disclosure can be modified in various ways without departing from the spirit and scope of the present disclosure.

[0126] The management device 5 does not need to include the model control unit 10. The second display control unit 23 may display a model image that is acquired in advance by the management device 5 or stored in the server 4 on the capture screen 42. In this case, the model image display element 81 does not need to be displayed on the capture screen 42.

[0127] The third display control unit 33 may display elements related to the design image 44 other than the design image 44 on the design screen 43. FIG. 11 is a diagram showing an example of a design screen according to a modified example. The design screen 43A displays design information 90 including a design image. In one example, the design screen 43A includes a design image display element 91 for displaying a model image and a design image. The design image display element 91 may be displayed superimposed on the main element and the model image display element. The design image display element 91 is a user interface that displays the model image 82, the design image 44, an image display adjustment button 92, a result explanation field 93, a regenerate button 94, and a store button 95, and allows operations related to confirmation and storage of the design image.

[0128] The design image display element 91 displays a model image 82 captured from a certain viewpoint by a user's input operation. The design image display element 91 displays a design image 44 generated by the design control unit 30 based on the model image 82. As the image display adjustment button 92 is slid up and down, the displayed sizes of the model image 82 and the design image 44 are changed. This allows the user to easily confirm the correspondence between the model image 82 and the design image 44 regarding the design of the object.

[0129] The result explanation field 93 displays the theme information and type information selected by the capture screen display element 86. When the user clicks the Regenerate button 94, generation of the design image is restarted. When the user clicks the Store button 95, the design image 44 is stored in the server 4.

[0130] The machine learning unit 9 may generate a plurality of image generation models according to the input information and the type of teacher image, train each image generation model, and store the image generation model in the image generation model database 6. The machine learning unit 9 may train an image generation model for each contractor in charge of at least one of the design and construction of a building, or for each type of building or building provided by the contractor. The type of building refers to a group classified according to the structure or use, such as a detached house, an apartment, an office building, or a public facility. At this time, the design image generation unit 32 may acquire an image generation model corresponding to the property information or input information of the building from the plurality of image generation models stored in the image generation model database 6. This allows the information processing system 1 to obtain a design image with a more appropriate design according to the contractor or building related to the building. In addition, the third input acceptance unit may accept an input operation of the user, and the design image generation unit 32 may acquire an image generation model according to the result of the input operation of the user from the plurality of image generation models stored in the image generation model database 6. This allows the information processing system 1 to obtain a design image with a design appropriate for the user.

[0131] The machine learning unit 9 may be a part of the information processing system 1A, or may be a computer system different from the information processing system 1A. Since the trained model is portable between computer systems, the information processing system 1 can use an image generation model generated in another computer system. FIG. 12 is a diagram showing an example of the overall configuration of the information processing system 1A according to the modified example. In the information processing system 1A according to the modified example, the management device 5A may not include the machine learning unit 9. In this case, the design image generation unit 32 may acquire an image generation model stored in the image generation model database 6 in a pre-trained state, and generate a design image.

[0132] Various exemplary embodiments included in the present disclosure will now be described in the following [Form 1] to [Form 11].

[0133] [Form 1] at least one processor; The at least one processor generates a design image related to the three-dimensional model by using an image generation model that generates a new image based on an image and a prompt, a model image that displays a three-dimensional model of the building, theme information that indicates a design indicator, and type information that indicates an attribute of the building that is indicated by the model image. Information processing system. [Form 2] The at least one processor: Acquire content information of a three-dimensional model related to the object of the building shown in the model image; generating the design image based on the content information using the image generation model; The information processing system according to [Mode 1]. [Form 3] The at least one processor: Obtain viewpoint information regarding a viewpoint indicating a position from which the building is viewed in the three-dimensional model; generating the design image using the image generation model based on the viewpoint information for the model image; The information processing system according to [Form 1] or [Form 2]. [Form 4] The at least one processor: generating the model image for a viewpoint based on viewpoint information regarding a viewpoint indicating a position from which the building is viewed in the three-dimensional model; generating the design image using the image generation model based on the generated model image; The information processing system according to any one of [Mode 1] to [Mode 3]. [Form 5] The at least one processor: generating a camera object having a set imaging area and a canvas area on which the three-dimensional model can be drawn; moving the camera object to the viewpoint; drawing the three-dimensional model within the imaging area of ​​the camera object on the canvas area; generating an image displaying the three-dimensional model drawn on the canvas area as the model image; The information processing system according to [Mode 4]. [Form 6] The theme information includes a plurality of design indexes provided by a contractor in charge of at least one of the design and construction of the building, The information processing system according to any one of [Mode 1] to [Mode 5]. [Form 7] The at least one processor: acquiring user information indicating a user's characteristics related to the building in the three-dimensional model; generating the design image by the image generation model based on the user information; The information processing system according to any one of [Mode 1] to [Mode 6]. [Form 8] The image generation model is a model constructed by machine learning based on at least one of the theme information and the type information, as well as input information including the model image and a teacher image corresponding to the input information. The information processing system according to any one of [Mode 1] to [Mode 7]. [Form 9] The at least one processor displays the design image on a screen of a display device. The information processing system according to any one of [Mode 1] to [Mode 8]. [Form 10] 1. An information processing method executed by an information processing system having at least one processor, comprising: A step of generating a design image relating to the three-dimensional model by the image generation model based on an image and a prompt, a model image displaying a three-dimensional model of a building, theme information indicating design indicators, and type information indicating attributes of the building indicated by the model image. An information processing method comprising: [Form 11] A step of generating a design image relating to the three-dimensional model by the image generation model based on an image and a prompt, a model image displaying a three-dimensional model of a building, theme information indicating design indicators, and type information indicating attributes of the building indicated by the model image. An information processing program that causes a computer to execute the above. [Explanation of symbols]

[0134] 1,1A...information processing system, 9...machine learning unit, 10...model control unit, 20...capture control unit, 30...design control unit, 44...design image, 50...3D model, 51...object, 60...content information, 70...viewpoint information, 80...input information, 82...model image, 101...processor, ID...building, P...information processing program.

Claims

1. At least one processor; The at least one processor generates a design image for the three-dimensional model using an image generation model that generates a new image based on an image and a prompt, a model image that displays a three-dimensional model of a building, theme information that indicates a design indicator, and type information that indicates an attribute of the building that the model image indicates, based on the image generation model. Information processing system.

2. The at least one processor acquiring content information of a three-dimensional model related to the object of the building shown in the model image; generating the design image based on the content information using the image generation model; The information processing system according to claim 1 .

3. The at least one processor Obtaining viewpoint information relating to a viewpoint indicating a position from which the building is viewed in the three-dimensional model; generating the design image using the image generation model based on the viewpoint information for the model image; 3. The information processing system according to claim 1 or 2.

4. The at least one processor generating the model image for a viewpoint based on viewpoint information relating to a viewpoint indicating a position from which the building is viewed in the three-dimensional model; generating the design image using the image generation model based on the generated model image; 3. The information processing system according to claim 1 or 2.

5. The at least one processor generating a camera object having a set imaging area and a canvas area on which the three-dimensional model can be drawn; moving the camera object to the viewpoint; drawing the three-dimensional model within the imaging area of ​​the camera object on the canvas area; generating an image displaying the three-dimensional model drawn on the canvas area as the model image; 5. The information processing system according to claim 4.

6. The theme information includes a plurality of design indexes provided by a contractor in charge of at least one of the design and construction of the building, 3. The information processing system according to claim 1 or 2.

7. The at least one processor acquiring user information indicating a user's characteristics related to the building in the three-dimensional model; generating the design image by the image generation model based on the user information; 3. The information processing system according to claim 1 or 2.

8. The image generation model is a model constructed by machine learning based on at least one of the theme information and the type information, as well as input information including the model image and a teacher image corresponding to the input information.

3. The information processing system according to claim 1 or 2.

9. The at least one processor displays the design image on a screen of a display device.

3. The information processing system according to claim 1 or 2.

10. 1. An information processing method executed by an information processing system having at least one processor, comprising: A step of generating a design image relating to the three-dimensional model by the image generation model, which generates a new image based on the image and the prompt, based on a model image displaying a three-dimensional model of the building, theme information indicating design indicators, and type information indicating attributes of the building indicated by the model image. An information processing method comprising:

11. A step of generating a design image relating to the three-dimensional model by the image generation model, which generates a new image based on the image and the prompt, based on a model image displaying a three-dimensional model of the building, theme information indicating design indicators, and type information indicating attributes of the building indicated by the model image. An information processing program that causes a computer to execute the above.

Images