Home decoration effect picture generation method and device, equipment and storage medium

Abstract

The invention provides a home decoration effect picture generation method and device, equipment and a storage medium. Relates to the technical field of artificial intelligence, in particular to the fields of image processing, deep learning and computer vision. The method comprises the following steps: acquiring a first room background image and a home decoration product image; performing feature recognition on the first room background image to obtain target feature information; determining a target home decoration product in the home decoration product graph; displaying a target home decoration product in the first room background image, and in response to a detected interaction operation for the target home decoration product, synthesizing the first room background image and the target home decoration product into a target image according to the interaction operation; and based on the target feature information and the target image, utilizing a large language model to obtain a home decoration effect picture. According to the scheme disclosed by the invention, the home decoration effect of drawing of the large language model can be accurately controlled, and the home decoration product and the room background can be more naturally fused by utilizing the large language model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of image processing, deep learning, and computer vision. Background Art

[0002] In the home decoration design industry, in order to view the effect of placing home decoration in a house, it is necessary to generate a rendering of the home decoration placed in the house. The renderings currently used in home decoration design drawings simply superimpose the home decoration products and the room background, and cannot accurately present the home decoration effect. Summary of the invention

[0003] The present disclosure provides a method, device, equipment and storage medium for generating a home decoration rendering to solve or alleviate one or more technical problems in the prior art.

[0004] In a first aspect, the present disclosure provides a method for generating a home decoration effect diagram, comprising:

[0005] Get the first room background image and home decoration product images;

[0006] Perform feature recognition on the background image of the first room to obtain target feature information;

[0007] Identify target home improvement products in the home improvement product map;

[0008] displaying the target home furnishing product in the first room background image, and in response to detecting an interactive operation for the target home furnishing product, synthesizing the first room background image and the target home furnishing product into a target image according to the interactive operation;

[0009] Based on the target feature information and target image, a large language model is used to obtain the home decoration renderings.

[0010] In a second aspect, the present disclosure provides a device for generating a home decoration effect diagram, comprising:

[0011] An acquisition module, used to acquire a first room background image and a home decoration product image;

[0012] A feature recognition module is used to perform feature recognition on the first room background image to obtain target feature information;

[0013] A home decoration determination module, used to determine the target home decoration product in the home decoration product map;

[0014] A home decoration adjustment module, configured to display a target home decoration product in a first room background image, and in response to detecting an interactive operation for the target home decoration product, synthesize the first room background image and the target home decoration product into a target image according to the interactive operation;

[0015] The first image generation module is used to obtain a home decoration rendering based on target feature information and a target image using a large language model.

[0016] In a third aspect, an electronic device is provided, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0021] The beneficial effects of the technical solution provided by the present disclosure include at least: the home decoration effect of the large language model can be accurately controlled, and the large language model can be used to more naturally integrate the home decoration products with the room background to accurately present the home decoration effect.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0024] Figure 1 It is a flowchart of a method for generating a home decoration effect diagram according to an embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of an application example of an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of another application example of the embodiment of the present disclosure;

[0027] Figure 4 is a schematic block diagram of a device for generating a home decoration effect diagram provided by an embodiment of the present disclosure;

[0028] Figure 5is a schematic block diagram of a device for generating a home decoration effect diagram provided by another embodiment of the present disclosure;

[0029] Figure 6 is a schematic block diagram of a device for generating a home decoration effect diagram provided by another embodiment of the present disclosure;

[0030] Figure 7 is a schematic block diagram of a device for generating a home decoration effect diagram provided by another embodiment of the present disclosure;

[0031] Figure 8 The block diagram is a block diagram of an electronic device used to implement the method for generating a home decoration effect diagram according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0033] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0034] In the related art, in order to generate renderings of home decoration design, it is necessary to use an image generation model to redesign and generate new room renderings based on the interior room reference map, required decoration style and prompt words provided by the user. The user is required to have a large amount of additional text description of the output image, and areas beyond the user's expectations may be redrawn. In addition, the user cannot specify the home decoration products to be drawn or the existing decorations that need to be replaced, and cannot customize the decoration style of the generated image, which results in the home decoration design renderings not being compliant.

[0035] In order to solve at least one of the above problems, the embodiments of the present disclosure provide a color matching method for a floor plan, which can perform feature recognition on a first room background image to obtain target feature information, and determine the target home improvement product in the home improvement product image, and then synthesize the first room background image and the target home improvement product into a target image based on an interactive operation. Finally, the home improvement effect of the output image can be controlled through the target feature information and the target image, and the home improvement product can be more naturally integrated with the room background by using a large language model to obtain a home improvement effect image.

[0036] Figure 1A schematic diagram of a method for generating a home decoration rendering provided by an embodiment of the present disclosure is shown. The method can be applied to a device for generating a home decoration rendering, and the device can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal can be a user equipment (User Equipment, UE) such as a mobile device, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the method may include the following steps S110 to S150.

[0037] Step S110, obtaining a first room background image and a home decoration product image.

[0038] The embodiments of the present disclosure can be applied to a tool platform for home decoration design. The first room background image can be understood as a scene image where home decoration products need to be placed. The home decoration product image may include images of one or more home decoration products, wherein the one or more home decoration products include home decoration products that need to be placed in the first room background image.

[0039] Exemplarily, the first room background image and the home decoration product image may be various types of images such as line drawings, actual photos, and renderings.

[0040] Optionally, the first room background image and the home decoration product image may come from photos taken by the user in real time, photo albums, candidate images provided by electronic devices, etc.

[0041] Optionally, the user can upload the first room background image and the home decoration product image through a user interaction interface, and the user interaction interface can be displayed on a display device of the electronic device executing the above method or on other user devices connected to the electronic device. The electronic device can obtain the first room background image and the home decoration product image uploaded by the user.

[0042] Alternatively, the electronic device may also provide the user with a plurality of candidate images of the first room background image and the home decoration product image, and the user may select the first room background image and the home decoration product image from the candidate images.

[0043] Step S120: Perform feature recognition on the first room background image to obtain target feature information.

[0044] Optionally, the target feature information may include at least one of the following: decoration type, room type, apartment type, and furniture type. Decoration type may include rough housing, fully decorated housing, etc. Room type may include living room, dining room, bedroom, study, kitchen, bathroom, etc. Apartment type may include empty apartment type and non-empty apartment type. Furniture type may include sofa, coffee table, desk, bedside table, table lamp, TV, etc.

[0045] Optionally, based on a back-end image classification model (e.g., BLIP-2), the first room background image can be classified by feature recognition in one or more dimensions, such as decoration type, room type, apartment type, and furniture type, so as to determine the features of one or more dimensions in the first room background image. These features provide a more accurate semantic description of the first room background image, thereby ensuring the basic information of the first room background image to the greatest extent.

[0046] Step S130: determining a target home furnishing product in the home furnishing product map.

[0047] In the disclosed embodiment, the target home furnishing product can be understood as the home furnishing product that needs to be placed in the background image of the first room. The target home furnishing product can be any home furnishing product in the home furnishing product image. After determining the target home furnishing product, the home furnishing product image can be cut out to retain only the target home furnishing product.

[0048] Exemplarily, the target home furnishing product may be a soft furnishing product in the home furnishing product picture, such as a sofa, a table, home appliances, etc., or may be a hard furnishing product in the home furnishing product picture, such as a door, a window, a suspended ceiling, etc.

[0049] Optionally, the home improvement product map may include multiple home improvement products, and the user may select a target home improvement product from the multiple home improvement products, or the electronic device may automatically recommend the target home improvement product to the user.

[0050] In one embodiment, when a home improvement product is included in the home improvement product map, the electronic device may automatically determine the home improvement product as a target home improvement product and only retain the target home improvement product in the home improvement product map.

[0051] In the embodiment of the present disclosure, the order in which the electronic device executes step S130 and step S120 is not limited, that is, step S130 may be executed first and then step S120, or step S120 may be executed first and then step S130, or step S130 and step S120 may be executed in parallel.

[0052] Step S140: displaying the target home furnishing product in the first room background image, and in response to detecting an interactive operation for the target home furnishing product, synthesizing the first room background image and the target home furnishing product into a target image according to the interactive operation.

[0053] In the disclosed embodiment, the electronic device can display the first room background image and the target home furnishing product to the user in the user interaction interface, and the user can perform interactive operations on the target home furnishing product (for example, dragging, zooming, rotating, translating, etc. within the range of the first room background image), and the size, position and viewing angle of the target home furnishing product in the first room background image can be controlled according to the interactive operation, and then the user can end the interactive operation by clicking to confirm, and based on the click to confirm the size, position and viewing angle of the target home furnishing product in the first room background image, synthesize the first room background image and the target home furnishing product to obtain the target image, that is, the target image can be the synthesis result of the first room background image and the target home furnishing product after the interactive operation is completed. Exemplarily, according to the interactive operation, the multiple pixels occupied by the target home furnishing product in the coordinate space of the first room background image can be determined, and the pixel values ​​of the target home furnishing product are used to cover or replace the pixel values ​​of these pixels in the first room background image, thereby obtaining the target image.

[0054] Step S150: Based on the target feature information and the target image, a home decoration rendering is obtained using a large language model.

[0055] In the embodiment of the present disclosure, the large language model may be a deep learning model. Exemplarily, the deep learning model may include a graph reasoning engine.

[0056] Optionally, the target feature information and the target image can be used to constrain the output effect of the large language model, and the target home improvement product can be AI (Artificial Intelligence) fused with the first room background image to generate a home improvement effect image. Optionally, the target feature information can be used to construct prompt words to use the prompt words to guide the large language model to AI fuse the target home improvement product with the first room background image according to certain features. Optionally, the prompt words can also guide the large language model not to change the apartment structure of the original room background.

[0057] Optionally, in the above method, a binary image can also be determined according to the interactive operation, and the binary image can include a first mask image, and the first mask image has the same size as the first room background image. The binary image can be obtained by setting the pixel values ​​of the pixels occupied by the target home improvement product in the first room background image to the first value, and setting the pixel values ​​of other pixels in the first room background image to the second value. Optionally, the first value is 1 and the second value is 0; or, the first value is 0 and the second value is 255. The first mask image can be used to determine the area where the target home improvement product is located. Optionally, the large model can be guided to treat the area of ​​the target home improvement product as the area that needs to be redrawn, and other areas as areas that should be retained as they are, so as to further improve the accuracy and naturalness of the fusion of the target home improvement product and the first room background image in the home improvement rendering.

[0058] In one embodiment, the generated home decoration renderings can be input into a large language model, and the user can continuously adjust the configuration parameters of the large language model until the image output by the large language model achieves a satisfactory effect.

[0059] It can be seen that according to the technical solution of the embodiment of the present disclosure, the feature recognition of the first room background image can be performed to obtain the target feature information, and the target home improvement product can be determined in the home improvement product image, and then the first room background image and the target home improvement product can be synthesized into a target image according to the interactive operation. Since the target image is not directly used as the final home improvement rendering, but the output effect of the large language model is constrained by the target feature information and the target image, the target home improvement product is AI-fused with the first room background image to generate a home improvement rendering. While accurately controlling the home improvement effect of the large language model output, the home improvement product can be more naturally fused with the room background, overcoming the problem of unnatural synthesis that exists when the first room background image and the target home improvement product in the target image are directly synthesized.

[0060] In some embodiments, obtaining the first room background image may include:

[0061] Get the background image of the second room;

[0062] In response to detecting a smearing operation on the interfering content in the second room background image, determining a smearing area in the second room background image;

[0063] The information of the painted area is eliminated in the background image of the second room to obtain the background image of the first room.

[0064] In the disclosed embodiment, the electronic device can receive a second room background image uploaded by a user, and display the second room background image to the user on a user interaction interface. The user can paint over the interference content in the second room background image on the user interaction interface to obtain the first room background image, that is, the first room background image can be the room background image after eliminating the interference content.

[0065] Optionally, the interference content may be some elements that the user does not need in the second room background image, such as decoration tools and sundries that appear in the second room background image. And / or, the interference content may also be home furnishing products that need to be replaced in the second room background image, such as furniture such as tables and chairs that already exist in the second room background image and need to be replaced with new furniture.

[0066] In one embodiment, after the user performs a smear operation on the interfering content, a second mask image can be generated. The smeared area can be clearly marked through the second mask image, thereby accurately eliminating the smeared area in the second room background image. The back end can use image restoration technology (such as the lama model) to fill the eliminated smeared area, and the filled image is returned to the front end.

[0067] According to the technical solution of the embodiment of the present disclosure, removing interfering content allows users to pre-process the room background image more flexibly, thereby ensuring the effect and quality of the final image to the greatest extent.

[0068] In some embodiments, determining a target home improvement product in a home improvement product map includes:

[0069] Perform semantic segmentation on the home decoration product image to determine multiple home decoration products in the home decoration product image;

[0070] Determine the target home improvement products based on multiple home improvement products.

[0071] In the embodiments of the present disclosure, any method in the related art can be used to perform semantic segmentation on the home furnishing product image. For example, a deep learning model can be used to perform semantic segmentation on the home furnishing product image.

[0072] In one embodiment, a oneformer model may be used to perform semantic segmentation on home furnishing product images. The oneformer model is good at performing segmentation in a semantic manner. For example, after inputting feature labels such as floor and wall, the area indicated by the label in the home furnishing product image can be segmented out.

[0073] In one embodiment, semantic segmentation can be performed on home improvement product images at the back end, and the results of the semantic segmentation can be sent to the front end, which can reduce the computing power requirements of the front-end device and improve the efficiency of semantic segmentation.

[0074] Optionally, the user may manually select a target home improvement product from a plurality of home improvement products, or the electronic device may automatically recommend the target home improvement product to the user.

[0075] In one embodiment, when a user manually selects a target home improvement product, the front end may extract data matching the location coordinates selected by the user as the target home improvement product based on the location coordinates selected by the user and the semantic segmentation data of multiple home improvement products.

[0076] According to the technical solution of the embodiment of the present disclosure, by performing semantic segmentation on the home improvement product image, the home improvement products in the home improvement product image can be accurately identified, and precise boundaries can be provided for each home improvement product, thereby accurately determining the target home improvement product.

[0077] In some embodiments, based on a plurality of home improvement products, determining a target home improvement product includes:

[0078] In response to a click operation on a pixel point or an area in the home improvement product image failing to hit a plurality of home improvement products, the home improvement product image is segmented based on the pixel points or the area using a segmentation model to obtain a target home improvement product in the home improvement product image.

[0079] In an embodiment of the present disclosure, a user can manually select a target home improvement product in the home improvement product picture. For example, the user can click on any point or any area in the home improvement product picture. When the click operation on a pixel point or area in the home improvement product picture hits a home improvement product, the hit home improvement product can be determined as the target home improvement product.

[0080] In the disclosed embodiment, if there is no corresponding home improvement product in the pixel point or area corresponding to the user's click operation, the home improvement product image can be segmented again using the segmentation model. For example, the foreground and background can be segmented based on the pixel point or area corresponding to the click operation, and the segmented foreground can be used as the target home improvement product in the home improvement product image.

[0081] Specifically, the segmentation model can use a region growing algorithm to segment the home improvement product images. The region growing algorithm is a classic region-based segmentation algorithm. The pixel point or region corresponding to the click operation can be used as a seed growth point, and the neighboring pixels with the same or similar properties as the seed are merged into the growing area, and the above steps are repeated as new seed points until no new pixel points are added, and the region growing is completed.

[0082] In one embodiment, the segmentation model may include a SAM model, which is good at performing segmentation in an interactive manner and can segment the home improvement product image based on a click operation on a pixel point or area in the home improvement product image.

[0083] In one embodiment, the backend can semantically segment the home furnishing product image based on the oneformer model, segmenting a series of home furnishing elements including sofas, chairs, wardrobes, chandeliers, televisions, etc., and return the segmented data structure to the frontend; at the same time, the frontend lazily loads the sam model and uses it as a supplementary solution when the oneformer model fails to cover the pixel points or areas corresponding to the click operation.

[0084] According to the technical solution of the embodiment of the present disclosure, even if the user's click operation does not directly hit any segmented home improvement product, the user's intention can be effectively identified and the target home improvement product in the home improvement product map can be accurately determined.

[0085] In some embodiments, the target feature information includes the decoration type corresponding to the first room background image; based on the target feature information and the target image, a home decoration effect image is obtained using a large language model, including:

[0086] When the decoration type corresponding to the first room background image is a rough house, determining a first style model based on the home decoration product image and / or user input information;

[0087] Based on the first style model and the target image, a large language model is used to obtain the home decoration renderings.

[0088] Exemplarily, the first style model may be determined based on user input information. Optionally, the user input information may include information such as an image with a decoration style, text describing the decoration style, or a vector describing the decoration style, and the first style model may be determined based on the user input information.

[0089] For example, the first style model may be automatically determined according to the design style of the home decoration product image, for example, by identifying the design style of the home decoration product image and determining the first style model based on the design style.

[0090] Exemplarily, the first style model may also be determined based on both the user input information and the home decoration product picture. For example, the first style model may be determined by combining the description text input by the user and the design style of the home decoration product picture.

[0091] Optionally, the first style model may include a LoRa model, which is a fine-tuning technology for a large language model. The LoRa model can be used to fine-tune the large language model so that the home decoration rendering output by the large language model can include the decoration style design of the room background.

[0092] In the disclosed embodiment, a large language model can be used to perform AI fusion on the apartment structure, the first style model, and the target home improvement products in the first room background image to generate a home improvement rendering. The decoration style design of the room background in the home improvement rendering can be completed without changing the apartment structure of the original room background. Not only can the decoration effect of the room background be previewed, but also the effect of placing the home improvement can be more vividly displayed.

[0093] According to the technical solution of the embodiment of the present disclosure, the decoration style design of the room background in the home decoration rendering can be completed for the decoration type corresponding to the first room background image, which is a rough house, thereby improving the display effect of the home decoration rendering.

[0094] In the embodiment of the present disclosure, a variety of different ways of determining the first style model are also provided, and an exemplary way is provided below.

[0095] Method 1: Determine the first style model based on the home decoration product image and / or user input information, including:

[0096] Display multiple style models in the user interface;

[0097] In response to detecting a selection operation of a user among the multiple style models, a first style model among the multiple style models is determined.

[0098] In an embodiment of the present disclosure, the electronic device may display an image corresponding to each style model and / or text describing the style model to the user, and the user may select the image and / or text in the user interaction interface to select the first style model among multiple style models.

[0099] Optionally, the multiple style models may include a style model preset in the electronic device, such as a style model in a database.

[0100] According to the above embodiment, the user can flexibly select the first style model from multiple style models to generate a home decoration rendering according to user needs, and further accurately control the background design style of the large language model output.

[0101] In some embodiments, the multiple style models include a second style model, and the second style model is trained based on a style picture training set and training parameters input by a user.

[0102] In the disclosed embodiment, the user can prepare multiple pictures with the same theme and style as a style picture training set, and set parameters such as learning rate and training rounds for it. The backend can use the model training tool to train according to the style picture training set and parameters to obtain a second style model, and add the second style model to the multiple style models. It can be understood that the first style model selected by the user can be the second style model or other style models.

[0103] According to the above embodiment, the user can flexibly train the style model and further accurately control the background design style of the image output by the large language model.

[0104] Method 2: Determine the first style model based on the home decoration product image and / or user input information, including:

[0105] In response to receiving a style reference map input by a user, a first style model is determined based on style features of the style reference map.

[0106] In an embodiment of the present disclosure, a user may manually upload a style reference image, and the electronic device may automatically extract style features of the style reference image, and determine a first style model based on the style features of the style reference image.

[0107] Optionally, the user can provide a style reference image, and the backend can use the image prompt adapter IP-Adapter to learn the style features in the reference image and transfer the style features to obtain the first style model.

[0108] According to the above embodiment, the user can flexibly select a style reference image, transfer the style features to obtain a first style model, and further accurately control the background design style of the image output by the large language model.

[0109] In order to more clearly understand the technical solution of the embodiment of the present disclosure, a specific application example is provided below. Figure 2 A schematic diagram of the process of determining the first style model is shown, Figure 2 As shown, the following multiple processing flows may be included:

[0110] S210, select a home decoration design style; the home decoration design style can be selected from the self-trained user style model 20, the public library style model 21 provided by the electronic device, and the uploaded style reference image 22.

[0111] The training process of the user style model 20 may include the following processing flow of S220-S250:

[0112] S220, the user prepares a style picture training set;

[0113] S230, setting model training parameters, etc.;

[0114] S240, the backend generates image annotation information and uses tools to train the model;

[0115] S250. Save the model file and add it to the user model list.

[0116] S260, determine the home decoration design style (i.e., the first style model); the user style model 20 or the public library style model 21 selected by the user can be directly used as the first style model. If the user uploads a style reference image 22, the backend can use the image prompt adapter IP-Adapter to learn the style features in the style reference image 22, thereby determining the first style model.

[0117] In some embodiments, the target feature information includes feature information of multiple dimensions; based on the first style model and the target image, a home decoration effect picture is obtained using a large language model, including:

[0118] Determine the prompt word based on feature information of multiple dimensions;

[0119] Based on the prompt words, the first style model and the target image, a large language model is used to obtain the home decoration renderings.

[0120] Optionally, the feature information of multiple dimensions may include room type, apartment type, furniture type, etc. Exemplarily, room types may include living room, dining room, bedroom, study, kitchen, bathroom, etc., apartment type may include empty apartment type and non-empty apartment type, and furniture type may include sofa, coffee table, desk, bedside table, table lamp, TV, etc.

[0121] Optionally, the electronic device may automatically determine all or part of the feature information of multiple dimensions as prompt words.

[0122] It can be seen that according to the above embodiment, by inputting prompt words into the large language model and guiding the large language model to generate a home decoration rendering, an image that is more in line with the expected picture can be generated, and the home decoration effect of the large language model output can be further controlled.

[0123] In some embodiments, based on feature information of multiple dimensions, the prompt word is determined, including:

[0124] The feature information of multiple dimensions is displayed, and a selection operation on the feature information of multiple dimensions is detected, and the feature information corresponding to the selection operation is determined as a prompt word.

[0125] In the disclosed embodiment, the electronic device can display characteristic information of multiple dimensions in the user interaction interface, and the user can select the characteristic information as a prompt word in the user interaction interface. Of course, the user can also manually input the prompt word.

[0126] It can be seen that in the embodiment of the present disclosure, the user can flexibly select prompt words to further control the effect of the large language model output.

[0127] In some embodiments, based on the target feature information and the target image, a home decoration effect picture is obtained using a large language model, which further includes:

[0128] When the decoration type corresponding to the background image of the first room is a fully-decorated house, a home decoration rendering is obtained based on the target image using a large language model.

[0129] In the disclosed embodiment, when the decoration type corresponding to the first room background image is a fully-decorated house, the decoration style of the first room background image can be retained, and the large language model only needs to merge the room background and the target home decoration products in the target image.

[0130] According to the technical solution of the embodiment of the present disclosure, the decoration type corresponding to the background image of the first room can be a fine-decorated house, and the target home decoration product can be more naturally integrated with the fine-decorated house.

[0131] In order to more clearly understand the technical solution of the embodiment of the present disclosure, a specific application example is provided below. Figure 3 A schematic diagram of a process for generating a home decoration effect diagram is shown. Figure 3As shown, the method for generating the home decoration effect diagram includes the following multiple processing flows:

[0132] S301. The user uploads room pictures (i.e., the background picture of the second room) and home decoration product pictures; the user chooses to upload room pictures and soft decoration product pictures from candidate images provided by photos, photo albums, electronic devices, etc. The uploaded pictures can be various types such as line drawings, real photos, renderings, etc.

[0133] S302, whether to eliminate the interfering content on the picture. If the interfering content on the picture needs to be eliminated, the next step can be S303. If the interfering content on the picture does not need to be eliminated, the next step can be S304. The option of whether to eliminate the interfering content on the picture can be displayed on the user interaction interface, and the user can make the choice.

[0134] S303, smear and eliminate the interfering content; the user can smear the useless interfering content on the room image to generate a mask image. The original image and the mask image after smearing will be sent to the back end for elimination processing. The back end uses the lama model to fill the smeared area and returns the image output by the model (i.e., the background image of the first room) to the front end.

[0135] S304. Use image segmentation models such as OneFormer and SAM to analyze pictures with home furnishing products (i.e., home furnishing product pictures); the backend can semantically segment the home furnishing product pictures based on the OneFormer model, segmenting a series of home furnishing elements including sofas, chairs, wardrobes, chandeliers, and televisions, and return the segmented data structure to the frontend; at the same time, the frontend lazily loads the SAM model file and uses it as a supplementary solution in areas that the OneFormer model does not cover or when the request fails.

[0136] S305: The user selects the furniture and decorative items (i.e., target home furnishing products) that need to be retained; the front end can extract pixel data that matches the subject selected by the user based on the location coordinates clicked by the user, the segmentation data of OneFormer, or the image data inferred by SAM.

[0137] S306. Based on the room map (i.e., the first room background map), appropriately adjust the size, position, direction and other information of the retained items (i.e., the target home furnishing products); the user can adjust the retained items to a suitable viewing angle by dragging, zooming, rotating, translating and other interactive operations, thereby obtaining the target image (i.e., the furniture position map and the furniture position mask map).

[0138] S307, determine whether the room decoration type is rough. If the room decoration type is rough, the next step can be S308. If the room decoration type is not rough, the next step can be S310. The back end can perform feature recognition of multiple dimensions such as decoration type, room type, apartment type, furniture type, etc. on the room image based on the image classification model BLIP-2, so as to determine whether the room decoration type is rough.

[0139] S308. Determine the home decoration design style type; the style type can select a lora model (i.e., the first style model) from the provided public library or self-trained style model list, or only provide a style reference picture; if the user wants to train the style model independently, it is necessary to collect more than a dozen pictures with the same theme and style as the training set, and set parameters such as learning rate and training rounds for them. The back-end will use the model training tool for training based on these data and parameters. The model after successful training will be used as the user's customized style model; if the user provides a style reference picture, the back-end can use the image prompt adapter IP-Adapter to learn the style features in the reference picture and integrate them into the generated image.

[0140] S309, generating corresponding prompt words for the image through the BLIP-2 model, and the user can also add additional prompt words to describe the image; that is, multiple dimensions such as room type, apartment type, furniture type, etc. can be automatically used as prompt words, and the user can also manually enter prompt words.

[0141] S310, organize the data information and submit it to the graphical reasoning engine; when the decoration type is not rough, that is, when the decoration type is fine, the first room background map, furniture position map, furniture position mask map, etc. can be directly submitted to the graphical reasoning engine. In addition to the information submitted to the graphical reasoning engine by the fine type, the rough type also needs to provide the home decoration design style (that is, the first style model) and prompt word information to generate a home decoration rendering of a specific style and picture.

[0142] S311, generating a home decoration rendering; the home decoration rendering can also continue to be used as an input diagram and input into the diagram reasoning engine, and the user can continuously adjust the configuration parameters until a satisfactory effect is achieved.

[0143] According to an embodiment of the present disclosure, the present disclosure also provides a device for generating a home decoration effect diagram. Figure 4 A schematic block diagram of a device for generating a home decoration effect diagram provided by an embodiment of the present disclosure is shown. Figure 4 As shown, the device for generating the home decoration effect diagram includes:

[0144] An acquisition module 410 is used to acquire a first room background image and a home decoration product image;

[0145] A feature recognition module 420 is used to perform feature recognition on the first room background image to obtain target feature information;

[0146] A home decoration determination module 430 is used to determine a target home decoration product in the home decoration product map;

[0147] The home decoration adjustment module 440 is used to display the target home decoration product in the first room background image, and in response to detecting an interactive operation for the target home decoration product, synthesize the first room background image and the target home decoration product into a target image according to the interactive operation;

[0148] The first image generation module 450 is used to obtain a home decoration effect picture based on the target feature information and the target image using a large language model.

[0149] In some embodiments, the acquisition module 410 is specifically used to:

[0150] Get the background image of the second room;

[0151] In response to detecting a smearing operation on the interfering content in the second room background image, determining a smearing area in the second room background image;

[0152] The information of the painted area is eliminated in the background image of the second room to obtain the background image of the first room.

[0153] In some embodiments, Figure 5 As shown, the home improvement determination module 430 includes:

[0154] The segmentation submodule 510 is used to perform semantic segmentation on the home improvement product image and determine a plurality of home improvement products in the home improvement product image;

[0155] The determination submodule 520 is used to determine a target home improvement product based on a plurality of home improvement products.

[0156] In some embodiments, the determination submodule 520 is specifically configured to:

[0157] In response to a click operation on a pixel point or an area in the home improvement product image failing to hit a plurality of home improvement products, the home improvement product image is segmented based on the pixel points or the area using a segmentation model to obtain a target home improvement product in the home improvement product image.

[0158] In some embodiments, the target feature information includes the decoration type corresponding to the first room background image; Figure 6 As shown, the first image generation module 450 includes:

[0159] The style determination submodule 610 is used to determine a first style model based on the home decoration product image and / or user input information when the decoration type corresponding to the first room background image is a rough house;

[0160] The image generation submodule 620 obtains a home decoration rendering based on the first style model and the target image using a large language model.

[0161] In some embodiments, the style determination submodule 610 is specifically used to:

[0162] Display multiple style models in the user interface;

[0163] In response to detecting a selection operation of a user among the multiple style models, a first style model among the multiple style models is determined.

[0164] In some embodiments, the multiple style models include a second style model, and the second style model is trained based on a style picture training set and training parameters input by a user.

[0165] In some embodiments, the style determination submodule 610 is specifically used to:

[0166] In response to receiving a style reference map input by a user, a first style model is determined based on style features of the style reference map.

[0167] In some embodiments, the target feature information includes feature information of multiple dimensions; the image generation submodule 620 is specifically used to:

[0168] Determine the prompt word based on feature information of multiple dimensions;

[0169] Based on the prompt words, the first style model and the target image, a large language model is used to obtain the home decoration renderings.

[0170] In some embodiments, the image generation submodule 620 is specifically used to:

[0171] The feature information of multiple dimensions is displayed, and a selection operation on the feature information of multiple dimensions is detected, and the feature information corresponding to the selection operation is determined as a prompt word.

[0172] In some embodiments, Figure 7 As shown, the second image generating module 710 is also included, and the second image generating module 710 is used to:

[0173] When the decoration type corresponding to the background image of the first room is a fully-decorated house, a home decoration rendering is obtained based on the target image using a large language model.

[0174] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0175] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0176] Figure 8 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 8 As shown, the electronic device includes: a memory 810 and a processor 820, and the memory 810 stores a computer program that can be run on the processor 820. The number of the memory 810 and the processor 820 can be one or more. The memory 810 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 830, which is used to communicate with external devices and perform data exchange transmission.

[0177] If the memory 810, the processor 820 and the communication interface 830 are implemented independently, the memory 810, the processor 820 and the communication interface 830 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0178] Optionally, in a specific implementation, if the memory 810, the processor 820 and the communication interface 830 are integrated on a chip, the memory 810, the processor 820 and the communication interface 830 can communicate with each other through an internal interface.

[0179] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0180] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0182] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0183] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0184] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0185] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0186] The above description is only an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a home decoration effect diagram, characterized in that: include: Get the first room background image and home decoration product images; Performing feature recognition on the first room background image to obtain target feature information; Determine a target home improvement product in the home improvement product map; Displaying the target home furnishing product in the first room background image, and in response to detecting an interactive operation for the target home furnishing product, synthesizing the first room background image and the target home furnishing product into a target image according to the interactive operation; Based on the target feature information and the target image, a home decoration rendering is obtained using a large language model.

2. The method according to claim 1, characterized in that: The obtaining of the first room background image includes: Get the background image of the second room; In response to detecting a smearing operation on the interfering content in the second room background image, determining a smearing area in the second room background image; The information of the painted area is eliminated in the second room background image to obtain the first room background image.

3. The method according to claim 1, characterized in that Determining the target home improvement product in the home improvement product map includes: Performing semantic segmentation on the home improvement product image to determine a plurality of home improvement products in the home improvement product image; Based on the multiple home improvement products, the target home improvement product is determined.

4. The method according to claim 3, characterized in that The determining the target home improvement product based on the plurality of home improvement products includes: In response to a click operation on a pixel point or area within the home improvement product image failing to hit the multiple home improvement products, the home improvement product image is segmented based on the pixel point or area using a segmentation model to obtain a target home improvement product in the home improvement product image.

5. The method according to any one of claims 1 to 4, characterized in that The target feature information includes the decoration type corresponding to the first room background image; and obtaining the home decoration effect image by using the large language model based on the target feature information and the target image includes: When the decoration type corresponding to the first room background image is a rough house, determining a first style model based on the home decoration product image and / or user input information; Based on the first style model and the target image, a home decoration rendering is obtained using a large language model.

6. The method according to claim 5, characterized in that The determining of the first style model based on the home decoration product image and / or the user input information includes: Display multiple style models in the user interface; In response to detecting a selection operation of the user among the multiple style models, a first style model among the multiple style models is determined.

7. The method according to claim 6, characterized in that The multiple style models include a second style model, and the second style model is trained based on a style picture training set and training parameters input by a user.

8. The method according to claim 5, characterized in that The determining of the first style model based on the home decoration product image and / or the user input information includes: In response to receiving a style reference graph input by a user, a first style model is determined based on style features of the style reference graph.

9. The method according to claim 5, characterized in that The target feature information includes feature information of multiple dimensions; the method of obtaining a home decoration effect picture by using a large language model based on the first style model and the target image includes: Determining a prompt word based on the feature information of the multiple dimensions; Based on the prompt word, the first style model and the target image, a home decoration effect picture is obtained using a large language model.

10. The method according to claim 9, characterized in that The determining of the prompt word based on the feature information of the multiple dimensions includes: The feature information of the multiple dimensions is displayed, and a selection operation on the feature information of the multiple dimensions is detected, and the feature information corresponding to the selection operation is determined as a prompt word.

11. The method according to claim 5, characterized in that The method of obtaining a home decoration effect picture based on the target feature information and the target image using a large language model further includes: When the decoration type corresponding to the first room background image is a fully-decorated house, the home decoration rendering is obtained based on the target image using a large language model.

12. A device for generating a home decoration effect picture, characterized in that: include: An acquisition module, used to acquire a first room background image and a home decoration product image; A feature recognition module, used to perform feature recognition on the first room background image to obtain target feature information; A home decoration determination module, used to determine a target home decoration product in the home decoration product map; a home decoration adjustment module, configured to display the target home decoration product in the first room background image, and in response to detecting an interactive operation for the target home decoration product, synthesize the first room background image and the target home decoration product into a target image according to the interactive operation; The first image generation module is used to obtain a home decoration rendering based on the target feature information and the target image using a large language model.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

Images