Describe the extended model training methods, image generation methods, devices, and media.

Abstract

This disclosure relates to a description extension model training method, an image generation method, an apparatus, and a medium. The method acquires first descriptive information that describes the elements contained in a sample image and conforms to the input paradigm of an image generation model. The first descriptive information is masked to construct second descriptive information. The first and second descriptive information are used as training data pairs to train the description extension model to recover a complete and accurate description from incomplete information. Simultaneously, it learns to output information conforming to the input paradigm of the image generation model. The trained description extension model is used to generate image generation prompts, ensuring the accuracy of the prompts and their matching with the image generation model, thus guaranteeing the quality of the prompts. This achieves data distribution alignment between the description extension model and the image generation model in the semantic space, ensuring the quality of subsequently generated images.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for training extended models, an image generation method, an apparatus, and a medium. Background Technology

[0002] With the development of computer technology, image generation models are increasingly widely used in fields such as interior design, digital art, and content creation. Image generation models all use input image generation prompts as conditions to guide and control the image generation process. Therefore, the quality of the image generation prompts has become a key factor affecting the final output. Currently, image generation prompts are usually generated manually; however, the quality of prompts generated in this way is poor, easily leading to results that do not match expectations, thus affecting the quality of the final generated image. Summary of the Invention

[0003] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a method for training extended models, an image generation method, an apparatus, and a medium.

[0004] One aspect of this disclosure provides a method for describing extended model training, the method comprising: Acquire training data pairs, the training data pairs including first description information and second description information, the first description information being used to describe the elements contained in the sample image, the first description information conforming to the input paradigm of the image generation model, and the second description information being obtained by masking a portion of the information in the first description information; The second description information is input into the description expansion model, which expands the content of the second description information and outputs predicted description information. The description extension model is trained based on the difference between the first description information and the predicted description information; The trained description expansion model is used to expand the input information to output information that can serve as image generation prompts. These image generation prompts are then input into the image generation model to generate the corresponding image.

[0005] Another aspect of this disclosure provides an image generation method, the method comprising: Obtain the description information to be expanded, which is used to describe the content contained in the image to be generated; The description information to be expanded is input into the description expansion model, which expands the content of the description information to be expanded and outputs the target description information. The target description information is used as an image generation prompt and input into the image generation model. The image generation model generates a corresponding image based on the image generation prompt and outputs the image. The description extension model is trained based on the description extension model training method described above.

[0006] Another aspect of this disclosure provides a description of an extended model training apparatus, the apparatus comprising: The data pair acquisition module is used to acquire training data pairs, the training data pairs including first description information and second description information. The first description information is used to describe the elements contained in the sample image. The first description information conforms to the input paradigm of the image generation model. The second description information is obtained by masking a portion of the information in the first description information. The prediction expansion module is used to input the second description information into the description expansion model, and the description expansion model expands the content of the second description information to output the predicted description information. The training module is used to train the description extension model based on the difference between the first description information and the predicted description information; The trained description expansion model is used to expand the input information to output information that can serve as image generation prompts. These image generation prompts are then input into the image generation model to generate the corresponding image.

[0007] In some embodiments, the first description information is obtained by an image recognition model recognizing the sample image, and the training data used when training the image generation model is generated based on the image recognition model.

[0008] In some embodiments, the image recognition model includes multiple element types; the data acquisition module is used to acquire the sample image; based on the image recognition model and the multiple element types, the sample image is recognized to obtain the first description information, which describes the elements contained in the sample image that belong to the element types; based on a masking strategy, a portion of the information in the first description information is masked to obtain the second description information.

[0009] In some embodiments, the sample image is an interior design image, and the first description information further includes at least one of the design style or room type of the interior design image; the data acquisition module is used to input the interior design image into the image recognition model, and the image recognition model recognizes the interior design image based on the multiple element types, and outputs reference description information, which is used to describe the elements contained in the interior design image that belong to the element types; at least one of the design style or room type of the interior design image is added to the reference description information to obtain the first description information.

[0010] In some embodiments, the data acquisition module is configured to: randomly determine the target element type to be masked from the element types of the elements contained in the sample image based on the masking strategy; randomly determine target sub-information from the first description information based on the masking strategy and the target element type, wherein the element described by the target sub-information belongs to the target element type; and mask the target sub-information in the first description information to obtain the second description information.

[0011] In some embodiments, the first description information includes at least one element type label, elements under each element type label, and attributes of each element, wherein the element type label is used to represent any one of the multiple element types; the data pair acquisition module is used to randomly determine a first label and a second label from the at least one element type label based on the masking strategy, wherein the first label indicates the target element type, and the second label is the element type label to be masked; based on the masking strategy and the first label, randomly determine the target sub-information from the elements under the first label and their corresponding attributes in the first description information, wherein the target sub-information is used to describe at least one of the elements or attributes of the elements in the sample image; and mask the target sub-information and the second label in the first description information to obtain the second description information.

[0012] In some embodiments, the description expansion model is a large language model; the prediction expansion module is used to input the second description information and the expansion prompt information into the large language model, and the large language model expands the content of the second description information based on the expansion prompt information to output the prediction description information; wherein, the expansion prompt information instructs the large language model to perform an information expansion task on the input information.

[0013] Another aspect of the present disclosure provides an image generation apparatus, the apparatus comprising: The information acquisition module is used to acquire description information to be expanded, which describes the content contained in the image to be generated; The information expansion module is used to input the description information to be expanded into the description expansion model, and the description expansion model expands the content of the description information to be expanded and outputs the target description information. The generation module is used to input the target description information as an image generation prompt into the image generation model, and the image generation model generates a corresponding image based on the image generation prompt and outputs the image. The description extension model is trained based on the description extension model training method described above.

[0014] Another aspect of this disclosure provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the above-described extended model training method or image generation method.

[0015] Another aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described extended model training method or image generation method.

[0016] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the above-described extended model training method or image generation method.

[0017] In the solution provided by this embodiment, first descriptive information that describes the elements contained in a sample image and conforms to the input paradigm of an image generation model is obtained. The first descriptive information is masked to construct incomplete second descriptive information. The first and second descriptive information are used as training data pairs to train the description extension model to master the ability to recover a complete and accurate description from incomplete information. At the same time, it can master the ability to output information that conforms to the input paradigm of the image generation model. The trained description extension model is used to generate image generation prompts to ensure the accuracy of the image generation prompts and their matching with the image generation model, thus ensuring the quality of the image generation prompts. The data distribution alignment between the description extension model and the image generation model in the semantic space is achieved, thereby ensuring the control precision of the subsequent image generation process and the quality of the subsequently generated images. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an extended model training method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating another method for training an extended model provided in this embodiment of the disclosure; Figure 3 This is a schematic flowchart of an image generation method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of an overall training and inference process provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram illustrating the structure of an extended model training device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0021] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0022] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0023] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. The concepts of "first", "second", etc., used in this disclosure are used only to distinguish different devices, modules, or units and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies. The modifications of "a" and "a plurality" used in this disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated in the context, they should be understood as "one or more".

[0024] As image generation models are increasingly widely used in fields such as interior design, digital art, and content creation, current image generation models rely on image generation prompts as conditions for control. The quality of these prompts largely determines the quality of the final generated image. For example, in the field of interior design renderings, there are many different design styles, each corresponding to different combinations of colors and materials. The quality of the image generation prompts determines the instructions for the interior design renderings. However, currently, image generation prompts generated manually cannot guarantee the quality of subsequent image generation. Therefore, to solve the above problems, this disclosure provides a method for training a descriptive extended model, which will be described below with reference to specific embodiments.

[0025] Figure 1 This is a flowchart illustrating a method for training an extended model according to an embodiment of this disclosure. This method can be executed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. Figure 1 As shown, the method provided in this embodiment includes the following steps: S101. Obtain training data pairs, which include first description information and second description information. The first description information is used to describe the elements contained in the sample image. The first description information conforms to the input paradigm of the image generation model. The second description information is obtained by masking a portion of the information in the first description information.

[0026] In this embodiment of the disclosure, training data pairs for training the description extension model are obtained. The training data pairs include first description information that conforms to the input paradigm of the image generation model and is rich in content, and second description information that is incomplete in content compared with the first description information. The description extension model is trained using the training data pairs so that the trained description extension model has the ability to extend the input information and can ensure that the output information can be used as the input of the image generation model. In this way, the trained description extension model can be used to generate image generation prompts for the image generation model, ensuring the quality of the image generation prompts. This ensures the quality of the generated images when the subsequent image generation model uses the input image generation prompts to generate the corresponding images.

[0027] The sample image can be from any field, such as interior design, digital art, or content creation. For example, if the sample image is an interior design rendering of a living room, it can showcase the living room's decor. The elements included in the sample image are the people or objects depicted within it. For instance, if the sample image is an interior design rendering of a living room, it includes elements such as walls, floors, sofas, coffee tables, or decorative items. The first descriptive information describes which elements the sample image contains. In some embodiments, the first descriptive information can also describe the attributes of each element in the sample image, such as color, shape, material, and size. Masking refers to the technical means of hiding, removing, altering, perturbing, or replacing data through direct specification or random selection, thereby introducing information loss, ambiguity, or abnormal states to break the integrity or continuity of data and simulate real data loss scenarios. Data masking operations include content deletion, placeholder replacement, noise injection, character emptying, and semantic obfuscation. The second descriptive information obtained after masking is an incomplete or damaged version of the first descriptive information. Compared with the first descriptive information, the second descriptive information is characterized by missing elements, that is, only retaining information of some elements in the sample image, while the information of the remaining elements is completely hidden or removed; or, the second descriptive information is characterized by blurred details, that is, while retaining all elements, the attributes or relationships of some elements are masked as general markers or become unrecognizable; or, the second descriptive information is characterized by both missing elements and blurred details. Both the first and second descriptive information can be represented in any form, for example, both the first and second descriptive information can be represented in the form of text.

[0028] In this embodiment, the input paradigm refers to the structure, format, syntax rules, or semantic space that the input data must follow to be recognized and processed by the image generation model. This input paradigm defines the standard form of input data that the image generation model can understand. The first descriptive information conforming to the input paradigm of the image generation model means that the first descriptive information has a hierarchical logic that the model can parse, follows the model's predefined encoding specifications in terms of format and syntax rules, and its data distribution in the semantic space is consistent with or highly compatible with the optimal input information expected by the image generation model. That is, the data distribution of the first descriptive information and the input of the image generation model are aligned in the semantic space, thereby achieving accurate image generation guidance. If the first descriptive information is used as an image generation prompt to guide the image generation model to generate an image, the generated image quality will not be poor due to incompatibility between the first descriptive information and the image generation model.

[0029] S102. Input the second description information into the description extension model, and the description extension model expands the content of the second description information to output the predicted description information.

[0030] In this embodiment of the disclosure, incomplete descriptive information (i.e., second descriptive information) from the training data pairs is used as input to the descriptive expansion model. This allows the descriptive expansion model to expand the input descriptive information based on its own content expansion capabilities and output a new descriptive information (i.e., predicted descriptive information). The predicted descriptive information can be represented in any form; for example, it can be represented as text. The descriptive expansion model can be any neural network model; for example, it can be a large language model.

[0031] In the embodiments of this disclosure, the description extension model may be a model with preliminary content extension capabilities or a model without content extension capabilities. Therefore, the predicted description information output by the description extension model may not be complete or accurate enough.

[0032] S103. Based on the difference between the first description information and the predicted description information, the description expansion model is trained; wherein, the trained description expansion model is used to expand the input information to output information that can serve as image generation prompts, and the image generation prompts are used as inputs to the image generation model to generate the corresponding image.

[0033] In this embodiment, since the first descriptive information is a complete descriptive information for the sample image and conforms to the input paradigm of the image generation model, and the predicted descriptive information is the descriptive information obtained by the descriptive expansion model for the incomplete descriptive information, the difference between the first descriptive information and the predicted descriptive information is equivalent to the difference between the expected descriptive information and the output of the descriptive expansion model. This can reflect the difference between the current capability and the expected capability of the descriptive expansion model. Using this as a basis to train the descriptive expansion model can not only improve the content expansion capability of the descriptive expansion model, but also ensure that the descriptive expansion model has the ability to output information that conforms to the input paradigm of the image generation model, thereby ensuring the quality of the information subsequently generated by the descriptive expansion model.

[0034] The image generation model can be any neural network model used to generate images. The input to the image generation model is an image generation cue, and the output is an image. The image generation cue describes the content of the image to be generated and can be represented in any form, such as text.

[0035] In this embodiment of the disclosure, after training the description extension model, the trained description extension model can extend any input information, and the information output by the trained description extension model can be directly input into the image generation model as an image generation prompt, so that the image generation model can generate the corresponding image based on the image generation prompt. That is, the trained description extension model and the image generation model are combined to achieve a solution for generating a high-quality image that matches the expectation for any information.

[0036] In the solution provided by this embodiment, first descriptive information that describes the elements contained in a sample image and conforms to the input paradigm of an image generation model is obtained. The first descriptive information is masked to construct incomplete second descriptive information. The first and second descriptive information are used as training data pairs to train the description extension model to master the ability to recover a complete and accurate description from incomplete information. At the same time, it can master the ability to output information that conforms to the input paradigm of the image generation model. The trained description extension model is used to generate image generation prompts to ensure the accuracy of the image generation prompts and their matching with the image generation model, thus ensuring the quality of the image generation prompts. The data distribution alignment between the description extension model and the image generation model in the semantic space is achieved, thereby ensuring the control precision of the subsequent image generation process and the quality of the subsequently generated images.

[0037] In some embodiments, an image recognition model is used to ensure that the first descriptive information conforms to the input paradigm of the image generation model. That is, the first descriptive information is obtained by the image recognition model from the sample image, and the training data used when training the image generation model is generated based on the image recognition model.

[0038] The image recognition model is used to recognize the input image and output information describing the input image. In some embodiments, the image recognition model is also used to generate a corresponding image based on the input image description information. The image recognition model can be any neural network model; for example, the image recognition model is a VLM (Visual Language Model), which is used to implement the mapping from visual input to text description, that is, the input of the VLM is an image, and the output is the image description information.

[0039] In this embodiment, the same image recognition model is used to obtain the first descriptive information for training the descriptive extension model and the training data for training the image generation model, so that the first descriptive information conforms to the input paradigm of the image generation model. In this way, by using the same image recognition model to generate the first descriptive information required for training the descriptive extension model and the image descriptive data required for training the image generation model, the distribution alignment and structural adaptation of the two are achieved at the data source. This data homology mechanism ensures that the descriptive extension model can learn a descriptive ability that is highly matched with the input requirements of the image generation model. Subsequently, the trained descriptive extension model is used to generate image generation prompts to ensure that the image generation prompts match the image generation model, thereby ensuring the quality of the generated image.

[0040] In this embodiment, the input to the image generation model is an image generation prompt, which is equivalent to the descriptive information corresponding to the image to be generated. The training data used when training the image generation model includes the image descriptive information and the image itself. The image descriptive information is obtained based on the image recognition model's recognition of the image. During training, the descriptive information from the training data is input into the image generation model so that the model generates a predicted image based on the input descriptive information. The images included in the training data are used as labels, and the predicted image and labels are combined to train the image generation model. In this way, the input adapted to the trained image generation model is consistent with or highly compatible with the output of the image recognition model; and the output of the descriptive extension model is also consistent with or highly compatible with the output of the image recognition model, making the output of the descriptive extension model consistent with or highly compatible with the input of the image generation model.

[0041] In some embodiments, the first descriptive information is obtained by transforming the initial descriptive information according to the input paradigm of the image generation model. That is, the process of obtaining the first descriptive information includes: obtaining initial descriptive information for describing the elements contained in the sample image; and transforming the initial descriptive information based on the input paradigm of the image generation model to obtain the first descriptive information that conforms to the input paradigm.

[0042] The initial description information is used to describe the content of the sample image. The initial description information can be obtained in any way, such as through manual annotation or recognition by any image recognition model.

[0043] In this embodiment, the initial description information is standardized and transformed based on the input paradigm defined by the image generation model. The initial description information, which may not conform to the recognition logic of the image generation model, is transformed into first description information with regular structure and semantic alignment, ensuring that the first description information can be accurately understood and processed by the image generation model. When training the description extension model based on the first description information, the description extension model can learn the description ability that is highly matched with the input requirements of the image generation model. Subsequently, the trained description extension model is used to generate image generation prompts to ensure that the image generation prompts match the image generation model, thereby ensuring the quality of the generated image.

[0044] Based on the embodiments shown above, the embodiments of this disclosure can combine multiple element types and masking strategies in the image recognition model to obtain training data pairs. For details of the process, please refer to the following embodiments. Figure 2 This is a flowchart illustrating another method for training an extended model provided in this disclosure, which can be executed by an electronic device. This electronic device can be exemplarily understood as a device such as a mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. Figure 2 As shown, the method for training the extended model can include the following steps: S201. Obtain sample images.

[0045] In the embodiments of this disclosure, sample images can be acquired in any manner, such as acquiring sample images from a locally stored image database; or, by interacting with other devices to acquire the sample images from other devices. This disclosure does not limit the scope of the acquisition.

[0046] S202. Based on the image recognition model and multiple element types, the sample image is identified to obtain first description information. The first description information is used to describe the elements contained in the sample image that belong to the element type.

[0047] In this embodiment of the disclosure, the image recognition model is provided with multiple element types. The image recognition model can identify the elements contained in the input image according to these multiple element types, that is, identify the elements contained in the image that belong to the element type. Therefore, the elements described by the first description information are all elements in the sample image that belong to the element type. For elements in the sample image that do not belong to any of these multiple element types, the first description information does not need to describe them.

[0048] Multiple element types are used to describe elements from different dimensions to distinguish them. In some embodiments, the element types corresponding to images in different fields are different. For example, in the field of interior design, the multiple element types are hard furnishings, soft furnishings, decorations, green plants, and home appliances. Hard furnishings refer to walls, floors, ceilings, etc.; soft furnishings refer to sofas, coffee tables, stools, etc.; decorations refer to cushions, paintings, etc.; green plants refer to indoor plants, flowers, etc.; and home appliances refer to indoor televisions, air conditioners, etc. In this embodiment of the disclosure, for these multiple element types, the sample image may only contain elements of at least one of the multiple element types. When the image recognition model recognizes the sample image, the first descriptive information output is only used to describe elements of this at least one element type.

[0049] In some embodiments, the first description information indicates the elements contained in the sample image that belong to each element type, as well as the attributes of the elements. The attributes of an element refer to its shape, color, material, size, etc. In this embodiment, the first description information indicates which elements in the sample image belong to each element type and the attributes of each element, so as to fully describe the content contained in the sample image, thereby ensuring the accuracy of the first description information and thus ensuring the training effect of the subsequent training description extension model.

[0050] In some embodiments, taking the field of interior design as an example, i.e., the sample image is an interior design image, the first description information also includes at least one of the design style or room type of the interior design image; that is, step S202 includes: inputting the interior design image into an image recognition model, the image recognition model recognizing the interior design image based on multiple element types, outputting reference description information, the reference description information being used to describe the elements contained in the interior design image that belong to the element type; adding at least one of the design style or room type of the interior design image to the reference description information to obtain the first description information.

[0051] Here, "design style" refers to the style of the interior design effect shown in the images, such as "creamy" or "modern minimalist." "Room type" refers to the type of room shown in the images, such as bedroom, study, living room, or dining room.

[0052] In this embodiment, an interior design image is input into an image recognition model, which then identifies the interior design image based on various element types. The model identifies elements belonging to each element type and outputs reference description information. At least one of the design style or room type from the interior design image is then added to the reference description information to obtain first description information. This ensures that the first description information not only describes the elements contained in the interior design image but also at least one of the design style or room type, enriching its content and enabling a more complete and accurate description of the interior design image. This guarantees the quality of the first description information and ensures its ability to expand the content of the description extension model when it is subsequently used to train the model.

[0053] In the embodiments of this disclosure, the design style or room type of the interior design image is specified by a person, or it is obtained by other neural network models specifically used to identify the design style or room type of the interior design image. This disclosure does not limit this.

[0054] In some embodiments, taking the first description information including design style and room type as an example, the first description information can not only indicate the elements contained in the interior design image that belong to the element type and the attributes of the elements, but also indicate the design style and room type of the interior design image.

[0055] It should be noted that this embodiment uses an interior design image from the field of interior design as an example for illustration. In another embodiment, if the sample image is from another field, the first descriptive information can not only describe the elements contained in the sample image, but also indicate other parameters of the sample image. For example, if the sample image is from the field of digital art, the first descriptive information can also indicate the type of the sample image, such as animation, film, game, etc.

[0056] S203. Based on the masking strategy, mask a portion of the information in the first description information to obtain the second description information.

[0057] In this embodiment of the disclosure, after obtaining the first description information and the second description information, the first description information and the second description information can be used as a training data pair so that the description extension model can be trained subsequently using this training data pair. The masking strategy indicates the method of masking part of the information in the description information. Based on the masking strategy, part of the information in the first description information is masked to obtain the incomplete second description information.

[0058] In this embodiment, the sample image is identified by a preset image recognition model with multiple element types to generate a first description information that is rich in content and complete in description. Then, a masking strategy is used to construct a second description information with incomplete content. The difference between the two descriptions is used to build effective training samples, which provides an efficient learning paradigm for the description extension model. When the description extension model is trained in the future, it can improve its ability to understand element associations and fill in missing information, and finally ensure that the image description it generates has higher accuracy and completeness.

[0059] In some embodiments, step S203 includes steps 1 to 3.

[0060] Step 1: Based on the masking strategy, randomly determine the target element type to be masked from the element types of the elements contained in the sample image.

[0061] The elements contained in the sample image may belong to one or more element types, and these element types can also be referred to as the element types corresponding to the first description information. The target element type may be one or more; in this embodiment, the randomly determined element type to be masked is simply referred to as the target element type. The masking strategy indicates the method of randomly determining the element type to be masked from multiple element types.

[0062] Step 2: Based on the masking strategy and the target element type, randomly determine the target sub-information from the first description information. The element described by the target sub-information belongs to the target element type.

[0063] In this embodiment of the disclosure, after determining the target element type to be masked, partial description information related to the target element type is first determined from the first description information. Then, based on a masking strategy, target sub-information is randomly determined from this partial description information. The masking strategy further specifies the method for randomly determining the sub-information to be masked from the partial description information corresponding to the element type to be masked. The target sub-information may be all the description information related to the target element type, or it may be a portion of all the description information related to the target element type.

[0064] For example, taking an interior design image as a sample image, the first descriptive information can describe elements under the hard decoration type, soft decoration type, decorative element type, green plant type, and home appliance type in the sample image. Taking the decorative element type as the target element type, the descriptive information used to describe the elements under the decorative element type is determined from the first descriptive information. Taking this part of the descriptive information as the local descriptive information, the target sub-information is subsequently determined from the local descriptive information. The target sub-information may be the local descriptive information or a part of the information in the local descriptive information.

[0065] Step 3: Mask the target sub-information in the first description information to obtain the second description information.

[0066] In this embodiment of the disclosure, a masking method is used to delete or cover the target sub-information in the first description information so that the obtained second description information no longer contains the target sub-information.

[0067] In this embodiment of the disclosure, since the element described by the first description information belongs to at least one element type, based on the masking strategy, the target element type to be masked is first randomly determined from the element types corresponding to the first description information. Then, based on the masking strategy, the target sub-information is randomly determined from the partial information contained in the first description information that represents the element under the target element type, and then the target sub-information is masked. This hierarchical random strategy ensures both the comprehensiveness and unpredictability of the masking operation and the integrity of the semantic information. This makes it necessary for the subsequent description extension model to learn the deep correlation rules between element types, elements, and element attributes, thereby effectively enhancing the reasoning and completion capabilities of the description extension model in complex scenarios.

[0068] In some embodiments, the first description information includes at least one element type label, elements under each element type label, and attributes of each element, wherein the element type label is used to represent any one of multiple element types; then the process of obtaining the second description information described in steps 1 to 3 above includes steps 4 to 6.

[0069] Step 4: Based on the masking strategy, randomly determine the first label and the second label from the element type labels included in the first description information. The first label indicates the target element type, and the second label is the element type label to be masked.

[0070] In this embodiment of the disclosure, each element type set in the image recognition model corresponds to a corresponding element type label; the first description information includes element type labels, elements under element labels, and element attributes. When masking part of the information in the first description information based on a masking strategy, it is possible to mask part of the type labels in the first description information, or to mask part or all of the elements under one or more element type labels, or to mask one or more attributes of the same element.

[0071] In this embodiment of the disclosure, a first tag and a second tag are randomly determined from the element type tags contained in the first description information based on a masking strategy, so as to mask the elements under the first tag or the attributes of the elements under the first tag, and to mask the second tag. The first tag may be one or more element type tags, and the second tag may also be one or more element type tags; the first tag may be the same as the second tag, or it may be different from the second tag.

[0072] In some embodiments, if some element type labels among the element type labels corresponding to multiple element types set in the image recognition model cannot be masked, then step 4 includes: randomly determining a first label from the element type labels included in the first description information based on a masking strategy; and randomly determining a second label from the maskable element type labels included in the first description information. In this embodiment, among the element type labels corresponding to multiple element types set in the image recognition model, some element type labels cannot be masked, while the remaining element type labels can be masked. When masking a portion of the information in the first description information, the maskable element type labels can be masked, and the elements or attributes under all element type labels can be masked. For example, in the field of interior design, hard furnishing type labels and soft furnishing type labels are element type labels that cannot be masked, while decorative item type labels, green plant type labels, and home appliance type labels are element type labels that can be masked.

[0073] Step 5: Based on the masking strategy and the first label, randomly determine the target sub-information from the elements and corresponding attributes under the first label in the first description information. The target sub-information is used to describe at least one of the elements or attributes of the elements in the sample image.

[0074] In this embodiment of the disclosure, based on the masking strategy and the first tag, the element to be masked or the attribute of the element to be masked is randomly determined from the elements and corresponding attributes under the first tag in the first description information, thereby determining the target sub-information to ensure the randomness and diversity of the target sub-information.

[0075] In some embodiments, if the target sub-information is used to describe the attributes of elements in a sample image, then the target sub-information can describe at least one of the multiple attributes corresponding to any element in the sample image. The multiple attributes corresponding to the same element can describe the element from different perspectives. For example, the multiple attributes can represent the element's shape, color, material, etc. Taking the target sub-information describing a sofa element in the sample image as an example, the target sub-information includes the sofa's color and material, and subsequently, the color and material of the sofa in the first description information can be masked.

[0076] Step 6: Mask the target sub-information and the second label in the first description information to obtain the second description information.

[0077] In this embodiment of the disclosure, not only the target sub-information can be masked, but the second tag can also be masked, which enriches the diversity of masking methods and thus ensures the diversity and randomness of the second description information.

[0078] In this embodiment, the target type and the label to be masked are first randomly determined from the element type labels. Then, the specific mask content is randomly selected from the elements and attributes corresponding to the target type. Finally, the specific description information and element type labels are masked simultaneously. This dual masking strategy not only preserves the structural integrity of semantic information, but also forces the model to deeply understand the intrinsic relationship between element type and attributes by hiding the type labels. This effectively enhances the cross-dimensional reasoning and semantic completion capabilities of the description extension model in complex scenarios, laying a solid foundation for generating high-quality, structured image descriptions.

[0079] It should be noted that the above embodiment is illustrated by simultaneously masking at least one of the elements or element attributes and the element type label. In another embodiment, it is not necessary to perform step 4 above. Instead, only the first label is randomly determined, and then only the target sub-information in the first description information is masked.

[0080] It should be noted that the embodiments disclosed herein are based on multiple element types and masking strategies in the image recognition model to obtain training data pairs. In another embodiment, it is not necessary to perform the above steps S201-S203, but other methods are adopted to obtain training data pairs. The training data pairs include first description information and second description information. The first description information is used to describe the elements contained in the sample image. The first description information conforms to the input paradigm of the image generation model. The second description information is obtained by masking part of the information in the first description information.

[0081] S204. Input the second description information into the description extension model, and the description extension model expands the content of the second description information to output the predicted description information.

[0082] In some embodiments, the description extension model is a large language model; then step S204 includes: inputting the second description information and the extended prompt information into the large language model, and having the large language model extend the content of the second description information based on the extended prompt information, and outputting predicted description information; wherein, the extended prompt information instructs the large language model to perform an information extension task on the input information.

[0083] In this embodiment, the input to the large language model is incomplete second descriptive information and extended prompt information. The extended prompt information instructs the large language model how to expand the second descriptive information, enabling it to output expanded descriptive information, i.e., predicted descriptive information. Since the extended prompt information, as a task instruction, explicitly requires the large language model to perform information expansion on the input second descriptive information, the large language model can focus on the specific goals of semantic completion and content enhancement, avoiding irrelevant or random outputs. This instruction-based fine-grained control mechanism not only ensures the standardization and practicality of the predicted descriptive information generated by the large language model but also significantly enhances the adaptability between its output and the input requirements of the image generation model, thereby providing a reliable guarantee for the quality of subsequent image generation.

[0084] S205. Based on the difference between the first description information and the predicted description information, the description expansion model is trained; wherein, the trained description expansion model is used to expand the input information to output information that can serve as image generation prompts, and the image generation prompts are used as inputs to the image generation model to generate the corresponding image.

[0085] In some embodiments, step S205 includes: calculating a first loss value based on the difference between the first descriptive information and the predicted descriptive information; and adjusting the model parameters of the descriptive extension model based on the first loss value. In this embodiment, training the descriptive extension model using loss calculation further ensures the training effect of the descriptive extension model. The first loss value can be calculated in any manner, for example, by calculating cross-entropy loss.

[0086] In some embodiments, step S205 includes: calculating a first loss value based on the difference between the first descriptive information and the predicted descriptive information; determining a second loss value based on the element coverage relationship between the predicted descriptive information and the second descriptive information; and adjusting the model parameters of the description expansion model based on the first loss value and the second loss value. The second loss value reflects whether the predicted descriptive information contains the elements described by the second descriptive information, ensuring that the model does not omit key elements during generation. If the elements described by the predicted descriptive information contain all the elements described by the second descriptive information, the second loss value is sufficiently small and can be ignored. In this embodiment, when training the description expansion model, it is necessary not only to ensure that the predicted descriptive information output by the description expansion model is sufficiently consistent with the first descriptive information, but also to consider that the elements described by the predicted descriptive information contain all the elements described by the second descriptive information, thereby expanding the input while ensuring that the output of the description expansion model is semantically aligned with the input.

[0087] In some embodiments, the process of determining the second loss value includes: extracting a target element set from the second description information, extracting a predicted element set from the predicted description information, determining the number of intersection elements between the target element set and the predicted element set, determining the ratio of the number of intersection elements to the total number of target elements as the coverage rate, and determining the value of 1 minus the coverage rate as the second loss value. In embodiments of this disclosure, the loss value is zero when the predicted description information completely contains all target elements, and the loss value increases with the degree of omission once any element is missing.

[0088] In some embodiments, the process of adjusting the model parameters of the descriptive extension model based on the first loss value and the second loss value includes: weighting and summing the first loss value and the second loss value based on the weights of the first loss value and the second loss value to obtain a sum of losses; and adjusting the model parameters of the descriptive extension model based on the sum of losses. In this embodiment, by weighting and summing the first loss value and the second loss value, the gradient of the sum of losses with respect to each model parameter is calculated based on the backpropagation algorithm, and the parameters are iteratively updated along the negative gradient direction to minimize the total loss. In this way, the performance of the descriptive extension model between "overall text semantic accuracy" and "completeness of key style elements" can be dynamically balanced, thereby ensuring that the generated description is natural and fluent while strengthening the accurate coverage of core design elements, and thus improving the generation quality and reliability of the descriptive extension model.

[0089] It should be noted that this embodiment only uses a single training data pair to train the description extension model as an example. In another embodiment, the description extension model can be iteratively trained according to steps S201-S205 described above. During the iterative training of the description extension model, if the number of iterations reaches a threshold, or if the loss value calculated in the current iteration is less than a loss value threshold, then the training of the description extension model is stopped. The number of iterations threshold is any integer, and the loss value threshold is any numerical value.

[0090] In the solution provided by this embodiment, first descriptive information that describes the elements contained in a sample image and conforms to the input paradigm of an image generation model is obtained. The first descriptive information is masked to construct incomplete second descriptive information. The first and second descriptive information are used as training data pairs to train the description extension model to master the ability to recover a complete and accurate description from incomplete information. At the same time, it can master the ability to output information that conforms to the input paradigm of the image generation model. The trained description extension model is used to generate image generation prompts to ensure the accuracy of the image generation prompts and their matching with the image generation model, thus ensuring the quality of the image generation prompts. The data distribution alignment between the description extension model and the image generation model in the semantic space is achieved, thereby ensuring the control precision of the subsequent image generation process and the quality of the subsequently generated images.

[0091] Based on the above-described description extension model training method embodiments, in this disclosure embodiment, the trained description extension model can be used to realize the following image generation method, the specific process of which is detailed in the following embodiments. Figure 3 This is a schematic flowchart illustrating an image generation method provided in an embodiment of this disclosure. This method can be executed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. Figure 3 As shown, this image generation method may include the following steps: S301. Obtain the description information to be expanded. The description information to be expanded is used to describe the content contained in the image to be generated.

[0092] The description information to be expanded can be any kind of description information, and it can be represented in the form of text or images. For example, if the description information to be expanded is represented in the form of text, in the field of interior design, the description information to be expanded would be "living room, modern minimalist style, black leather sofa, coffee table, TV cabinet".

[0093] In some embodiments, the description information to be expanded is input by the user through a terminal. For example, the user's terminal displays an interactive interface with an information input area and a confirmation option. After the user inputs the description information to be expanded in the information input area of ​​the interactive interface through the terminal, the user clicks the confirmation option on the terminal, and the terminal sends the description information to be expanded to the electronic device, which then receives the description information.

[0094] In some embodiments, the electronic device directly acquires the description information to be expanded, which is represented in text form; or, after acquiring the description information to be expanded, which is represented in image form, it uses an image recognition model to recognize the description information to be expanded, and obtains the description information to be expanded in text form.

[0095] In this embodiment of the disclosure, the input to the description expansion model is text, so the description information to be expanded in text form can be directly obtained. If an image is obtained, the image is converted into the description information to be expanded in text form by the image recognition model in the manner shown in the above embodiment.

[0096] S302. Input the description information to be expanded into the description expansion model. The description expansion model expands the content of the description information to be expanded and outputs the target description information.

[0097] In this embodiment of the disclosure, the description information to be expanded is input into the description expansion model. The description expansion model uses its own description expansion capability to expand the content of the description information to be expanded, so as to expand other information of these elements or expand the description information of other related elements while ensuring that the original elements described by the description information to be expanded are preserved, thereby obtaining the target description information.

[0098] The target description information is used to describe the content contained in the image to be generated. The elements described by the description information to be expanded are called user-specified elements. The elements described by the target description information include at least user-specified elements and other related elements.

[0099] For example, in the field of interior design, the description information to be expanded is "living room, modern minimalist style, black leather sofa, coffee table, TV cabinet," while the target description information is "modern minimalist style; hard furnishings include walls (grayish-white, latex paint) and floors (black, marble material, flat tiling); soft furnishings include sofas (black, leather, L-shaped) and coffee tables (white, round, marble material); greenery includes Monstera deliciosa (green); decorative items include cushions (brown, fabric material)." In other words, the target description information includes the elements described in the description information to be expanded, as well as other related elements, and expands the attributes of the elements.

[0100] S303. Input the target description information as an image generation prompt into the image generation model, and the image generation model generates the corresponding image based on the image generation prompt, and outputs the image.

[0101] In this embodiment of the disclosure, since the target description information output by the description extension model conforms to the input paradigm of the image generation model, the target description information is used as an image generation prompt input to the image generation model. The image generation model generates the corresponding image based on the image generation prompt and outputs the image, so that the image matches the target description information and also matches the description information to be extended, so that the image can meet the user's expectations and ensure the image generation effect.

[0102] In this embodiment, a simplified description information to be expanded is obtained, and then the description expansion model is used to semantically enhance and supplement its content to generate a complete and clearly structured target description information. Finally, the target description information is used as an accurate image generation prompt input to the image generation model, so that the image generation model can generate the corresponding image based on the image generation prompt. In this way, by using a specially trained description expansion model, an automated process from text description to high-quality image generation is constructed. Since the data distribution of the description expansion model and the image generation model is aligned in the semantic space, that is, the target description information output by the description expansion model conforms to the input paradigm of the image generation model, the quality of the image generation prompt is improved, so as to ensure that the final generated image has richness of detail, semantic accuracy and visual consistency, providing reliable technical support for automated image generation in fields such as interior design and digital content creation.

[0103] Based on the embodiments shown above, this disclosure provides a flowchart illustrating the overall training and inference process. Taking the image recognition model as a Visual Language Model (VLM) and the description extension model as a Prompt Large Language Model (Prompt-LLM) optimized for image-generated prompts, as an example... Figure 4 As shown, the method includes a training phase and an inference phase.

[0104] The goal of the training phase is to enable the large language model to grasp the correspondence between style, room type, and specific elements, as well as element attributes (such as color, material, and shape), thereby generating complete descriptive information based on a simple descriptive information. The training phase includes data construction and core learning. Data construction is the most crucial step in the training phase, utilizing the visual language model to construct training data pairs, including: 1) Constructing a complete labeling system for the interior design domain and configuring the labeling system in the VLM. The labeling system is divided into five categories (i.e., five element types): hard furnishings, soft furnishings, decorative items, greenery, and appliances. Each element type is assigned corresponding attribute labels from three dimensions: color, material, and shape. 2) The VLM receives an interior design rendering and analyzes it to identify elements such as hard furnishings (e.g., walls, floors, ceilings), soft furnishings (e.g., sofas, tables), decorative items, and greenery, along with their corresponding attribute labels. 3) Extract the element information from the interior design renderings using VLM, and combine it with the style tags (such as modern minimalist style) and room type (such as living room) corresponding to the interior design renderings to form the first description information; selectively and randomly mask some attribute information of the first description information to construct an incomplete second description information, and use the first description information and the second description information as training data pairs.

[0105] The core learning process involves using pre-constructed training data to train a large language model. This means inputting incomplete second-level descriptive information into the large language model so that it can output expanded descriptive information. The training goal is for the large language model to generate complete image-based prompts that include all core elements, decorative elements, and user-specified elements within the given style, based on the input descriptive information.

[0106] In this embodiment, the VLM used to construct the training data pairs is consistent with the Caption (image generation cue) model used during the training of the image generation model. This ensures that the "style dictionary" learned by the large language model is naturally aligned with the data distribution of the image generation model itself, thus resolving the inconsistency between manually generated image generation cues and the training data of the image generation model. Through this stage of training, the large language model forms a style-element knowledge base, which can be understood as a "style dictionary," meaning that the large language model has the ability to analyze the relationship between styles and elements.

[0107] The goal of inference is to generate a high-quality, complete image generation prompt that can be directly used in the image generation model based on a simple user instruction. The user instruction can be an image or a simple text description. If the user instruction is a reference image, the reference image is identified using the VLM described above to obtain the corresponding text description. Then, the image generation prompt is generated through the following process. If the user instruction is a text description, the image generation prompt is generated directly through the following process. The process of generating an image generation prompt includes: 1) User instruction parsing: The user gives an instruction, such as "living room, modern minimalist style, black leather sofa, coffee table, TV cabinet". This instruction includes style, room type, and core elements. The core elements in the user instruction (e.g., "black leather sofa, coffee table, TV cabinet") are used as constraints to ensure that the final generated image generation prompt contains these elements specified by the user. 2) Core inference of the large language model: Based on the style, room type, and elements specified by the user, the large language model infers the general element combinations under that style from its internal "style dictionary". For example, the large language model infers that a "modern minimalist" living room typically includes elements such as "marble flooring," "white walls," and "round rug." Simultaneously, the large language model seamlessly integrates the core elements input by the user ("black leather sofa, coffee table, TV cabinet") into its generated list of general elements. 3) Complete image generation prompt generation: The large language model outputs a complete image generation prompt that integrates user specifications and model inference. This image generation prompt includes both the core elements explicitly requested by the user (such as a black leather sofa) and general elements and decorations that conform to the data distribution within this style, ensuring that the generated image generation prompt is complete and of high quality.

[0108] In this embodiment of the disclosure, through this reasoning process, the solution achieves an automated transformation from a simple user instruction to a complex, high-quality image generation prompt, greatly improving the user experience and generation effect.

[0109] In this embodiment, leveraging the powerful generative capabilities of a large language model, a simple style and room type description (e.g., "cream-colored living room") is automatically expanded into a complete, high-quality image generation prompt that includes color, material, shape, core elements, and decorations, making the image generation prompt richer and more accurate. Furthermore, by constructing training data pairs based on the captions actually used by the image generation model during training, the large language model ensures that the final image generation prompt generated by the large language model is highly aligned with the image generation model in terms of grammar, vocabulary, and expression habits. In this way, because the image generation prompts generated by the large language model are complete, accurate, and highly matched with the data distribution of the image generation model, it can significantly improve the image generation model's ability to understand and follow the image generation prompts (Prompt-Following), thereby generating higher-quality interior design renderings that better meet user expectations and improving image generation quality. Users only need to provide simple descriptive information such as style and room type to obtain professional, high-quality image generation prompts, which greatly lowers the application threshold of image generation tools in the field of interior design and improves work efficiency. Utilizing the generalization ability of the large language model, it can easily handle and generate complex, popular, or fusion design styles that contain multiple element combinations, supporting complex styles and expanding the application boundaries of image generation models in the field of interior design.

[0110] It should be noted that the above-mentioned multiple optional embodiments can be combined in any way, and this disclosure does not limit them.

[0111] Figure 5 This is a schematic diagram of the structure of a descriptive extended model training device provided in an embodiment of this disclosure. This descriptive extended model training device can be understood as the aforementioned electronic device or a functional module within the aforementioned electronic device. For example... Figure 5 As shown, the description of the extended model training device includes: The data pair acquisition module 501 is used to acquire training data pairs. The training data pairs include first description information and second description information. The first description information is used to describe the elements contained in the sample image. The first description information conforms to the input paradigm of the image generation model. The second description information is obtained by masking a portion of the information in the first description information. The prediction extension module 502 is used to input the second description information into the description extension model, and the description extension model expands the content of the second description information to output the prediction description information. Training module 503 is used to train the description extension model based on the difference between the first description information and the predicted description information; The trained description expansion model is used to expand the input information to output information that can serve as image generation cues. These image generation cues are then input into the image generation model to generate the corresponding image.

[0112] In some embodiments, the first descriptive information is obtained by the image recognition model recognizing the sample image, and the training data used when training the image generation model is generated based on the image recognition model.

[0113] In some embodiments, the image recognition model is provided with multiple element types; the data acquisition module 501 is used to acquire sample images; based on the image recognition model and multiple element types, the sample images are recognized to obtain first description information, which is used to describe the elements contained in the sample images that belong to the element types; based on a masking strategy, some information in the first description information is masked to obtain second description information.

[0114] In some embodiments, the sample image is an interior design image, and the first description information further includes at least one of the design style or room type of the interior design image; the data acquisition module 501 is used to input the interior design image into the image recognition model, and the image recognition model recognizes the interior design image based on multiple element types and outputs reference description information, which is used to describe the elements contained in the interior design image that belong to the element type; at least one of the design style or room type of the interior design image is added to the reference description information to obtain the first description information.

[0115] In some embodiments, the data acquisition module 501 is configured to: randomly determine the target element type to be masked from the element types of the elements contained in the sample image based on a masking strategy; randomly determine target sub-information from the first description information based on the masking strategy and the target element type, wherein the element described by the target sub-information belongs to the target element type; and mask the target sub-information in the first description information to obtain the second description information.

[0116] In some embodiments, the first description information includes at least one element type label, elements under each element type label, and attributes of each element. The element type label is used to represent any one of multiple element types. The data pair acquisition module 501 is used to randomly determine a first label and a second label from at least one element type label based on a masking strategy. The first label indicates the target element type, and the second label is the element type label to be masked. Based on the masking strategy and the first label, target sub-information is randomly determined from the elements under the first label and their corresponding attributes in the first description information. The target sub-information is used to describe at least one of the elements or attributes of the elements in the sample image. The target sub-information and the second label in the first description information are masked to obtain the second description information.

[0117] In some embodiments, the description extension model is a large language model; the prediction extension module 502 is used to input the second description information and the extension prompt information into the large language model, and the large language model expands the content of the second description information based on the extension prompt information to output the prediction description information; wherein, the extension prompt information instructs the large language model to perform an information expansion task on the input information.

[0118] Figure 6 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this disclosure. This image generation apparatus can be understood as the aforementioned electronic device or a functional module within the aforementioned electronic device. For example... Figure 6 As shown, the image generation apparatus includes: The information acquisition module 601 is used to acquire the description information to be expanded, which describes the content contained in the image to be generated. The information expansion module 602 is used to input the description information to be expanded into the description expansion model, and the description expansion model expands the content of the description information to be expanded and outputs the target description information. The generation module 603 is used to input the target description information as an image generation prompt into the image generation model, and the image generation model generates the corresponding image based on the image generation prompt and outputs the image. The description extension model is trained based on the above-described description extension model training method embodiment.

[0119] It should be noted that the apparatus provided in this embodiment can execute the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0120] This disclosure also provides an electronic device, which includes: a memory storing a computer program; and a processor for executing the computer program. When the computer program is executed by the processor, it can implement the described extended model training method or image generation method in the above method embodiments.

[0121] Example, Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device 700 in the embodiments of this disclosure. The electronic device 700 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0122] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0123] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0124] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the extended model training method or image generation method described in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the extended model training method or image generation method of the embodiments of this disclosure.

[0125] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM). ROM, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0126] In some implementations, clients and servers may communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and may interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0127] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0128] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to implement the aforementioned description of the extended model training method or image generation method.

[0129] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0132] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, and portable compact disc read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the described extended model training method or image generation method of any of the above embodiments. The execution method and beneficial effects are similar and will not be described again here.

[0135] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0136] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0137] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0138] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for describing the training of an extended model, characterized in that, The method includes: Acquire training data pairs, the training data pairs including first description information and second description information, the first description information being used to describe the elements contained in the sample image, the first description information conforming to the input paradigm of the image generation model, and the second description information being obtained by masking a portion of the information in the first description information; The second description information is input into the description expansion model, which expands the content of the second description information and outputs predicted description information. The description extension model is trained based on the difference between the first description information and the predicted description information; The trained description expansion model is used to expand the input information to output information that can serve as image generation prompts. These image generation prompts are then input into the image generation model to generate the corresponding image.

2. The method according to claim 1, characterized in that, The first descriptive information is obtained by the image recognition model recognizing the sample image, and the training data used when training the image generation model is generated based on the image recognition model.

3. The method according to claim 2, characterized in that, The image recognition model includes multiple element types; The acquisition of training data pairs includes: Acquire the sample image; Based on the image recognition model and the multiple element types, the sample image is identified to obtain the first description information, which is used to describe the elements contained in the sample image that belong to the element types. Based on a masking strategy, a portion of the information in the first description information is masked to obtain the second description information.

4. The method according to claim 3, characterized in that, The sample image is an interior design image, and the first description information also includes at least one of the design style or room type of the interior design image; The step of recognizing the sample image based on the image recognition model and the multiple element types to obtain the first descriptive information includes: The interior design image is input into the image recognition model, which then identifies the interior design image based on the multiple element types and outputs reference description information. The reference description information is used to describe the elements contained in the interior design image that belong to the element types. Add at least one of the design style or room type of the interior design image to the reference description information to obtain the first description information.

5. The method according to claim 3, characterized in that, The method of masking a portion of the information in the first description information to obtain the second description information based on a masking strategy includes: Based on the masking strategy, the target element type to be masked is randomly determined from the element types of the elements contained in the sample image. Based on the masking strategy and the target element type, target sub-information is randomly determined from the first description information, wherein the element described by the target sub-information belongs to the target element type; The target sub-information in the first description information is masked to obtain the second description information.

6. The method according to claim 5, characterized in that, The first description information includes at least one element type label, elements under each element type label, and attributes of each element, wherein the element type label is used to represent any one of the multiple element types; The step of randomly determining the target element type to be masked from the element types of the elements contained in the sample image based on the masking strategy includes: Based on the masking strategy, a first label and a second label are randomly determined from the at least one element type label, wherein the first label indicates the target element type and the second label is the element type label to be masked; The step of randomly determining target sub-information from the first description information based on the masking strategy and the target element type includes: Based on the masking strategy and the first label, the target sub-information is randomly determined from the elements and corresponding attributes under the first label in the first description information. The target sub-information is used to describe at least one of the elements or attributes of the elements in the sample image. The step of masking the target sub-information in the first description information to obtain the second description information includes: The target sub-information and the second tag in the first description information are masked to obtain the second description information.

7. The method according to any one of claims 1 to 6, characterized in that, The description expansion model is a large language model; the step of inputting the second description information into the description expansion model, and having the description expansion model expand the content of the second description information and output predicted description information includes: The second description information and the extended prompt information are input into the large language model, and the large language model expands the content of the second description information based on the extended prompt information to output the predicted description information; The extended prompt information instructs the large language model to perform an information expansion task on the input information.

8. An image generation method, characterized in that, The method includes: Obtain the description information to be expanded, which is used to describe the content contained in the image to be generated; The description information to be expanded is input into the description expansion model, which expands the content of the description information to be expanded and outputs the target description information. The target description information is used as an image generation prompt and input into the image generation model. The image generation model generates a corresponding image based on the image generation prompt and outputs the image. The description extension model is trained based on the description extension model training method described in any one of claims 1 to 6.

9. A device for describing an extended model training method, characterized in that, The device includes: The data pair acquisition module is used to acquire training data pairs, the training data pairs including first description information and second description information. The first description information is used to describe the elements contained in the sample image. The first description information conforms to the input paradigm of the image generation model. The second description information is obtained by masking a portion of the information in the first description information. The prediction expansion module is used to input the second description information into the description expansion model, and the description expansion model expands the content of the second description information to output the predicted description information. The training module is used to train the description extension model based on the difference between the first description information and the predicted description information; The trained description expansion model is used to expand the input information to output information that can serve as image generation prompts. These image generation prompts are then input into the image generation model to generate the corresponding image.

10. An electronic device, characterized in that, The electronic device includes: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the description extended model training method as described in any one of claims 1 to 7; or to implement the image generation method as described in claim 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for executing the description extended model training method as described in any one of claims 1 to 7; or for implementing the image generation method as described in claim 8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the description of the extended model training method as described in any one of claims 1 to 7; or, implements the image generation method as described in claim 8.

Images