Image generation method and device, storage medium and computer equipment

Abstract

The invention discloses an image generation method and device, a storage medium and computer equipment, relates to the technical field of image generation and the field of financial science and technology, and mainly aims at solving the problem that elements in an existing generated image do not meet layout requirements. Comprising the following steps: selecting an image text pair data sample with text length and data distribution meeting layout requirements from an image text pair data set; trimming a corresponding object picture from the image text pair data sample, and generating an explanation character corresponding to the object picture; performing division processing on image data in the image text pair data sample according to regions, and determining an overall layout label corresponding to the image data based on a region division result and description characters; and constructing a layout data set based on the overall layout label, and performing adjustment processing on the image generation model by adopting the layout data set to obtain a layout enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout enhanced image generation model.

Description

Technical Field

[0001] The present invention relates to the field of image generation technology and financial technology, and in particular to an image generation method and device, a storage medium, and a computer device. Background Art

[0002] Image generation is a very active research area in the field of computer vision with many practical application scenarios. The images generated by the diffusion model are more diverse and of higher quality than those generated by GAN. The training process of the diffusion model is also more stable, making image generation service businesses such as promotional marketing and creative design a reality.

[0003] In the actual business of auto insurance, life insurance, banking, commercial promotion, sales promotion, advertising, etc., when constructing pictures such as promotional posters, marketing posters, event posters, etc., there are often many requirements for the content layout of the pictures. For example, a car is required to be driving on the road, with modern buildings on the left, other elements on the right, and some other signs in front, etc. However, due to the randomness of the diffusion generation process and the lack of domain adaptation capabilities, the generated results often cannot directly meet the business needs. Although the generated image contains the required elements, the layout of each element in the picture does not meet the requirements. Summary of the invention

[0004] In view of this, the present invention provides an image generation method and apparatus, a storage medium, and a computer device, the main purpose of which is to solve the problem that each element in the image generated by the existing image generation method does not meet the layout requirements.

[0005] According to one aspect of the present invention, there is provided an image generation method, comprising:

[0006] Acquire an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set;

[0007] Trimming a corresponding object image from the image-text pair data sample, and generating a caption corresponding to the object image;

[0008] Dividing the image data in the image text data sample into regions, and determining an overall layout label corresponding to the image data based on the region division result and the explanatory text;

[0009] A layout data set is constructed based on the overall layout label, and the image generation model is adjusted using the layout data set to obtain a layout enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout enhanced image generation model.

[0010] Furthermore, selecting image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set includes:

[0011] Using a semantic segmentation and recognition model to perform segmentation and recognition processing on the image data in the image text pair data set, respectively, to obtain object and position information corresponding to each of the image data;

[0012] Performing scoring processing on each of the image data based on the layout requirement, the object and the position information to obtain a layout score for each of the image data;

[0013] The target image data meeting the layout score threshold is determined based on the layout score, and the image-text pair data corresponding to the target image data is determined as the image-text pair data sample meeting the layout requirement.

[0014] Furthermore, the step of trimming a corresponding object image from the image-text pair data sample and generating a caption corresponding to the object image includes:

[0015] Acquire position information and text data corresponding to the image data in the image-text pair data sample;

[0016] Trimming the corresponding object image from the image-text pair data sample based on the position information;

[0017] The text data and the corresponding object image are input into a multimodal pre-training model for similarity matching to obtain the caption corresponding to the object image.

[0018] Further, the determining of the overall layout label corresponding to the image data based on the region division result and the explanatory text includes:

[0019] Determine the objects corresponding to the respective regions based on the region division result, and obtain the description text corresponding to the objects;

[0020] Configuring a corresponding region name for each region, and constructing a basic layout tag corresponding to the image data based on the region name and the description text;

[0021] A relative position tag is constructed based on the basic layout tag, and the basic layout tag and the relative position tag are combined to obtain the overall layout tag.

[0022] Furthermore, the step of constructing a relative position tag based on the basic layout tag includes:

[0023] Extracting entities in the description text corresponding to each object in the basic layout label;

[0024] The relative position relationship between the regions in the region division result is obtained, and the relative position label is constructed based on the relative position relationship and the entity.

[0025] Furthermore, constructing a layout dataset based on the overall layout tag includes:

[0026] Acquire text data in the image-text pair data sample;

[0027] Calculating the cosine similarity between the overall layout label and the text data; and screening the image-text pair data samples based on the cosine similarity to obtain image-text pair data samples that meet the screening index;

[0028] The overall layout tag is used to optimize the image-text pair data samples that meet the screening index, and the optimized image-text pair data samples are stored in the layout data set.

[0029] Furthermore, the method further comprises:

[0030] Obtaining user evaluation information on the target image;

[0031] Performing update processing on the layout data set based on the evaluation information to obtain an updated layout data set;

[0032] The layout enhancement image generation model is optimized and adjusted using the updated layout data set to obtain an optimized layout enhancement image generation model.

[0033] According to another aspect of the present invention, there is provided an image generating device, comprising:

[0034] A sample selection module is used to obtain an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set;

[0035] A trimming and generating module, used for trimming corresponding object images from the image-text pair data sample, and generating captions corresponding to the object images;

[0036] A label determination module, used to divide the image data in the image text pair data sample into regions, and determine the overall layout label corresponding to the image data based on the region division result and the description text;

[0037] A model adjustment module is used to construct a layout data set based on the overall layout label, and use the layout data set to adjust the image generation model to obtain a layout enhanced image generation model, so as to generate a target image corresponding to the input text based on the layout enhanced image generation model.

[0038] Furthermore, the sample selection module includes:

[0039] A segmentation and recognition unit, used to perform segmentation and recognition processing on the image data in the image text pair data set respectively using a semantic segmentation and recognition model to obtain object and position information corresponding to each of the image data;

[0040] A scoring unit, configured to score each of the image data based on the layout requirement, the object and the position information to obtain a layout score for each of the image data;

[0041] The sample determination unit is used to determine the target image data that meets the layout score threshold based on the layout score, and determine the image-text pair data corresponding to the target image data as the image-text pair data sample that meets the layout requirement.

[0042] Furthermore, the pruning and generating module further includes:

[0043] An acquisition unit, used for acquiring position information and text data corresponding to the image data in the image-text pair data sample;

[0044] A trimming unit, configured to trim the corresponding object image from the image-text pair data sample based on the position information;

[0045] The matching processing unit is used to input the text data and the corresponding object image into the multimodal pre-training model for similarity matching processing to obtain the caption corresponding to the object image.

[0046] Furthermore, the label determination module is also used for:

[0047] Determine the objects corresponding to the respective regions based on the region division result, and obtain the description text corresponding to the objects;

[0048] Configuring a corresponding region name for each region, and constructing a basic layout tag corresponding to the image data based on the region name and the description text;

[0049] A relative position tag is constructed based on the basic layout tag, and the basic layout tag and the relative position tag are combined to obtain the overall layout tag.

[0050] Furthermore, the label determination module is also used for:

[0051] Extracting entities in the description text corresponding to each object in the basic layout label;

[0052] The relative position relationship between the regions in the region division result is obtained, and the relative position label is constructed based on the relative position relationship and the entity.

[0053] Furthermore, the model adjustment module includes a layout data set construction unit, and the layout data set construction unit is used to:

[0054] Acquire text data in the image-text pair data sample;

[0055] Calculating the cosine similarity between the overall layout label and the text data; and screening the image-text pair data samples based on the cosine similarity to obtain image-text pair data samples that meet the screening index;

[0056] The overall layout tag is used to optimize the image-text pair data samples that meet the screening index, and the optimized image-text pair data samples are stored in the layout data set.

[0057] Furthermore, the device also includes a model optimization module, which is used to:

[0058] Obtaining user evaluation information on the target image;

[0059] Performing update processing on the layout data set based on the evaluation information to obtain an updated layout data set;

[0060] The layout enhancement image generation model is optimized and adjusted using the updated layout data set to obtain an optimized layout enhancement image generation model.

[0061] According to another aspect of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the above-mentioned image generation method.

[0062] According to another aspect of the present invention, there is provided a computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0063] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned image generation method.

[0064] By means of the above technical scheme, the technical scheme provided by the embodiment of the present invention has at least the following advantages: the present invention provides an image generation method and device, a storage medium, and a computer device. Compared with the prior art, the present invention obtains an image-text pair data set, and selects an image-text pair data sample whose text length and data distribution meet the layout requirements from the image-text pair data set; trims the corresponding object image from the image-text pair data sample, and generates a caption corresponding to the object image; divides the image data in the image-text pair data sample by region, and determines the overall layout label corresponding to the image data based on the region division result and the caption; constructs a layout data set based on the overall layout label, and uses the layout data set to adjust the image generation model to obtain a layout-enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout-enhanced image generation model, thereby realizing layout control of the target image. The present invention obtains a layout data set carrying layout information by processing the existing image-text pair data set by sample selection, object image trimming, matching captions, dividing regions, and determining the overall layout label. Therefore, the layout enhanced image generation model adjusted by the layout data set has a higher layout capability, which can be used to control and guide image generation, thereby achieving the goal of improving the controllability of the generated image layout. The present invention greatly enhances its ability to generate images that meet the actual business requirements of auto insurance, life insurance, banking, commercial promotion, sales promotion, advertising, etc. by improving the controllability of the generated image layout, so the present invention has great business value.

[0065] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0067] Figure 1 A schematic diagram of a process of an image generating method provided by an embodiment of the present invention is shown;

[0068] Figure 2 A schematic diagram showing a flow chart of another image generating method provided by an embodiment of the present invention;

[0069] Figure 3A schematic diagram showing a flow chart of another image generating method provided by an embodiment of the present invention;

[0070] Figure 4 A schematic diagram showing a flow chart of another image generating method provided by an embodiment of the present invention;

[0071] Figure 5 A schematic diagram showing a flow chart of another image generating method provided by an embodiment of the present invention;

[0072] Figure 6 A schematic structural diagram of an image generating device provided by an embodiment of the present invention is shown;

[0073] Figure 7 A schematic structural diagram of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0074] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0075] The embodiment of the present invention provides an image generation method, such as Figure 1 As shown, the method includes:

[0076] 101. Obtain an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set;

[0077] In an embodiment of the present invention, the current execution end obtains an existing image-text pair data set, such as the clip400M data set, which collects 400 million (400M) image-text pair data in a network environment, and the embodiment of the present invention does not make specific limitations. The current execution end selects image-text pair data samples whose text length and data distribution meet the layout requirements from the existing image-text pair data set. The amount of sample selection can be specifically set as needed, such as selecting samples of specifications such as 1M, 10M, 15M, 20M, etc., and the embodiment of the present invention does not make specific limitations. Among them, the layout requirements can be judged by experienced personnel through scoring based on the text length and data distribution, and the embodiment of the present invention does not make specific limitations.

[0078] 102. Trimming a corresponding object image from the image-text pair data sample, and generating a caption corresponding to the object image;

[0079] In an embodiment of the present invention, the current execution end trims the corresponding object image from the image text pair data sample selected in step 101, such as trimming the corresponding object from the original image and fitting it to the white background of the minimum circumscribed rectangle to obtain the corresponding object image, which is not specifically limited in the embodiment of the present invention. After obtaining the object image, the current execution end also needs to configure corresponding captions for each object image. The captions include information such as the entity name, alias, common name, and colloquial name corresponding to the object image, which is not specifically limited in the embodiment of the present invention.

[0080] 103. Divide the image data in the image text data sample into regions, and determine an overall layout label corresponding to the image data based on the region division result and the explanatory text;

[0081] In an embodiment of the present invention, the current execution end divides the image data in the data sample of the image text selected in step 101 by region to obtain a region division result. Among them, the purpose of the region division process is to facilitate the individual setting of each divided region, and the region division can be performed specifically according to the layout requirements. For example, the entire image can be divided into five regions: left, right, top, bottom, and middle, and each region accounts for 20%; the entire image can also be divided into six regions: upper left, upper right, middle left, middle right, lower left, and lower right, and the proportion of each region is set as needed, etc., which is not specifically limited in the embodiment of the present invention. After performing the region division, the current execution end also determines the overall layout label corresponding to the image data based on the region division result and the explanatory text obtained in step 102. Among them, the overall layout label contains the explanatory text corresponding to each divided region, as well as the relative position information of the objects in each divided region, which is not specifically limited in the embodiment of the present invention.

[0082] 104. Construct a layout data set based on the overall layout label, and use the layout data set to adjust the image generation model to obtain a layout enhanced image generation model, so as to generate a target image corresponding to the input text based on the layout enhanced image generation model.

[0083] In the embodiment of the present invention, the current execution end optimizes each image-text pair data sample using the overall layout label obtained in step 103, so that each optimized image-text pair data sample contains the overall layout label, that is, contains the explanatory text corresponding to each divided area of ​​the image, and the relative position information of the objects in each divided area. The current execution end saves the optimized image-text pair data sample into a new data set, thereby obtaining a layout data set containing layout control information. Then, the layout data set containing layout control information is used to adjust the image generation model to obtain a layout enhanced image generation model. In the embodiment of the present invention, the image generation model is preferably a diffusion model. The diffusion model is a generation model based on a probability diffusion process. It generates high-quality data samples by a stepwise denoising method. It has the advantages of strong diversity, high quality, and stable training process in generating images, making the promotion and marketing, creative design, etc. of actual business of image generation service businesses such as auto insurance, life insurance, banking, product promotion, and advertising a reality. As a multimodal model, diffusion input includes images and texts, and its original semantic control ability mainly comes from the text model therein. Similarly, the text model's ability to understand the layout of each element in the text semantics indirectly reflects the ability of the diffusion model to control the layout when generating an image. The embodiment of the present invention improves the controllability of the layout of the image generated by diffusion by expanding the layout control ability of each element of the text model. Among them, the text model is preferably an open-clip-g model, which is a multimodal pre-trained neural network model, which is a variant of the CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is an open source, multimodal, zero-sample model. The learning method of the model is to pre-train with a large amount of paired data of images and texts to learn the alignment relationship between images and texts, so as to have the ability of multimodal learning, and can simultaneously understand the information of two different modes of images and texts, and establish a connection between them. In a specific auto insurance business scenario, a poster for promoting auto insurance is to be designed, requiring a car to travel on a road, with modern buildings on the left, other elements on the right, and some other signs in front, etc., and then the target image that meets the above layout requirements is obtained by inputting text.

[0084] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to select samples that are more suitable for layout control and to facilitate the rapid acquisition of images of various objects in the image text pair data set, another image generation method is provided, such as Figure 2 As shown, the step selects image-text pair data samples whose text length and data distribution meet the layout requirements from the image-text pair data set, including:

[0085] 201. Use a semantic segmentation and recognition model to perform segmentation and recognition processing on the image data in the image text pair data set respectively, and obtain object and position information corresponding to each of the image data;

[0086] In the embodiment of the present invention, the current execution end uses a semantic segmentation model to perform segmentation and recognition processing on the image data in the image text pair data set, and identifies the objects and position information corresponding to each image data. For example, if it is identified that image data A includes objects such as cars and buildings, the corresponding car and car position information, as well as the building and building position information are obtained; if it is identified that image data B includes balloons, bicycles, and roads, the corresponding balloon and balloon position information, bicycle and bicycle position information, and road and road position information are obtained, etc., which are not specifically limited in the embodiment of the present invention.

[0087] 202. Perform scoring processing on each of the image data based on the layout requirement, the object, and the position information to obtain a layout score for each of the image data;

[0088] 203. Determine target image data that meets a layout score threshold based on the layout score, and determine the image-text pair data corresponding to the target image data as the image-text pair data sample that meets the layout requirement.

[0089] In an embodiment of the present invention, the current execution end scores each image data based on the text length and data distribution artificially set in the layout requirements, as well as the objects corresponding to each image data and the position information of the objects obtained in step 201, to obtain a layout score for each image data. Then, a set layout score threshold is obtained, the layout score of each image data is compared with the set layout score threshold, and the image data with a layout score greater than the layout score threshold is determined as the target image data. It should be noted that the layout score threshold can be adjusted according to the required number of samples. If the required number of samples is large, the set layout score threshold can be relatively reduced; if the required number of samples is small, the set layout score threshold can be relatively increased. The embodiment of the present invention does not make specific limitations. The current execution end determines the image-text pair data corresponding to the obtained target image data as the image-text pair data sample that meets the layout requirements.

[0090] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to extract each object from the selected image-text pair data sample and obtain accurate description text corresponding to the object, another image generation method is provided, such as Figure 3As shown, the step of trimming the corresponding object image from the image-text pair data sample and generating a caption corresponding to the object image includes:

[0091] 301. Obtaining position information and text data corresponding to image data in the image-text pair data sample;

[0092] 302. Trim the corresponding object image from the image-text pair data sample based on the position information;

[0093] In the embodiment of the present invention, since the object and position information corresponding to the image data have been analyzed in step 201, the current execution end directly obtains the object, position information and text data corresponding to the image data in the image-text pair data sample. Among them, the text data is the original text data in the image-text pair data sample. Then, the current execution end crops the corresponding object from the image-text pair data sample based on the position information of the object, and can fit the cropped object to the white background of the minimum circumscribed rectangle to obtain the corresponding object image, which is not specifically limited in the embodiment of the present invention.

[0094] 303. Input the text data and the corresponding object image into a multimodal pre-training model for similarity matching to obtain the caption corresponding to the object image.

[0095] In an embodiment of the present invention, the current execution end inputs the text data and the corresponding object image into the multimodal pre-training model for similarity matching processing to obtain the caption corresponding to the object image. Among them, the multimodal pre-training model preferably uses the BLIP (Bootstrapping Language-Image Pre-training) model, which aims to achieve unified understanding and generation of visual and language information. The model enables the model to understand the information of images and texts at the same time by jointly training the image encoder and the text encoder. In an embodiment of the present invention, the BLIP model achieves similarity matching between object images and text data by encoding the object images and text data into the same vector space, and obtains the caption corresponding to the object image, which can significantly improve the accuracy and recall rate.

[0096] Further, as a refinement and extension of the above-mentioned specific implementation methods, in order to improve the controllability of the generated picture layout, another image generation method is provided, such as Figure 4 As shown, the step of determining the overall layout label corresponding to the image data based on the area division result and the description text includes:

[0097] 401. Determine an object corresponding to each region based on the region division result, and obtain the description text corresponding to the object;

[0098] In the embodiment of the present invention, the current execution end obtains the region division result after the image data is divided by region in step 101, and determines the objects corresponding to each region based on each region divided in the region division result. For example, when the image data C is divided by region, it is divided into five regions: left, right, top, bottom, and middle. Then, based on the region division result, object 1 corresponding to the left region, object 3 corresponding to the right region, object 4 corresponding to the top region, object 5 corresponding to the bottom region, object 2 corresponding to the middle region, etc. are determined, and the embodiment of the present invention does not make specific limitations.

[0099] In an embodiment of the present invention, the current execution end also obtains the explanatory text corresponding to each object, such as obtaining the explanatory text "object 1caption" corresponding to object 1 in the left area of ​​image data C; obtaining the explanatory text "object 3caption" corresponding to object 3 in the right area of ​​image data C; obtaining the explanatory text "object 4caption" corresponding to object 4 in the upper area of ​​image data C, obtaining the explanatory text "object 5caption" corresponding to object 5 in the lower area of ​​image data C, obtaining the explanatory text "object 2caption" corresponding to object 2 in the middle area of ​​image data C, etc., which is not specifically limited in the embodiment of the present invention.

[0100] 402. Assign a corresponding region name to each region, and construct a basic layout tag corresponding to the image data based on the region name and the description text;

[0101] In an embodiment of the present invention, the current execution end configures corresponding area names for each area. The area names can be directly named using the directions "up", "down", "left", "right", "middle", "upper left", "lower right", etc., and can also be named using codes for each direction, such as giving each direction a code. The code can include text, numbers, symbols, etc., which are not specifically limited in the embodiment of the present invention.

[0102] In the embodiment of the present invention, the current execution end also constructs a basic layout label corresponding to the image data based on the configured region name and the caption text corresponding to each object in the image data. For example, the basic layout label of the image data C is "left: object 1 caption, right: object 3 caption, top: object 4 caption, bottom: object 5 caption, middle: object 2 caption", etc., which is not specifically limited in the embodiment of the present invention.

[0103] 403. Construct a relative position tag based on the basic layout tag, and combine the basic layout tag with the relative position tag to obtain the overall layout tag.

[0104] In the embodiment of the present invention, the current execution end reconstructs the corresponding relative position tag based on the basic layout tag constructed in step 401. The specific construction method of the relative position tag includes:

[0105] a. Use a named entity extraction method to extract entities in the captions corresponding to each object in the basic layout label; for example, extract entity 1 from the caption "object 1 caption", extract entity 3 from the caption "object 3 caption", extract entity 4 from the caption "object 4 caption", extract entity 5 from the caption "object 5 caption", extract entity 2 from the caption "object 2 caption", etc., which is not specifically limited in the embodiments of the present invention.

[0106] b. The current execution end obtains the relative position relationship between the regions in the region division result, and constructs a relative position label based on the relative position relationship and the entities extracted from the description text. For example, the constructed relative position label is "entity 1 is on the left of entity 3, entity 4 is above entity 5", etc., which is not specifically limited in the embodiment of the present invention.

[0107] Further, as a refinement and extension of the above-mentioned specific implementation method, in order to add layout-related information to the image-text pair data sample, another image generation method is provided, such as Figure 5 As shown, the step of constructing a layout dataset based on the overall layout label includes:

[0108] 501. Obtain text data in the image-text pair data sample;

[0109] In the embodiment of the present invention, the current execution end obtains text data in the image-text pair data sample, wherein the text data is the original text data in the image-text pair data sample.

[0110] 502. Calculate the cosine similarity between the overall layout label and the text data; and screen the image-text pair data samples based on the cosine similarity to obtain image-text pair data samples that meet the screening index;

[0111] In an embodiment of the present invention, the current execution end compares the overall layout label of the text type constructed in step 403 with the original text data in the image-text pair data sample, such as using the cosine similarity calculation method to compare the similarity between the two. After obtaining the cosine similarity between the two, the current execution end screens the image-text pair data sample based on the cosine similarity, such as deleting samples based on the z-score being less than -3 or greater than 3, which is not specifically limited in the embodiment of the present invention. After the screening process is performed, an image-text pair data sample that meets the screening criteria is obtained. The image-text pair data sample that meets the screening criteria obtained in this step has a high similarity with the overall layout label constructed in step 403, that is, a sample whose description of the text data is relatively consistent with the layout of the image data is screened out from the image-text pair data sample.

[0112] 503. Use the overall layout tag to optimize the image-text pair data samples that meet the screening index, and store the optimized image-text pair data samples into the layout data set.

[0113] In an embodiment of the present invention, the current execution end uses the overall layout label obtained in steps 401 to 403 to optimize the image-text pair data sample that meets the filtering criteria, such as combining the overall layout label with the text data in the image-text pair data sample that meets the filtering criteria to obtain an optimized image-text pair data sample, and then storing the optimized image-text pair data sample into the constructed layout data set, thereby obtaining a layout data set containing layout control information.

[0114] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to further improve the layout control capability of the generated image, another image generation method is provided, the method further comprising:

[0115] Obtaining user evaluation information on the target image;

[0116] Performing update processing on the layout data set based on the evaluation information to obtain an updated layout data set;

[0117] The layout enhancement image generation model is optimized and adjusted using the updated layout data set to obtain an optimized layout enhancement image generation model.

[0118] In an embodiment of the present invention, the current execution end obtains the user's evaluation information on the target image obtained in step 104; then, the layout data set is updated based on the evaluation information to obtain an updated layout data set; such as storing the target image that the user is satisfied with in the evaluation information as a new image text data sample into the constructed layout data set, etc., the embodiment of the present invention does not make specific limitations. Next, the layout enhancement image generation model is optimized and adjusted using the updated layout data set to obtain an optimized layout enhancement image generation model. It can be seen from this that the solution of the present invention has a strong expansion capability. By grasping the data set construction process, a general model can be constructed for different business units BU (Business Unit) and different business needs such as auto insurance, life insurance, banks, product promotions, and advertising. It can also construct scenario models according to segmented businesses, and has strong business adaptation capabilities in the field of financial technology.

[0119] The embodiment of the present invention provides an image generation method. Compared with the prior art, the present invention obtains an image-text pair data set, and selects an image-text pair data sample whose text length and data distribution meet the layout requirements from the image-text pair data set; trims the corresponding object image from the image-text pair data sample, and generates a caption corresponding to the object image; divides the image data in the image-text pair data sample by region, and determines the overall layout label corresponding to the image data based on the region division result and the caption; constructs a layout data set based on the overall layout label, and uses the layout data set to adjust the image generation model to obtain a layout-enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout-enhanced image generation model, thereby realizing layout control of the target image. The present invention obtains a layout data set carrying layout information by processing the existing image-text pair data set through sample selection, object image trimming, matching caption, dividing regions, and determining the overall layout label. Therefore, the layout-enhanced image generation model adjusted by the layout data set has a higher layout capability, which can be used to control the generation of guidance images, so as to achieve the goal of improving the controllability of the generated image layout. The present invention improves the controllability of the generated image layout, thereby greatly enhancing the ability to generate images that meet the actual business requirements of auto insurance, life insurance, banking, commercial promotion, sales promotion, advertising, etc., so the present invention has great business value.

[0120] As the above Figure 1 To implement the method shown in FIG. 1 , an embodiment of the present invention provides an image generating device, such as Figure 6 As shown, the device comprises:

[0121] A sample selection module 61 is used to obtain an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set;

[0122] A trimming and generating module 62, used for trimming the corresponding object image from the image-text pair data sample, and generating a caption corresponding to the object image;

[0123] A label determination module 63, configured to divide the image data in the image text pair data sample into regions, and determine an overall layout label corresponding to the image data based on the region division result and the description text;

[0124] The model adjustment module 64 is used to construct a layout data set based on the overall layout label, and use the layout data set to adjust the image generation model to obtain a layout enhanced image generation model, so as to generate a target image corresponding to the input text based on the layout enhanced image generation model.

[0125] Furthermore, the sample selection module 61 includes:

[0126] A segmentation and recognition unit, used to perform segmentation and recognition processing on the image data in the image text pair data set respectively using a semantic segmentation and recognition model to obtain object and position information corresponding to each of the image data;

[0127] A scoring unit, configured to score each of the image data based on the layout requirement, the object and the position information to obtain a layout score for each of the image data;

[0128] The sample determination unit is used to determine the target image data that meets the layout score threshold based on the layout score, and determine the image-text pair data corresponding to the target image data as the image-text pair data sample that meets the layout requirement.

[0129] Furthermore, the pruning and generating module 62 further includes:

[0130] An acquisition unit, used for acquiring position information and text data corresponding to the image data in the image-text pair data sample;

[0131] A trimming unit, configured to trim the corresponding object image from the image-text pair data sample based on the position information;

[0132] The matching processing unit is used to input the text data and the corresponding object image into the multimodal pre-training model for similarity matching processing to obtain the caption corresponding to the object image.

[0133] Furthermore, the label determination module 63 is also used for:

[0134] Determine the objects corresponding to the respective regions based on the region division result, and obtain the description text corresponding to the objects;

[0135] Configuring a corresponding region name for each region, and constructing a basic layout tag corresponding to the image data based on the region name and the description text;

[0136] A relative position tag is constructed based on the basic layout tag, and the basic layout tag and the relative position tag are combined to obtain the overall layout tag.

[0137] Furthermore, the label determination module 63 is also used for:

[0138] Extracting entities in the description text corresponding to each object in the basic layout label;

[0139] The relative position relationship between the regions in the region division result is obtained, and the relative position label is constructed based on the relative position relationship and the entity.

[0140] Furthermore, the model adjustment module 64 includes a layout data set construction unit, and the layout data set construction unit is used to:

[0141] Acquire text data in the image-text pair data sample;

[0142] Calculating the cosine similarity between the overall layout label and the text data; and screening the image-text pair data samples based on the cosine similarity to obtain image-text pair data samples that meet the screening index;

[0143] The overall layout tag is used to optimize the image-text pair data samples that meet the screening index, and the optimized image-text pair data samples are stored in the layout data set.

[0144] Furthermore, the device also includes a model optimization module, which is used to:

[0145] Obtaining user evaluation information on the target image;

[0146] Performing update processing on the layout data set based on the evaluation information to obtain an updated layout data set;

[0147] The layout enhancement image generation model is optimized and adjusted using the updated layout data set to obtain an optimized layout enhancement image generation model.

[0148] The embodiment of the present invention provides an image generation device. Compared with the prior art, the present invention obtains an image-text pair data set, and selects an image-text pair data sample whose text length and data distribution meet the layout requirements from the image-text pair data set; trims the corresponding object image from the image-text pair data sample, and generates a caption corresponding to the object image; divides the image data in the image-text pair data sample by region, and determines the overall layout label corresponding to the image data based on the region division result and the caption; constructs a layout data set based on the overall layout label, and uses the layout data set to adjust the image generation model to obtain a layout-enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout-enhanced image generation model, thereby realizing layout control of the target image. The present invention obtains a layout data set carrying layout information by processing the existing image-text pair data set through sample selection, object image trimming, matching caption, dividing regions, and determining the overall layout label. Therefore, the layout-enhanced image generation model adjusted by the layout data set has a higher layout capability, which can be used to control the generation of guidance images, so as to achieve the goal of improving the controllability of the generated image layout. The present invention improves the controllability of the generated image layout, thereby greatly enhancing the ability to generate images that meet the actual business requirements of auto insurance, life insurance, banking, commercial promotion, sales promotion, advertising, etc., so the present invention has great business value.

[0149] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the image generation method in any of the above method embodiments.

[0150] Figure 7 A schematic diagram of the structure of a computer device provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0151] like Figure 7 As shown, the computer device may include: a processor (processor) 702 , a communication interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 .

[0152] The processor 702 , the communication interface 704 , and the memory 706 communicate with each other via a communication bus 708 .

[0153] The communication interface 704 is used to communicate with other devices such as clients or other servers.

[0154] The processor 702 is used to execute the program 710, and specifically can execute the relevant steps of the above-mentioned image generation method.

[0155] Specifically, the program 710 may include program codes, which include computer operation instructions.

[0156] The processor 702 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0157] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0158] The program 710 may be specifically configured to enable the processor 702 to perform the following operations:

[0159] Acquire an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set;

[0160] Trimming a corresponding object image from the image-text pair data sample, and generating a caption corresponding to the object image;

[0161] Dividing the image data in the image text data sample into regions, and determining an overall layout label corresponding to the image data based on the region division result and the explanatory text;

[0162] A layout data set is constructed based on the overall layout label, and the image generation model is adjusted using the layout data set to obtain a layout enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout enhanced image generation model.

[0163] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: include: Acquire an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set; Trimming a corresponding object image from the image-text pair data sample, and generating a caption corresponding to the object image; Dividing the image data in the image text data sample into regions, and determining an overall layout label corresponding to the image data based on the region division result and the explanatory text; A layout data set is constructed based on the overall layout label, and the image generation model is adjusted using the layout data set to obtain a layout enhanced image generation model, so that a target image corresponding to the input text is generated based on the layout enhanced image generation model.

2. The method according to claim 1, characterized in that The step of selecting image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set includes: Using a semantic segmentation and recognition model to perform segmentation and recognition processing on the image data in the image text pair data set, respectively, to obtain object and position information corresponding to each of the image data; Performing scoring processing on each of the image data based on the layout requirement, the object and the position information to obtain a layout score for each of the image data; The target image data meeting the layout score threshold is determined based on the layout score, and the image-text pair data corresponding to the target image data is determined as the image-text pair data sample meeting the layout requirement.

3. The method according to claim 1, characterized in that The step of trimming a corresponding object image from the image-text pair data sample and generating a caption corresponding to the object image includes: Acquire position information and text data corresponding to the image data in the image-text pair data sample; Trimming the corresponding object image from the image-text pair data sample based on the position information; The text data and the corresponding object image are input into a multimodal pre-training model for similarity matching to obtain the caption corresponding to the object image.

4. The method according to claim 1, characterized in that: The determining of the overall layout label corresponding to the image data based on the region division result and the description text includes: Determine the objects corresponding to the respective regions based on the region division result, and obtain the description text corresponding to the objects; Configuring a corresponding region name for each region, and constructing a basic layout tag corresponding to the image data based on the region name and the description text; A relative position tag is constructed based on the basic layout tag, and the basic layout tag and the relative position tag are combined to obtain the overall layout tag.

5. The method according to claim 4, characterized in that The step of constructing a relative position tag based on the basic layout tag includes: Extracting entities in the description text corresponding to each object in the basic layout label; The relative position relationship between the regions in the region division result is obtained, and the relative position label is constructed based on the relative position relationship and the entity.

6. The method according to claim 1, characterized in that The constructing a layout dataset based on the overall layout label includes: Acquire text data in the image-text pair data sample; Calculating the cosine similarity between the overall layout label and the text data; and screening the image-text pair data samples based on the cosine similarity to obtain image-text pair data samples that meet the screening index; The overall layout tag is used to optimize the image-text pair data samples that meet the screening index, and the optimized image-text pair data samples are stored in the layout data set.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining user evaluation information on the target image; Performing update processing on the layout data set based on the evaluation information to obtain an updated layout data set; The layout enhancement image generation model is optimized and adjusted using the updated layout data set to obtain an optimized layout enhancement image generation model.

8. An image generating device, characterized in that: include: A sample selection module is used to obtain an image-text pair data set, and select image-text pair data samples whose text length and data distribution meet layout requirements from the image-text pair data set; A trimming and generating module, used for trimming corresponding object images from the image-text pair data sample, and generating captions corresponding to the object images; A label determination module, used to divide the image data in the image text pair data sample into regions, and determine the overall layout label corresponding to the image data based on the region division result and the description text; A model adjustment module is used to construct a layout data set based on the overall layout label, and use the layout data set to adjust the image generation model to obtain a layout enhanced image generation model, so as to generate a target image corresponding to the input text based on the layout enhanced image generation model.

9. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction executes an operation corresponding to the image generation method according to any one of claims 1 to 7.

10. A computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the image generation method according to any one of claims 1 to 7.

Images