Image generation model training and image generation method

Abstract

The disclosure provides an image generation model training and image generation method, and belongs to the technical field of image processing. The method constructs an image training set, which includes an image training subset and a structure feature training subset, and the image training subset corresponds to the structure feature training subset; a first target image generation model is obtained by training a first to-be-trained model based on a first feature point of any structure feature in the structure feature training subset and an image feature of an image corresponding to the structure feature, so that the structure feature and the image feature are fused in the image generation process; further, a second target image generation model is obtained by training a second to-be-trained model based on a second feature point of any structure feature in the structure feature training subset and an image feature of an image corresponding to the structure feature; since the second feature point is a target feature point in the first feature point, the quality of image generation is improved based on the first training model.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image generation model training and image generation method. Background Technology

[0002] Image generation involves using text content to describe the information of the image to be generated, and then employing an image generation model to produce an image corresponding to that text content. However, in the process of generating images from text content, it is impossible to use other types of content to describe the information of the image to be generated, thus preventing the generation of a more accurate image. Summary of the Invention

[0003] This disclosure proposes an image generation model training method and an image generation method.

[0004] The first aspect of this disclosure proposes an image generation model training method, including:

[0005] Construct an image training set, which includes an image training subset and a structural feature training subset, wherein the image training subset and the structural feature training subset correspond one-to-one;

[0006] Based on the first feature point of any structural feature in the structural feature training subset and the image features of the image corresponding to the structural feature, a first model to be trained is trained to obtain a first target image generation model;

[0007] Based on the second feature point of any structural feature in the structural feature training subset and the image features of the image corresponding to the structural feature, a second training model is trained to obtain a second target image generation model, wherein the second feature point is the target feature point among the first feature points.

[0008] In this embodiment of the disclosure, constructing the image training set includes:

[0009] Obtain the initial image set;

[0010] For any initial image in the initial image set, extract the segmentation mask of the initial image;

[0011] Based on the initial image and the segmentation mask, extract the image feature map of the initial image;

[0012] Based on the relationship between the segmentation mask and the image feature map, determine whether the initial image meets the preset standard;

[0013] The image training subset is constructed by selecting grayscale images corresponding to the initial images that meet the preset criteria from the initial image set; and the structural feature training subset is constructed by using the image feature maps corresponding to the initial images in the image training subset, wherein the grayscale images corresponding to the initial images are obtained by setting the pixel values ​​of non-image features in the initial images to grayscale values, and the non-image features are obtained by subtracting the image feature map from the initial images.

[0014] In this embodiment of the disclosure, the step of extracting the image feature map of the initial image based on the initial image and the segmentation mask includes:

[0015] The initial structural features of the initial image are extracted using the segmentation mask;

[0016] The pixel values ​​of the initial structural features are mapped to the corresponding pixel points in the preset image to obtain the image feature map.

[0017] In this embodiment of the disclosure, determining whether the initial image meets a preset standard based on the relationship between the segmentation mask and the image feature map includes:

[0018] Based on the segmentation mask and the image feature map, the intersection-union ratio (IUGR) is calculated.

[0019] If the intersection-over-union ratio is greater than a preset threshold, then the initial image meets the preset standard;

[0020] If the intersection-to-union ratio is less than or equal to the preset threshold, then the initial image does not meet the preset standard.

[0021] In this embodiment of the disclosure, the step of training a first training model based on any structural feature in the structural feature training subset and the image features of the image corresponding to the structural feature to obtain a first target image generation model includes:

[0022] For any structural feature in the structural feature training subset, the first structural point feature of the first feature point of the structural feature is extracted using the first model to be trained;

[0023] Based on the first structural point features and the image features of the corresponding images in the image training subset, the first loss value of the first model to be trained is calculated.

[0024] Based on the relationship between the first loss value and the first preset loss standard, the first model to be trained is optimized to output the first target image generation model.

[0025] In this embodiment of the disclosure, the step of training a second model based on the second feature point of any structural feature in the structural feature training subset and the image features of the corresponding image of the structural feature to obtain a second target image generation model includes:

[0026] For any structural feature in the structural feature training subset, the second structural point feature of the second feature point of the structural feature is extracted using the second model to be trained;

[0027] The second structural point feature and the image features of the corresponding image in the image training subset are fused to obtain the semantic features of the second structural point feature;

[0028] Calculate the second loss value of the second model to be trained based on the semantic features and the second feature points;

[0029] Based on the relationship between the second loss value and the second preset loss standard, the second model to be trained is optimized to output the second target image generation model.

[0030] A second aspect of this disclosure provides an image generation method, the method comprising:

[0031] Obtain the structural features of the image to be generated;

[0032] Using the first target image generation model, an initial target image is obtained based on the structural features and a preset generated image;

[0033] Using a second target image generation model, the initial target image is optimized based on the structural features to generate a target image. The first target image generation model and the second target image generation model are obtained according to the method described in the first aspect and any optional embodiment of the first aspect.

[0034] In this embodiment of the disclosure, the preset generated image is an image generated based on the text information corresponding to the image to be generated.

[0035] In this embodiment of the disclosure, the step of obtaining an initial target image based on the structural features and a preset generated image using a first target image generation model includes:

[0036] Using the first target image generation model, a first structural point feature is generated based on the first feature point of the structural feature, wherein the first feature point is all the feature points of the structural feature;

[0037] Calculate the pixel loss value between the first structural point feature and the preset generated image;

[0038] Based on the relationship between the pixel loss value and the first preset pixel standard, the preset generated image is optimized to obtain the initial target generated image.

[0039] In this embodiment of the disclosure, the step of using a second target image generation model to optimize the initial target image based on the structural features to generate a target image includes:

[0040] Using the second target image generation model, a second structural point feature is obtained based on the second feature point of the structural feature, where the second feature point is the target feature point of the structural feature;

[0041] The second structural point features and the image features of the initial target generated image are fused to obtain fused features;

[0042] The structural loss value of the second training model is calculated based on the fusion features and the second structural point features.

[0043] Based on the relationship between the structural loss value and the second preset structural standard, the fusion features are optimized, and the target image is output.

[0044] An embodiment of the third aspect of this disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described in the first aspect or any optional embodiment of the first aspect, the second aspect, and any optional embodiment of the second aspect.

[0045] An embodiment of the fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the methods described in the first aspect and any optional embodiments of the first aspect, the second aspect and any optional embodiments of the second aspect.

[0046] The technical solutions provided in this disclosure have at least the following technical effects or advantages:

[0047] An image training set is constructed, comprising an image training subset and a structural feature training subset, with a one-to-one correspondence between the two. Based on the first feature point of any structural feature in the structural feature training subset and the image features of the corresponding image, a first training model is trained to obtain a first target image generation model, thus achieving the fusion of structural features and image features during the image generation process. Further, based on the second feature point of any structural feature in the structural feature training subset and the image features of the corresponding image, a second training model is trained to obtain a second target image generation model. Since the second feature point is the target feature point among the first feature points, the quality of subsequent image generation is improved based on the first training model.

[0048] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of this disclosure. Attached Figure Description

[0049] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0050] In the attached diagram:

[0051] Figure 1 A flowchart of an image generation model training method provided in an embodiment of this disclosure is shown;

[0052] Figure 2 This diagram illustrates the training of a first model to be trained in an image generation model training method provided in an embodiment of the present disclosure.

[0053] Figure 3 This diagram illustrates the training of a second model to be trained in an image generation model training method provided in an embodiment of the present disclosure.

[0054] Figure 4 A flowchart of an image generation method provided in an embodiment of this disclosure is shown;

[0055] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown;

[0056] Figure 6 A schematic diagram of a storage medium provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0057] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0058] It should be noted that, unless otherwise stated, the technical or scientific terms used in this disclosure shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains.

[0059] This disclosure proposes an image generation model training method, such as... Figure 1 The image shown is an embodiment of an image generation model training method provided by this disclosure, including the following steps:

[0060] In step S11, an image training set is constructed.

[0061] The image training set includes an image training subset and a structural feature training subset, with a one-to-one correspondence between the image training subset and the structural feature training subset.

[0062] For example, the image training subset and the structural feature training subset in the image training set correspond to each other. According to the application scenario of this disclosure embodiment, the image training set can be constructed according to the type of application scenario. Specifically, if the image generation model is used for generating images of human structures after training, then the image training subset in the image training set includes human images, and the structural feature training subset includes the structural features corresponding to the human images. If the image generation model is used for generating images of buildings after training, then the image training subset in the image training set includes building images, and the structural feature training subset includes the structural features corresponding to the building images.

[0063] It is important to note that for the same application type and corresponding scenario, the image should also be of the same type. For example, for a scenario involving human anatomy, all images should include human anatomy. This disclosure uses an application scenario that generates images of human anatomy as an example.

[0064] In some embodiments, when obtaining the image training set, it is necessary to map the image training subset to the structural feature training subset. Therefore, the corresponding structural features in the structural feature training subset can be generated based on the images in the image training subset. Specifically, the image training set can be determined in the following way.

[0065] Obtain an initial image set; for any initial image in the initial image set, extract the segmentation mask of the initial image; based on the initial image and the segmentation mask, extract the image feature map of the initial image; based on the relationship between the segmentation mask and the image feature map, determine whether the initial image meets the preset criteria; select grayscale images corresponding to the initial images that meet the preset criteria from the initial image set to construct an image training subset; and use the image feature maps corresponding to the initial images in the image training subset to construct a structural feature training subset.

[0066] The grayscale image corresponding to the initial image is obtained by setting the pixel values ​​of non-image features in the initial image to grayscale values, and the non-image features are obtained by subtracting the image feature map from the initial image.

[0067] For example, all images in the initial training image set include human figures. For the initial images in the initial training image set, a Mask R-CNN network can be used to extract segmentation masks of the human figures in the initial images. The image feature maps can be obtained by extracting human feature maps from the images or by encoding the images. The intersection-over-union (IoU) ratio between the segmentation mask and the image feature maps is calculated. A subset of grayscale images corresponding to the initial images that meet a preset standard is selected from the initial image set to construct the image training subset. The preset standard can be that an IoU ratio greater than 0.5 is considered to meet the preset standard.

[0068] Specifically, based on the segmentation mask and image feature map, the intersection-union ratio (IUR) is calculated; if the IUR is greater than a preset threshold, the initial image meets the preset standard; if the IUR is less than or equal to the preset threshold, the initial image does not meet the preset standard.

[0069] In some embodiments, the image feature map can also be obtained by: extracting the initial structural features of the initial image using a segmentation mask; mapping the pixel values ​​of the initial structural features to the corresponding pixel points in the preset image to obtain the image feature map.

[0070] For example, the initial image can be a grayscale image or a color image.

[0071] The segmentation mask is a two-dimensional array of the same size as the initial image, where pixel values ​​represent the category or region affiliation of each pixel. The segmentation mask is used to extract pixels for a specific region from the initial image. Specifically, this is achieved by multiplying the pixel-by-pixel mask by the pixel-by-pixel value of the image (regions with a mask value of 1 are retained, while regions with a value of 0 are ignored).

[0072] Structural features are extracted from the regions extracted from the mask. The preset image can be a blank image of the same size as the initial image, or it can be another image. The pixel values ​​of the extracted structural features (such as a structural map) are mapped to the corresponding positions in the preset image. This is usually done by assigning values ​​pixel by pixel.

[0073] In step S12, the first model to be trained is trained based on the first feature point of any structural feature in the structural feature training subset and the image features of the image corresponding to the structural feature, thereby obtaining the first target image generation model.

[0074] For example, in this embodiment of the disclosure, a Vision Transformer (ViT) network can be used to extract a first feature point from any structural feature in the structural feature training subset. The first feature point can be a feature point that preserves all human structures, for example, 27533 points. Furthermore, a Contrastive Language-Image Pre-training (CLIP) image encoder can be used to encode the images in the image training subset to obtain image features.

[0075] Locate the key first feature points from the structural features. For example, corner detection algorithms (such as Harris corner detection) or feature point extraction algorithms (such as SIFT, SURF) can be used to locate feature points.

[0076] The input to the first model to be trained includes the first feature points of the structural features. The output of the first model to be trained is the predicted image features, that is, the global image features predicted from the feature points of the input structural features.

[0077] The first model to be trained can choose a suitable model architecture, such as: Fully connected neural network: If the input is a feature vector, a fully connected neural network can be used. Convolutional neural network (CNN): If the input is a feature map, a CNN can be used. Transformer architecture: If you need to process sequential feature points, you can use a Transformer.

[0078] The data pairs of structural point features and image features corresponding to the structural features are used, that is, the image features of the corresponding images in the image training subset are used as the ground truth, while the predicted image features obtained by the first model to be trained based on the structural point features are used as the training values. An appropriate loss function is selected, such as mean squared error (MSE) or cross-entropy loss, to calculate the difference between the ground truth and the training values. The model parameters are adjusted using optimization algorithms (such as Adam or SGD) to minimize the loss function, thereby obtaining the first target image generation model.

[0079] Furthermore, the first target image generation model can be optimized and evaluated. Based on the evaluation results, the model architecture or training parameters can be adjusted, such as increasing the number of layers, adjusting the learning rate, or using data augmentation. The training and evaluation process is repeated until the first target image generation model achieves satisfactory performance.

[0080] In some embodiments, step S12 above can also be implemented in the following way: for any structural feature in the structural feature training subset, extract the first structural point feature of the first feature point of the structural feature using the first model to be trained; calculate the first loss value of the first model to be trained based on the first structural point feature and the image features of the corresponding image in the image training subset; optimize the first model to be trained based on the relationship between the first loss value and the first preset loss standard, and output the first target image generation model.

[0081] For example, such as Figure 2 As shown, in this embodiment, the first model to be trained can be a ViT model. The ViT model is used to extract the first structural point features of the first feature points. The first loss value can be calculated using a cosine similarity comparison loss function to calculate the first loss value between the first structural point features and the image features. Optimization algorithms (such as SGD or Adam) can be used to update the ViT model parameters based on the gradient of the first loss value. The learning rate is dynamically adjusted according to the training progress, for example, using a learning rate decay strategy. If the loss value no longer decreases within a certain number of iterations, training is stopped to prevent overfitting.

[0082] In step S13, a second training model is trained based on the second feature point of any structural feature in the structural feature training subset and the image features of the corresponding image of the structural feature to obtain a second target image generation model. The second feature point is the target feature point in the first feature point.

[0083] For example, the method of training the second model to be trained is different from that in step S12 above. After the second structural point feature is predicted by the second feature point of any structural feature, the second structural point feature needs to be fused with the image feature and then the loss value is calculated with the second feature point. Then, the parameters of the second model to be trained are adjusted according to the loss value.

[0084] In some embodiments, step S13 includes: for any structural feature in the structural feature training subset, extracting the second structural point feature of the second feature point of the structural feature using the second model to be trained; fusing the second structural point feature and the image features of the corresponding image in the image training subset to obtain the semantic feature of the second structural point feature; calculating the second loss value of the second model to be trained based on the semantic feature and the second feature point; optimizing the second model to be trained based on the relationship between the second loss value and the second preset loss standard, and outputting the second target image generation model.

[0085] For example, the second structural point feature corresponding to the second feature point can be generated by the first target image generation model obtained in step S12, or it can be generated by other models or networks, which are not limited here.

[0086] like Figure 3 As shown, the human image is processed by the CLIP image encoder to obtain image feature vectors K and V. The second structural point features are used as the Q vector as input to the second model to be trained, and feature fusion is performed to obtain semantic features. Further, based on the semantic features and the second structural point features, a second loss value is calculated, and the parameters of the second model to be trained are further adjusted until the second loss value meets the requirements, thus obtaining the second target image generation model.

[0087] The image generation model training method of this application constructs an image training set, which includes an image training subset and a structural feature training subset, with a one-to-one correspondence between the image training subset and the structural feature training subset. Based on the first feature point of any structural feature in the structural feature training subset and the image features of the corresponding image, a first training model is trained to obtain a first target image generation model, thus realizing the fusion of structural features and image features in the image generation process. Further, based on the second feature point of any structural feature in the structural feature training subset and the image features of the corresponding image, a second training model is trained to obtain a second target image generation model. Since the second feature point is the target feature point among the first feature points, the quality of subsequent image generation is improved based on the first training model.

[0088] Corresponding to the above embodiments, this disclosure also provides an image generation method, which is as described above. Figure 1 The application methods of the image reconstruction model obtained by the image generation model training method shown are as follows: Figure 4 As shown, the method includes:

[0089] In step S41, the structural features of the image to be generated are obtained.

[0090] For example, when generating images, structural features can be used to provide generation guidance based on related image generation, thereby improving image accuracy. For instance, in this embodiment of the disclosure, when generating an image of a human body structure, the structural features can be the human body structure information corresponding to the image to be generated. Specifically, this can be the position of each key point in the human body structure, as well as the positional relationship between each point.

[0091] In step S42, the first target image generation model is used to obtain the initial target image based on structural features and a preset generated image.

[0092] For example, a preset generated image is a blank image with the same height and width as the image to be generated. A blank image refers to an image in which pixel values ​​are default values. In some embodiments, the preset generated image may also be a preliminary image pre-generated based on text content.

[0093] Further, the structural features are input into the first target image generation model trained in the above embodiments to generate an initial target image. Specifically, to obtain a more accurate initial target image, in some embodiments, step S42 can also be implemented in the following way: using the first target image generation model, based on the first feature points of the structural features, a first structural point feature is generated, where the first feature points are all the feature points of the structural features; the pixel loss value between the first structural point feature and the preset generated image is calculated; based on the relationship between the pixel loss value and the first preset pixel standard, the preset generated image is optimized to obtain the initial target generated image.

[0094] For example, a first target image is processed using a first target image generation model to extract all feature points (first feature points) of its structural features. Based on the extracted first feature points, first structural point features are generated. This may involve converting the feature points into feature vectors or feature maps for subsequent processing.

[0095] Design a loss function to calculate the pixel loss between the first structural point features and the pre-generated image. Commonly used loss functions include mean squared error (MSE), absolute error (MAE), or more complex perceptual loss.

[0096] The first structural point feature is compared with the preset generated image, and the pixel loss value is calculated using the defined loss function. An optimization algorithm is selected to minimize the pixel loss value. Common optimization algorithms include gradient descent (such as SGD, Adam, etc.) or advanced optimization methods based on genetic algorithms, particle swarm optimization, etc. The parameters of the optimization algorithm (such as learning rate, number of iterations, etc.) are initialized. In each iteration, the pixel values ​​of the preset generated image are updated using the optimization algorithm based on the current pixel loss value of the preset generated image. The iteration process is repeated until the preset stopping condition is met (such as loss convergence, reaching the maximum number of iterations, etc.). After the optimization process is completed, the optimized preset generated image is obtained, which is the initial target generated image. The initial target generated image is verified to meet the expected structural feature requirements.

[0097] In step S43, the initial target image is optimized based on structural features using the second target image generation model to generate the target image. The first target image generation model and the second target image generation model are trained according to the above embodiments.

[0098] For example, the accuracy of the initial target image generated in step S42 above does not meet the application requirements in practical applications. Therefore, it is necessary to further utilize the second target image generation model to optimize the initial target image based on the second feature points in the structural features to obtain the target image.

[0099] Unlike step S42 above, after predicting the second structural point feature through the second feature point of any structural feature, the second structural point feature needs to be fused with the initial target generated image feature, and then the loss value is calculated with the second feature point. Then, the parameters of the second training model are adjusted based on the loss value.

[0100] Specifically, step S43 above can also be implemented in the following way: using the second target image generation model, based on the second feature points of the structural features, a second structural point feature is obtained, where the second feature point is the target feature point of the structural features; feature fusion is performed on the second structural point feature and the image features of the initial target generated image to obtain fused features; the structural loss value of the second model to be trained is calculated based on the fused features and the second structural point feature; based on the relationship between the structural loss value and the second preset structural standard, the fused features are optimized, and the target image is output.

[0101] Specifically, this optimization method is basically the same as the training method of the second target image generation model in the above embodiment. The difference is that the adjustment of the model parameters is changed to the adjustment of the initial target image, so that the target image is obtained after the iteration stopping condition is reached.

[0102] The image generation method of this application obtains the structural features of the image to be generated; using a first target image generation model, an initial target image is obtained based on the structural features and a preset generated image; using a second target image generation model, the initial target generated image is optimized based on the structural features to generate the target image. Through a two-stage generation process, the first stage generates the initial target image, and the second stage further optimizes the image details. This staged approach can gradually improve the quality of the generated image and reduce the errors that may arise from a single generation. The second target image generation model can focus on optimizing the details, texture, and structure of the image based on the first stage, thereby generating a more realistic and higher-quality image. It can better capture the core information of the input features and avoid overfitting the noise of the training data during a single generation process.

[0103] Corresponding to the above implementation methods, this disclosure also provides an image generation model training apparatus for performing the above-described methods. Figure 1 The illustrated embodiment of the image generation model training method, the image generation model training apparatus includes:

[0104] A dataset construction module is used to construct an image training set, which includes an image training subset and a structural feature training subset, with each image training subset corresponding to the structural feature training subset.

[0105] The first training module is used to train the first training model based on the first feature point of any structural feature in the structural feature training subset and the image features of the image corresponding to the structural feature, thereby obtaining the first target image generation model.

[0106] The second training module is used to train a second model to be trained based on the second feature point of any structural feature in the structural feature training subset and the image features of the image corresponding to the structural feature, thereby obtaining a second target image generation model, wherein the second feature point is the target feature point among the first feature points.

[0107] The image generation model training apparatus provided in the above embodiments of this disclosure and the image generation model training method provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0108] Corresponding to the above implementation methods, this disclosure also provides an image reconstruction apparatus for performing the above-described methods. Figure 4 An image reconstruction method according to any of the illustrated embodiments, the image reconstruction apparatus comprising:

[0109] The feature acquisition module is used to acquire the structural features of the image to be generated;

[0110] The first generation module is used to obtain an initial target image based on the structural features and a preset generation image using a first target image generation model;

[0111] The second generation module is used to optimize the initial target image based on the structural features using the second target image generation model to generate a target image. The first target image generation model and the second target image generation model are obtained according to the above embodiments.

[0112] The image generation apparatus and the image generation method provided in the above embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0113] This disclosure also provides an electronic device for performing the above-described method. Please refer to... Figure 5 This illustrates a schematic diagram of an electronic device provided by some embodiments of the present disclosure. For example... Figure 5 As shown, the electronic device includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the aforementioned provisions of this disclosure. Figure 1 or Figure 4 The method provided by any of the illustrated embodiments.

[0114] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0115] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 501 is used to store programs, and the processor 500 executes the programs after receiving execution instructions. Figure 1 or Figure 5 The illustrated method can be applied to or implemented by the processor 500.

[0116] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0117] The electronic devices and methods provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0118] This disclosure also provides a computer-readable storage medium corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 6 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored, which, when run by a processor, executes the methods provided in any of the foregoing embodiments.

[0119] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0120] The computer-readable storage medium provided in the above embodiments of this disclosure and the methods provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0121] It should be noted that:

[0122] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0123] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of this disclosure, various features of this disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting a schematic diagram in which the claimed disclosure requires more features than are explicitly stated in each embodiment. The inventive aspect lies in fewer than all features of a single foregoing disclosed embodiment. Therefore, the implementation following the detailed description is hereby explicitly incorporated into this detailed description, wherein each embodiment itself serves as a separate embodiment of this disclosure.

[0124] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this disclosure and form different embodiments.

[0125] The above description is merely a preferred embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training an image generation model, characterized in that, The method includes: Construct an image training set, which includes an image training subset and a structural feature training subset, wherein the image training subset and the structural feature training subset correspond one-to-one; For any structural feature in the structural feature training subset, the first structural point feature of the first feature point of the structural feature is extracted using the first model to be trained; based on the first structural point feature and the image features of the corresponding image in the image training subset, the first loss value of the first model to be trained is calculated; based on the relationship between the first loss value and the first preset loss standard, the first model to be trained is optimized to output the first target image generation model. For any structural feature in the structural feature training subset, a second structural point feature of the second feature point of the structural feature is extracted using a second model to be trained; the second structural point feature and the image features of the corresponding image in the image training subset are fused to obtain the semantic feature of the second structural point feature; a second loss value of the second model to be trained is calculated based on the semantic feature and the second feature point; the second model to be trained is optimized based on the relationship between the second loss value and the second preset loss standard, and a second target image generation model is output; the second feature point is the target feature point among the first feature points.

2. The method according to claim 1, characterized in that, The construction of the image training set includes: Obtain the initial image set; For any initial image in the initial image set, extract the segmentation mask of the initial image; Based on the initial image and the segmentation mask, extract the image feature map of the initial image; Based on the relationship between the segmentation mask and the image feature map, determine whether the initial image meets the preset standard; The image training subset is constructed by selecting grayscale images corresponding to the initial images that meet the preset criteria from the initial image set; and the structural feature training subset is constructed by using the image feature maps corresponding to the initial images in the image training subset, wherein the grayscale images corresponding to the initial images are obtained by setting the pixel values ​​of non-image features in the initial images to grayscale values, and the non-image features are obtained by subtracting the image feature map from the initial images.

3. The method according to claim 2, characterized in that, The step of extracting the image feature map of the initial image based on the initial image and the segmentation mask includes: The initial structural features of the initial image are extracted using the segmentation mask; The pixel values ​​of the initial structural features are mapped to the corresponding pixel points in the preset image to obtain the image feature map.

4. The method according to claim 2, characterized in that, The step of determining whether the initial image meets the preset criteria based on the relationship between the segmentation mask and the image feature map includes: Based on the segmentation mask and the image feature map, the intersection-union ratio (IUGR) is calculated. If the intersection-over-union ratio is greater than a preset threshold, then the initial image meets the preset standard; If the intersection-to-union ratio is less than or equal to the preset threshold, then the initial image does not meet the preset standard.

5. An image generation method, characterized in that, The method includes: Obtain the structural features of the image to be generated; Using the first target image generation model, an initial target image is obtained based on the structural features and a preset generated image; Using a second target image generation model, the initial target image is optimized based on the structural features to generate a target image, wherein the first target image generation model and the second target image generation model are obtained according to the method described in any one of claims 1-4.

6. The method according to claim 5, characterized in that, The preset generated image is an image generated based on the text information corresponding to the image to be generated.

7. The method according to claim 5, characterized in that, The step of using the first target image generation model to obtain the initial target image based on the structural features and the preset generated image includes: Using the first target image generation model, a first structural point feature is generated based on the first feature point of the structural feature, wherein the first feature point is all the feature points of the structural feature; Calculate the pixel loss value between the first structural point feature and the preset generated image; Based on the relationship between the pixel loss value and the first preset pixel standard, the preset generated image is optimized to obtain the initial target generated image.

8. The method according to claim 5, characterized in that, The step of using a second target image generation model to optimize the initial target image based on the structural features and generate a target image includes: Using the second target image generation model, a second structural point feature is obtained based on the second feature point of the structural feature, where the second feature point is the target feature point of the structural feature; The second structural point features and the image features of the initial target generated image are fused to obtain fused features; The structural loss value of the second training model is calculated based on the fusion features and the second structural point features. Based on the relationship between the structural loss value and the second preset structural standard, the fusion features are optimized, and the target image is output.

Images