Training method and device of image generation model and image generation method and device

Abstract

Embodiments of the invention disclose an image generation model training method and apparatus, and an image generation method and apparatus. The method comprises the steps of obtaining a first text feature vector of a training text; performing low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text; performing image generation on the second text feature vector through a generator network in a to-be-trained image generation model to obtain a first prediction image of the training text; the first prediction image is discriminated through a discriminator network in the to-be-trained image generation model, and a discrimination result of the first prediction image is obtained; according to the discrimination result, constructing a first loss of the to-be-trained image generation model; and updating model parameters of the to-be-trained image generation model according to the first loss.

Description

Technical Field

[0001] The present disclosure relates to the field of image generation technology, and more specifically, to a training method and device for an image generation model, and an image generation method and device. Background Art

[0002] In recent years, Generative Adversarial Networks (GAN) has been applied to traditional broadcasting and television and online audio-visual fields as one of the mainstream technologies for image generation. For example, in broadcasting and television news reports, image generation technology can automatically generate virtual scenes and characters based on text descriptions to assist in quickly restoring news events. However, the images generated by traditional generative adversarial networks are of low quality. Summary of the invention

[0003] The embodiments of the present disclosure provide a method and device for training an image generation model, and a method and device for generating an image.

[0004] According to a first aspect of the present disclosure, a method for training an image generation model is provided, the method comprising:

[0005] Obtain a first text feature vector of a training text;

[0006] Performing low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text;

[0007] Performing image generation on the second text feature vector by using a generator network in an image generation model to be trained to obtain a first predicted image of the training text;

[0008] discriminate the first predicted image by using a discriminator network in the image generation model to be trained to obtain a discriminant result of the first predicted image;

[0009] Constructing a first loss of the image generation model to be trained according to the discrimination result;

[0010] Update model parameters of the image generation model to be trained according to the first loss.

[0011] Optionally, performing low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text includes:

[0012] Performing low-rank mapping processing on the first text feature vector to obtain a low-rank mapping expression of the first text feature vector;

[0013] Add disturbance noise to the low-rank mapping expression to obtain a second text feature vector of the training text.

[0014] Optionally, constructing a first loss of the image generation model to be trained according to the discrimination result includes:

[0015] Obtaining an evaluation value of the first predicted image;

[0016] According to the discrimination result and the evaluation value, a first loss of the image generation model to be trained is constructed.

[0017] Optionally, the evaluation value includes at least one of the following: image fidelity, semantic rationality;

[0018] Wherein, the image fidelity includes at least one of the following: global fidelity, local detail fidelity;

[0019] The semantic rationality includes at least one of the following: semantic consistency and relational logical rationality.

[0020] Optionally, the first loss includes a second loss of the generator network and a third loss of the discriminator network,

[0021] The updating of the model parameters of the image generation model to be trained according to the first loss includes:

[0022] updating a first network parameter of the generator network according to the second loss; and,

[0023] According to the third loss, updating the second network parameters of the discriminator network;

[0024] Among them, the model parameters of the image generation model to be trained include the first network parameters and the second network parameters.

[0025] Optionally, updating the model parameters of the image generation model to be trained according to the first loss includes:

[0026] In a first training phase of the image generation model to be trained, updating a first network parameter of the generator network according to the second loss, and updating a second network parameter of the discriminator network according to the third loss;

[0027] In the second training phase of the image generation model to be trained, the second network parameters are fixed, and the first network parameters of the generator network are updated according to the second loss.

[0028] Optionally, the method further comprises:

[0029] In a first training phase of the image generation model to be trained, adjusting the update frequency of the discriminator network according to the discrimination result;

[0030] The update frequency is the frequency of updating the first network parameter.

[0031] According to a second aspect of the present disclosure, there is provided an image generation method, the method comprising:

[0032] Get the text to be predicted;

[0033] Generate an image of the text to be predicted by using a generator network in a trained image generation model to obtain a second predicted image of the text to be predicted;

[0034] Wherein, the image generation model is trained according to the training method of the image generation model described in the first aspect above.

[0035] According to a third aspect of the present disclosure, a training device for an image generation model is provided, the device comprising:

[0036] An acquisition module, used for acquiring a first text feature vector of a training text;

[0037] A processing module, used for performing low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text;

[0038] An image generation module, used for performing image generation on the second text feature vector through a generator network in an image generation model to be trained, so as to obtain a first predicted image of the training text;

[0039] An image discrimination module, configured to discriminate the first predicted image through a discriminator network in the image generation model to be trained, and obtain a discrimination result of the first predicted image;

[0040] A construction module, used for constructing a first loss of the image generation model to be trained according to the discrimination result;

[0041] An updating module is used to update the model parameters of the image generation model to be trained according to the first loss.

[0042] According to a fourth aspect of the present disclosure, there is provided an image generating device, the device comprising:

[0043] An acquisition module, used to acquire the text to be predicted;

[0044] An image generation module, used to generate an image of the text to be predicted by using a generator network in a trained image generation model to obtain a second predicted image of the text to be predicted;

[0045] Wherein, the image generation model is trained according to the training method of the image generation model described in the first aspect above.

[0046] According to the training method of the image generation model of the embodiment of the present disclosure, the first text feature vector of the training text is first subjected to low-rank mapping processing, and after obtaining the second text feature vector of the training text, the second text feature vector is input into the image generation model to be trained for image generation, which is equivalent to adding a low-rank prior as a regularization term in the generator network. The low-rank regularization term effectively reduces the offset of the generated distribution in high dimensions, enhances global consistency, and significantly improves the quality of the generated image; and the first loss is dynamically adjusted during the training process according to the discrimination result of the discriminator network for the first predicted image, and the first predicted image is generated based on low-rank features. In this way, the first loss can not only capture the intrinsic low-dimensional structural information of the real distribution by using low-rank features, but also can be adjusted in an adaptive manner. Through the first loss, the balance between the original space constraints and the low-rank space constraints can be adaptively controlled, so that the trained generator network not only matches the overall distribution, but also approaches the key low-rank features, thereby improving the quality of the generated image.

[0047] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0049] Figure 1 is a schematic diagram of the hardware configuration of an electronic device according to an embodiment of the present disclosure;

[0050] Figure 2 is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure;

[0051] Figure 3 is a flowchart of a method for training an image generation model according to an example of the present disclosure;

[0052] Figure 4 is a flowchart of an image generating method according to an embodiment of the present disclosure;

[0053] Figure 5 is a principle block diagram of a training device for an image generation model according to an embodiment of the present disclosure;

[0054] Figure 6 is a principle block diagram of an image generating device according to an embodiment of the present disclosure;

[0055] Figure 7 is a schematic diagram of the hardware configuration of an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0057] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0058] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0059] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0060] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0061] <Hardware Configuration>

[0062] Figure 1 is a block diagram of a hardware configuration of an electronic device 1000 according to an embodiment of the present disclosure.

[0063] In one embodiment, the electronic device 1000 may be a server or a terminal device. The server may be an integrated server or a distributed server across multiple computers or computer data centers. The terminal device may be a portable computer, a desktop computer, a wearable device, or any other device having a computing device such as a processor and a storage device such as a memory, which is not limited in this embodiment.

[0064] like Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like.

[0065] The processor 1100 may be a mobile version processor. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, a USB interface, a headphone interface, etc. The communication device 1400 may, for example, be capable of wired or wireless communication. The communication device 1400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1500 may be, for example, a liquid crystal display, a touch display, etc. The input device 1600 may include, for example, a touch screen, a keyboard, etc. The electronic device 1000 may output audio information through a speaker 1700, and may collect audio information through a microphone 1800.

[0066] Despite Figure 1 Although multiple devices are shown for the electronic device 1000, the present disclosure may only involve some of the devices, for example, the electronic device 1000 only involves the memory 1200 and the processor 1100.

[0067] In the embodiment of the present disclosure, the memory 1200 of the electronic device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to execute the training method of the image generation model or the image generation method provided by the embodiment of the present disclosure.

[0068] In the above description, the technician can design instructions according to the scheme disclosed in the present disclosure. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0069] <Method Example 1>

[0070] In this embodiment, a training method for an image generation model is provided. The training method for the image generation model can be implemented by an electronic device, and the electronic device can be as follows: Figure 1 An electronic device 1000 is shown.

[0071] according to Figure 2 As shown, the training method of the image generation model of the embodiment of the present disclosure may include the following steps S2100 to S2600.

[0072] Step S2100, obtaining a first text feature vector of a training text.

[0073] Step S2200: Perform low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text.

[0074] In one example, before the first text feature vector is input into the generator network in the image generation model to be trained for image generation, in order to more effectively reduce the offset of the generated distribution in high dimensions, the first text feature vector can be processed by low-rank mapping to obtain a low-rank mapping expression, and the low-rank mapping expression is input into the generator network in the image generation model to be trained as the second text feature vector of the training text. This is equivalent to adding a low-rank prior as a regularization term in the generator network, which effectively reduces the offset of the generated distribution in high dimensions, enhances global consistency, and significantly improves the quality of the generated image.

[0075] In one example, this step S2200 performs low-rank mapping processing on the first text feature vector to obtain the second text feature vector of the training text, which may further include: performing low-rank mapping processing on the first text feature vector to obtain a low-rank mapping expression of the first text feature vector; adding disturbance noise to the low-rank mapping expression to obtain the second text feature vector of the training text.

[0076] In this example, refer to Figure 3 After the first text feature vector is processed by low-rank mapping to obtain a low-rank mapping expression, a small disturbance noise is added to the low-rank expression to obtain a second text feature vector of the training text. The purpose of adding disturbance noise is to make the second text feature vector disordered so that the image generation model to be trained can better generate images based on the second text feature vector.

[0077] Step S2300: performing image generation on the second text feature vector through a generator network in an image generation model to be trained to obtain a first predicted image of the training text.

[0078] The generator network is used to generate an image for the second text feature vector.

[0079] In this embodiment, refer to Figure 3 , the second text feature vector is input into the generator network in the image generation model to be trained, and the second text feature vector can be image generated through the generator network to obtain the first predicted image of the training text.

[0080] Step S2400: discriminate the first predicted image through the discriminator network in the image generation model to be trained to obtain a discrimination result of the first predicted image.

[0081] The discriminator network is used to determine whether the first predicted image is associated with the training text.

[0082] In this embodiment, refer to Figure 3 , the first predicted image is input into the discriminator network in the image generation model to be trained, and the first predicted image can be discriminated by the discriminator network to obtain the discrimination result of the first predicted image.

[0083] Step S2500: construct a first loss of the image generation model to be trained according to the discrimination result.

[0084] The first loss is used to update the model parameters of the image generation model to be trained, and the first loss includes the second loss of the generator network and the third loss of the discriminator network. Correspondingly, the model parameters of the image generation model to be trained include the network parameters of the generator network and the network parameters of the discriminator network. The second loss is used to update the network parameters of the generator network, and the third loss is used to update the network parameters of the discriminator network.

[0085] In an optional embodiment, this step S2500, based on the discrimination result, constructing the first loss of the image generation model to be trained may further include: obtaining an evaluation value of the first predicted image; and constructing the first loss of the image generation model to be trained based on the discrimination result and the evaluation value.

[0086] The evaluation value includes at least one of the following: image fidelity and semantic rationality. The image fidelity includes at least one of the following: global fidelity and local detail fidelity; the semantic rationality includes at least one of the following: semantic consistency and relational logic rationality.

[0087] Among them, the global realism uses a pre-trained image classification model to determine whether the overall scene layout and style of the first predicted image are natural and realistic.

[0088] The local detail fidelity uses a GAN-based authenticity detection model to judge the texture quality of the local area of ​​the first predicted image.

[0089] Among them, semantic consistency determines whether each element in the first predicted image conforms to the description of the training text through semantic segmentation and object detection.

[0090] The logical rationality of the relationship is used to evaluate whether the relationship between the instances in the first predicted image conforms to common sense. The relationship between the instances includes, for example but not limited to, relative positions and objects used in conjunction.

[0091] Reference Figure 3After the first predicted image is generated, on the one hand, the evaluation value of the first predicted image can be obtained, and on the other hand, the first predicted image is discriminated according to the discriminator network to obtain a discrimination result, and then the first loss is constructed according to the evaluation value and the discrimination result of the discriminator network.

[0092] In this embodiment, the effect of the generated first predicted image is evaluated based on the two dimensions of image fidelity and semantic rationality, wherein image fidelity includes two sub-dimensions of global fidelity and local detail fidelity, and semantic rationality includes two sub-dimensions of semantic consistency and relational logical rationality, and each sub-dimension can be represented by a value between 0 and 1. The weighted average of these sub-dimensions can be used as the evaluation value of the first predicted image, and the first loss can be constructed based on the evaluation value and the discrimination result of the discriminator network, and then the model parameters of the image generation model to be trained are updated based on the first loss.

[0093] Through this embodiment, the first loss can not only utilize low-rank features to capture the intrinsic low-dimensional structural information of the real distribution, but also can be adjusted in an adaptive manner. Through this first loss, the balance between the original spatial constraints and the low-rank spatial constraints can be adaptively controlled, so that the trained generator network not only matches the overall distribution, but also approaches the key low-rank features, thereby improving the quality of the generated image.

[0094] Step S2600: Update the model parameters of the image generation model to be trained according to the first loss.

[0095] In an example, updating the model parameters of the image generation model to be trained according to the first loss in step S2600 may further include: updating the first network parameters of the generator network according to the second loss; and updating the second network parameters of the discriminator network according to the third loss.

[0096] In an example, updating the model parameters of the image generation model to be trained according to the first loss in step S2600 may further include: in the first training stage of the image generation model to be trained, updating the first network parameters of the generator network according to the second loss, and updating the second network parameters of the discriminator network according to the third loss; in the second training stage of the image generation model to be trained, fixing the second network parameters and updating the first network parameters of the generator network according to the second loss.

[0097] The first training stage is usually the early training stage of the image generation model to be trained, and the second training stage is usually the late training stage of the image generation model to be trained. Usually, due to the obvious asymmetry between the capabilities of the discriminator network and the generator network, if the traditional GAN ​​training method is directly adopted, it is easy to cause the generator network to be overly biased to adapt to the discriminator network, resulting in problems such as mode collapse.

[0098] In this example, in the first training stage, the first network parameters of the generator network are updated according to the second loss, and the second network parameters of the discriminator network are updated according to the third loss. In the second training stage, the second network parameters of the discriminator network are fixed, and only the first network parameters of the generator network are updated according to the second loss. In this way, the generator network can be forced to converge to the real data distribution and avoid over-approaching the discriminator network.

[0099] In one example, the training method of the image generation model of the embodiment of the present disclosure further includes: in the first training stage of the image generation model to be trained, adaptively updating the update frequency of the discriminator network according to the discrimination result.

[0100] The update frequency is the frequency of updating the first network parameter.

[0101] In this embodiment, refer to Figure 3 In order to avoid over-optimization of the discriminator network, in the first training stage of the image generation model to be trained, the update frequency of the discriminator network is adaptively updated according to the discriminator network's discrimination result for the first predicted image. For example, in the early stage of training, when the discrimination result is poor, the update frequency of the discriminator network can be reduced to avoid overfitting, and when the discrimination result is good, the update frequency of the discriminator network can be increased, so as to avoid over-optimization of the discriminator network.

[0102] According to the method of the embodiment of the present disclosure, the first text feature vector of the training text is first subjected to low-rank mapping processing, and after obtaining the second text feature vector of the training text, the second text feature vector is input into the image generation model to be trained for image generation, which is equivalent to adding a low-rank prior as a regularization term in the generator network. The low-rank regularization term effectively reduces the offset of the generated distribution in high dimensions, enhances global consistency, and significantly improves the quality of the generated image; and the first loss is dynamically adjusted during the training process according to the discrimination result of the discriminator network for the first predicted image, and the first predicted image is generated based on low-rank features. In this way, the first loss can not only utilize low-rank features to capture the intrinsic low-dimensional structural information of the real distribution, but also can be adjusted in an adaptive manner. Through the first loss, the balance between the original space constraints and the low-rank space constraints can be adaptively controlled, so that the trained generator network not only matches the overall distribution, but also approaches the key low-rank features, thereby improving the quality of the generated image.

[0103] <Method Example 2>

[0104] In this embodiment, an image generation method is provided. The image generation method can be implemented by an electronic device. The electronic device can be as follows: Figure 1 An electronic device 1000 is shown.

[0105] according to Figure 4 As shown, the image generating method of the embodiment of the present disclosure may include the following steps S4100 to S4200.

[0106] Step S4100, obtaining the text to be predicted.

[0107] Step S4200: Generate an image of the text to be predicted by using a generator network in a trained image generation model to obtain a second predicted image of the text to be predicted.

[0108] The image generation model is trained according to the image generation model training method mentioned in the first embodiment of the above method.

[0109] According to the embodiments of the present disclosure, the quality of the generated image can be improved by performing image generation on the text to be predicted using the trained image generation model.

[0110] <Device Example 1>

[0111] In this embodiment, a training device 500 for an image generation model is provided. Figure 5 As shown, the training device 500 for the image generation model may include an acquisition module 510, a processing module 520, an image generation module 530, an image discrimination module 540, a construction module 550 and an update module 560.

[0112] An acquisition module 510 is used to acquire a first text feature vector of a training text;

[0113] A processing module 520 is used to perform low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text;

[0114] An image generation module 530 is used to generate an image of the second text feature vector by using a generator network in an image generation model to be trained, so as to obtain a first predicted image of the training text;

[0115] An image discrimination module 540, configured to discriminate the first predicted image through a discriminator network in the image generation model to be trained, and obtain a discrimination result of the first predicted image;

[0116] A construction module 550, configured to construct a first loss of the image generation model to be trained according to the discrimination result;

[0117] An updating module 560 is used to update the model parameters of the image generation model to be trained according to the first loss.

[0118] In one embodiment, the processing module 520 is specifically used to perform low-rank mapping processing on the first text feature vector to obtain a low-rank mapping expression of the first text feature vector; and add disturbance noise to the low-rank mapping expression to obtain a second text feature vector of the training text.

[0119] In one embodiment, the construction module 550 is specifically used to obtain an evaluation value of the first predicted image; and construct a first loss of the image generation model to be trained according to the discrimination result and the evaluation value.

[0120] In one embodiment, the evaluation value includes at least one of the following: image fidelity, semantic rationality;

[0121] Wherein, the image fidelity includes at least one of the following: global fidelity, local detail fidelity;

[0122] The semantic rationality includes at least one of the following: semantic consistency and relational logical rationality.

[0123] In one embodiment, the first loss includes the second loss of the generator network and the third loss of the discriminator network. The updating module 560 is specifically used to update the first network parameters of the generator network according to the second loss and update the second network parameters of the discriminator network according to the third loss in the first training stage of the image generation model to be trained; fix the second network parameters in the second training stage of the image generation model to be trained, and update the first network parameters of the generator network according to the second loss.

[0124] In one embodiment, the updating module 560 is further used to adaptively update the updating frequency of the discriminator network according to the discrimination result in the first training stage of the image generation model to be trained;

[0125] The update frequency is the frequency of updating the first network parameter.

[0126] According to this embodiment, the first text feature vector of the training text is first subjected to low-rank mapping processing, and after obtaining the second text feature vector of the training text, the second text feature vector is input into the image generation model to be trained for image generation, which is equivalent to adding a low-rank prior as a regularization term in the generator network. The low-rank regularization term effectively reduces the offset of the generated distribution in high dimensions, enhances global consistency, and significantly improves the quality of the generated image; and the first loss is dynamically adjusted during the training process according to the discrimination result of the discriminator network for the first predicted image, and the first predicted image is generated based on low-rank features. In this way, the first loss can not only utilize low-rank features to capture the intrinsic low-dimensional structural information of the real distribution, but also can be adjusted in an adaptive manner. Through the first loss, the balance between the original space constraints and the low-rank space constraints can be adaptively controlled, so that the trained generator network not only matches the overall distribution, but also approaches the key low-rank features, thereby improving the quality of the generated image.

[0127] <Device Example 2>

[0128] In this embodiment, an image generating device 600 is provided. Figure 6 As shown, the image generating device 600 may include an acquisition module 610 and an image generating module 620 .

[0129] An acquisition module 610 is used to acquire the text to be predicted;

[0130] An image generation module 620 is used to generate an image of the text to be predicted by using a generator network in a trained image generation model to obtain a second predicted image of the text to be predicted;

[0131] The image generation model is trained according to the image generation model training method described in the first embodiment of the above method.

[0132] According to the embodiments of the present disclosure, the quality of the generated image can be improved by performing image generation on the text to be predicted using the trained image generation model.

[0133] <Equipment Embodiment>

[0134] Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. Figure 7 As shown, the electronic device 700 includes a processor 710 and a memory 720 .

[0135] The memory 720 may be used to store executable computer instructions.

[0136] The processor 710 can be used to execute the training method of the image generation model or the image generation method according to the embodiment of the method of the present disclosure under the control of the executable computer instructions.

[0137] The electronic device 700 may be Figure 1 The electronic device 1000 shown may also be a device with other hardware structures, which is not limited here.

[0138] In another embodiment, the electronic device 7000 may include the training device 500 or the image generating device 600 of the above image generating model.

[0139] In one embodiment, each module of the above image generation model training device 500 or image generation device 600 can be implemented by the processor 710 running computer instructions stored in the memory 720 .

[0140] <Computer Readable Storage Medium>

[0141] The embodiments of the present disclosure also provide a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the training method of the image generation model or the image generation method provided by the embodiments of the present disclosure is executed.

[0142] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0143] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0144] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0145] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0146] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0149] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0150] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims. .

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Obtain a first text feature vector of a training text; Performing low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text; Performing image generation on the second text feature vector by using a generator network in an image generation model to be trained to obtain a first predicted image of the training text; discriminate the first predicted image by using a discriminator network in the image generation model to be trained to obtain a discriminant result of the first predicted image; Constructing a first loss of the image generation model to be trained according to the discrimination result; Update model parameters of the image generation model to be trained according to the first loss.

2. The method according to claim 1, characterized in that The step of performing low-rank mapping on the first text feature vector to obtain a second text feature vector of the training text comprises: Performing low-rank mapping processing on the first text feature vector to obtain a low-rank mapping expression of the first text feature vector; Add disturbance noise to the low-rank mapping expression to obtain a second text feature vector of the training text.

3. The method according to claim 1, characterized in that The step of constructing the first loss of the image generation model to be trained according to the discrimination result includes: Obtaining an evaluation value of the first predicted image; According to the discrimination result and the evaluation value, a first loss of the image generation model to be trained is constructed.

4. The method according to claim 3, characterized in that The evaluation value includes at least one of the following: image fidelity, semantic rationality; Wherein, the image fidelity includes at least one of the following: global fidelity, local detail fidelity; The semantic rationality includes at least one of the following: semantic consistency and relational logical rationality.

5. The method according to claim 1, characterized in that: The first loss includes a second loss of the generator network and a third loss of the discriminator network, The updating of the model parameters of the image generation model to be trained according to the first loss includes: updating a first network parameter of the generator network according to the second loss; and, According to the third loss, updating the second network parameters of the discriminator network; Among them, the model parameters of the image generation model to be trained include the first network parameters and the second network parameters.

6. The method according to claim 5, characterized in that The updating of the model parameters of the image generation model to be trained according to the first loss includes: In a first training phase of the image generation model to be trained, updating a first network parameter of the generator network according to the second loss, and updating a second network parameter of the discriminator network according to the third loss; In the second training phase of the image generation model to be trained, the second network parameters are fixed, and the first network parameters of the generator network are updated according to the second loss.

7. The method according to claim 6, characterized in that The method further comprises: In a first training phase of the image generation model to be trained, adjusting the update frequency of the discriminator network according to the discrimination result; The update frequency is the frequency of updating the first network parameter.

8. An image generation method, characterized in that: The method comprises: Get the text to be predicted; Generate an image of the text to be predicted by using a generator network in a trained image generation model to obtain a second predicted image of the text to be predicted; The image generation model is trained according to the image generation model training method according to any one of claims 1-7.

9. A training device for an image generation model, characterized in that: The device comprises: An acquisition module, used for acquiring a first text feature vector of a training text; A processing module, used for performing low-rank mapping processing on the first text feature vector to obtain a second text feature vector of the training text; An image generation module, used for performing image generation on the second text feature vector through a generator network in an image generation model to be trained, so as to obtain a first predicted image of the training text; An image discrimination module, configured to discriminate the first predicted image through a discriminator network in the image generation model to be trained, and obtain a discrimination result of the first predicted image; A construction module, used for constructing a first loss of the image generation model to be trained according to the discrimination result; An updating module is used to update the model parameters of the image generation model to be trained according to the first loss.

10. An image generating device, characterized in that: The device comprises: An acquisition module, used to acquire the text to be predicted; An image generation module, used to generate an image of the text to be predicted by using a generator network in a trained image generation model to obtain a second predicted image of the text to be predicted; The image generation model is trained according to the image generation model training method according to any one of claims 1-7.

Images