A cross-modal character clothing design generation method based on multimodal codebook

Abstract

The invention discloses a cross-modal character clothing design generation method based on a multimodal codebook, comprising the following steps: inputting a first target clothing image into an image encoder, encoding to obtain a feature vector map, and constructing a first feature codebook; after discretizing the feature vector map, selecting different discretization codes from the first feature codebook, and combining to obtain a discretized encoding feature map; inputting the encoding feature map into a decoder for decoding, decoding to obtain a restored second target clothing image, performing block loss calculation on the second target clothing image, and updating the image encoder and decoder in steps through reverse gradient propagation; using marked contrast loss to learn the correspondence between visual features and word embedding for training, and obtaining a second feature codebook rich in semantics and image information; training a neural network based on an attention mechanism by inputting a target image to predict a sequence in the image, sampling from the second feature codebook to obtain a sequence value, and using a second decoder to decode the sequence value and output to obtain the final image.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual clothing generation, and more specifically, to a cross-modal character clothing design generation method based on a multimodal codebook. Background Art

[0002] In the field of clothing design, the design process often requires manual drawing by humans. The cross-modal clothing design method combined with the attention mechanism can automatically generate and design a series of clothing by combining multimodal information, which has great application prospects and potential value. Cross-modal fashion design, in which the clothing image changes based on various forms of control signals, such as plain text, sketches, partial images, etc., has the potential to completely change the fashion design process.

[0003] Existing techniques mainly focus on achieving text-guided image synthesis, making the elements of clothing design monolithic, which greatly limits their practicality. There is an increasing demand for frameworks that allow the integration of different controls. However, it is not easy to design a unified framework to process multimodal signals simultaneously due to their inherent representation differences, such as sketches are concise images that can only describe the overall outline of an object, partial images are incomplete images of missing areas, and plain text is different from two corresponding images, which usually describes the main features of a specific object. The representation differences between different modalities make it difficult for most existing cross-modal synthesis works to integrate control signals from different modalities.

[0004] Recently, attention-based neural networks have been shown to have a powerful ability to unify different forms of control signals. These methods usually use vector quantized variational autoencoders in the first stage to compress the input image into a low-resolution discrete feature map and then reconstruct the input image to learn the local feature codebooks of each visual part in the real image. In the second stage, the vector of control signals is input into the attention-based neural network to predict the sequence of synthetic images and sample the sequence values ​​from the feature codebook learned in the first stage. The global expressiveness of the attention-based neural network unifies the control signals of different modalities. This method can handle any type of cross-modal control signals in a single model and generate reasonable results that meet most situations.

[0005] However, the feature codebook construction mechanism used in the above two-stage paradigm inevitably leads to a problem that affects the generalization of the model. Since the feature codebook is learned and constructed by reconstructing images from the training set, the feature codebook can only contain visual information and represent the local features of each visual part of the training image. During the test process, if there is no visual component corresponding to the input control signal in the training set, the model will not be able to predict the precise feature encoding and thus will not be able to generate images that conform to the control signal. For example, if the training set only contains white-collar T-shirts, the trained model will not be able to generate blue-collar T-shirts because there is no encoding corresponding to this image information in the learned feature codebook. The main reason for this problem is that only the local visual features of the image are encoded in the feature codebook, while the corresponding textual information is ignored. This information provides more discriminative features, which leads to low-level representations and rough semantics. Summary of the invention

[0006] In order to solve the problem that the feature coding books in the above prior art only encode the local visual features of the image and ignore the corresponding text information, the present invention provides a cross-modal character clothing design generation method based on a multimodal coding book, which uses text information with finer-grained semantics to enhance the expressive power of the feature coding book and improves the precision of clothing generation combining multimodal information.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0008] A method for generating cross-modal character clothing design based on a multimodal codebook, the method comprising the following steps:

[0009] S1: Input the first target clothing image into the image encoder, encode it to obtain a feature vector map, and construct a first feature codebook; after discretizing the feature vector map, select different discretization codes from the first feature codebook, and combine them to obtain a discretized encoding feature map;

[0010] S2: Input the encoded feature map into the decoder for decoding, decode to obtain the restored second target clothing image, perform block loss calculation on the second target clothing image, and update the image encoder and decoder step by step through reverse gradient propagation;

[0011] S3: Use the marked contrast loss to learn the correspondence between visual features and word embeddings for training, and obtain the second feature codebook rich in semantic and image information;

[0012] S4: By inputting the target image, a neural network based on the attention mechanism is trained to predict the sequence in the target image, the sequence value is sampled from the second feature code book, and the sequence value is decoded by the second decoder to output the final image.

[0013] Preferably, step S1 is specifically as follows:

[0014] S101: input a first target clothing image, use an image encoder composed of a multi-layer convolutional network, and use the image encoder to encode the first target clothing image to obtain a feature vector image;

[0015] S102: discretize the obtained feature vector map, divide the feature vector map into blocks, calculate the distance value between each feature vector block and all the codes in the first feature code book, select the closest distance value as the code of the block; select the discretized code value in the first feature code book to construct a code map.

[0016] Further, step S2 is specifically as follows:

[0017] S201: construct a decoder based on a multi-layer convolutional network, input the encoded feature map into the decoder for decoding, and obtain a restored second target clothing image;

[0018] S202: extracting features from the first target clothing image and the restored second target clothing image through a VGG16 network respectively, and calculating a visual loss between the first target clothing image and the restored second target clothing image;

[0019] S203: Construct a discriminator based on a multi-layer convolutional network, divide the restored second target clothing image into blocks, input them into the discriminator to judge whether they are true or false, and combine the output values ​​to obtain the adversarial generation network loss, perform reverse gradient propagation on the loss, and update the parameters of the image encoder, decoder and discriminator step by step.

[0020] Furthermore, step S3 is specifically as follows:

[0021] S301: In the training phase, given an image-text pair Image I k The image encoder composed of a multi-layer convolutional network is encoded into visual features TextT k Encoded as text features by the attention-based neural network

[0022] S302: Calculate the similarity between the image and the text, and thereby calculate the image-text tag similarity loss of a training batch and its text-image counterparts, and calculate a total loss function;

[0023] S303: Perform gradient back propagation according to the obtained total loss function, update the feature codebook, and obtain a second feature codebook.

[0024] Furthermore, in step S302, the similarity between the image and the text is calculated:

[0025]

[0026] in, I p is the pth target clothing image input, T q is the qth paragraph of text input, is the transposed value of the feature value of the image sequence, is the feature value of the text sequence, and n1 represents the length of the image sequence;

[0027] Formula (3) is p and T q Similarity calculation of ;

[0028]

[0029] in, n2 represents the length of the text sequence;

[0030] Formula (4) is T q and I p Similarity calculation of ;

[0031]

[0032]

[0033] In the formula, I p is the pth target clothing image input, T q is the qth paragraph of text input;

[0034] Equations (5) and (6) represent the image-text label similarity loss of a training batch and its text-image counterparts;

[0035] The total loss function is calculated as:

[0036]

[0037] Furthermore, step S4 is specifically as follows:

[0038] S401: converting one or more of the text, outline or partial image of the input target image into a text tag sequence and / or an image tag sequence, wherein the text tag sequence is directly obtained by an encoder based on the attention mechanism; for the image sequence, learning a feature codebook for each type of target image and quantizing each target image into a discrete image sequence;

[0039] S402: Combine different text tag sequences and image tag sequences together, and use a special tag [SEP] to indicate the separation between the modes; combine the sequences to form a cross-modal sequence, and use it as a conditional input of a neural network based on an attention mechanism during training. Train the neural network based on the attention mechanism to maximize the log-likelihood of the predicted sequences based on various target images, i.e., minimize the training loss, sample the sequence value from the second feature codebook, and finally output the generated image of the clothing design.

[0040] Furthermore, the network loss function used to train the attention-based neural network is expressed as follows:

[0041]

[0042] p(s|c)=∏ i p(s i ∣s <i ,c) (9)

[0043] Among them, s represents the predicted sequence, c represents the input conditional sequence, and p is the probability value of the control input sequence being similar to the input sequence.

[0044] A cross-modal clothing design generation system based on attention mechanism includes an image encoding module, a feature encoding module, a decoding module, a feature codebook construction module and a sequence prediction module; wherein,

[0045] The image encoding module is used to obtain a first target clothing image and encode it to obtain a feature vector image;

[0046] The feature coding module is used to construct a feature coding book, discretize the feature vector graph, select different discretization codes from the feature coding book, and combine them to obtain a discretized coding feature graph;

[0047] The decoding module is used to decode the encoded feature map to obtain the input target clothing image, and perform block loss calculation on the obtained target clothing image;

[0048] The feature codebook construction module is used to learn the correspondence between visual features and word embeddings for training using marked contrast loss to obtain a second feature codebook rich in semantic and image information;

[0049] The sequence prediction module is used to train a neural network based on an attention mechanism by inputting a target image to predict a sequence in an image, sample a sequence value from a second feature codebook, and decode the sequence value with a second decoder to output a final image.

[0050] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0051] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0052] The beneficial effects of the present invention are as follows:

[0053] The present invention improves the construction process of the feature coding map and utilizes an expressive feature coding map to merge fine-grained semantic text information, so that the feature coding map can be combined with text information to achieve finer-grained image generation control. It can solve the problem that the model cannot predict the accurate feature coding due to the lack of visual components corresponding to the input control signal in the data set, and thus cannot generate an image that meets the control signal. It has high efficiency and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The figure is a flow chart of an embodiment of the present invention.

[0055] Figure 2 An embodiment of the present invention provides a design generation effect diagram under different control signal combination inputs (such as text, text plus sketch, text plus partial image).

[0056] Figure 3 A schematic diagram of a questionnaire used in user evaluation provided by the cross-modal clothing design generation system based on the attention mechanism of the present invention.

[0057] Figure 4 An embodiment and other methods provided by the present invention combine multimodal information (such as text, text plus sketch, text plus partial image) to design and generate a comparison chart of corresponding clothing effects.

[0058] Figure 5 This is a design generation effect diagram of the impact of missing modules on the results of an ablation experiment in an embodiment of the present invention.

[0059] Figure 6 A schematic diagram of data sampling of part of the data set used in model training provided by the cross-modal clothing design generation system based on the attention mechanism of the present invention.

[0060] Figure 7 A structural schematic diagram is provided for the system described in the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1

[0063] like Figure 1 As shown, a method for generating cross-modal character clothing design based on a multimodal codebook, the method comprising the following steps:

[0064] S1: Input the first target clothing image into the image encoder, encode it to obtain a feature vector map, and construct a first feature code book; after discretizing the feature vector map, select different discretization codes from the first feature code book, and combine them to obtain a discretized encoding feature map.

[0065] In a specific embodiment, step S1 is as follows:

[0066] S101: Inputting a first target clothing image An image encoder composed of a multi-layer convolutional network is used to encode the image to obtain a feature vector map of the first target clothing image.

[0067] S102: discretize the obtained feature vector graph and divide the feature vector graph into blocks:

[0068]

[0069] in, is the encoding value in the feature encoding map, is the divided feature vector block.

[0070] For each spatial position vector By calculating the distance between each feature vector and all codes in the first feature code book, the nearest distance value is selected as the code of the block; the discretized code value is selected in the first feature code book to construct a code map.

[0071] S2: Input the encoded feature map into the decoder for decoding, decode to obtain the restored second target clothing image, perform block loss calculation on the second target clothing image, and update the image encoder and decoder step by step through reverse gradient propagation.

[0072] In a specific embodiment, step S2 is as follows:

[0073] S201: construct a decoder based on a multi-layer convolutional network, input the encoded feature map into the decoder for decoding, and obtain a restored second target clothing image.

[0074] The formula expression of the decoder is as follows:

[0075]

[0076] Among them, I is the first target clothing image input, E is the image encoder, q is the process of discretizing to obtain the encoding image, G is the decoder, is the restored second target clothing image.

[0077] S202: extracting features from the first target clothing image and the restored second target clothing image through the VGG16 network respectively, and calculating the visual loss between the first target clothing image and the restored second target clothing image.

[0078] S203: Construct a discriminator based on a multi-layer convolutional network, divide the restored second target clothing image into blocks, input them into the discriminator to judge whether they are true or false, and combine the output values ​​to obtain the adversarial generation network loss, perform reverse gradient propagation on the loss, and update the parameters of the image encoder, decoder and discriminator step by step.

[0079] S3: Use the marked contrast loss to learn the correspondence between visual features and word embeddings for training, and obtain the second feature codebook rich in semantic and image information.

[0080] In a specific embodiment, step S3 is as follows:

[0081] S301: In the training phase, given an image-text pair Image I k The image encoder composed of a multi-layer convolutional network is encoded into visual features TextT k Encoded as text features by the attention-based neural network

[0082] S302: Calculate the similarity between the image and the text, and thereby calculate the image-text tag similarity loss of a training batch and its text-image counterparts, and calculate a total loss function.

[0083] In a specific embodiment, step S302 calculates the similarity between the image and the text:

[0084]

[0085] in, I p is the pth target clothing image input, T q is the qth paragraph of text input, is the transposed value of the feature value of the image sequence, is the feature value of the text sequence, and n1 represents the length of the image sequence;

[0086] Formula (3) is p and Tq Similarity calculation of ;

[0087]

[0088] in, n2 represents the length of the text sequence;

[0089] Formula (4) is T q and I p Similarity calculation of ;

[0090]

[0091]

[0092] In the formula, I p is the pth target clothing image input, T q is the qth paragraph of text input;

[0093] Equations (5) and (6) represent the image-text label similarity loss of a training batch and its text-image counterparts;

[0094] The total loss function is calculated as:

[0095]

[0096] S303: Perform gradient back propagation according to the obtained total loss function, update the feature codebook, and obtain a second feature codebook.

[0097] S4: By inputting the target image, a neural network based on the attention mechanism is trained to predict the sequence in the image, the sequence value is sampled from the second feature code book, and the sequence value is decoded by the second decoder to output the final image.

[0098] In a specific embodiment, step S4 is as follows:

[0099] S401: Figure 2 As shown, one or more of the text, contour or partial image of the input target image is converted into a text tag sequence and an image tag sequence, wherein the text tag sequence is directly obtained by an encoder based on an attention mechanism; for the image sequence, a feature encoding book is learned for each type of target image, and each target image is quantized into a discrete image sequence; wherein the partial image is just a portion of the image cut out from the target image, and the contour is a sketch of the target image.

[0100] S402: combining different text tag sequences and image tag sequences together, and using a special tag [SEP] to indicate the separation between the modes; combining the sequences to form a cross-modal sequence, and using the combined sequences as conditional inputs of a neural network based on an attention mechanism during training, training the neural network based on the attention mechanism to maximize the log-likelihood values ​​of the predicted sequences based on various target images, i.e., minimize the training loss, sample the sequence values ​​from the second feature codebook, and finally output the generated image for clothing design.

[0101] In a specific embodiment, the network loss function for training an attention-based neural network is expressed as follows:

[0102]

[0103] p(s|c)=∏ i p(s i ∣s <i ,c) (9)

[0104] Where s represents the predicted sequence; c represents the input conditional sequence, such as the sequence obtained after encoding text, sketches, or partial images; and p is the probability value that the input sequence is similar to the input sequence.

[0105] In the embodiment of the present invention, the multimodal clothing design dataset used is the CM-Fashion dataset, which covers most of the conventional clothing categories, including pants, dresses, jackets, etc. In addition, the dataset contains information about gender and target. Each type of clothing is divided into men's clothing, women's clothing and children's clothing. The images are accompanied by detailed text descriptions and sketches. These sketches are binary images obtained by the Canny edge detection algorithm. The partial images required for input are obtained by random cropping.

[0106] The following will illustrate the cross-modal clothing design generation effect of the present invention with reference to the accompanying drawings:

[0107] The following will qualitatively and quantitatively analyze the cross-modal clothing design generation effect of the present invention. Among them, a data set is used, under which the method of the embodiment of the present invention is quantitatively compared with two existing cross-modal clothing design generation methods: TediGAN and VQGAN. LPIPS, FID, PSNR and HE are used to evaluate the cross-modal clothing design generation effect. The data evaluation index results show that the feature coding map improvement method proposed in the embodiment of the present invention is better than other methods, proving that the method of the embodiment of the present invention can combine finer-grained text information to synthesize more sophisticated cross-modal clothing. The embodiment of the present invention also conducted a user evaluation study such as Figure 3As shown, the specific operation for the task of generating images from text is to show a text and the corresponding image generated from the text to the platform staff on a certain platform, and let the staff choose a more realistic and detailed picture from the cross-modal clothing design generated pictures generated by three different generation methods. 77% of users believe that the results of this method are more realistic. In order to compare with existing methods more fairly, the embodiment of the present invention further tests this method on the Fashiongen clothing design generation dataset. The results show that the method of the embodiment of the present invention still achieves the best in various indicators, such as Figure 4 shown.

[0108] In addition, the embodiment of the present invention also conducts an ablation experiment to verify the effectiveness of the method for updating the feature codebook. In order to verify the effectiveness of the text perception module, we train our model without the text perception module. We compare the comprehensive results of the two models on three cross-modal synthesis tasks, namely, text-oriented synthesis, text sketch-oriented synthesis, and text part image-oriented synthesis. Figure 5 As shown, the feature encoding of our updated method without the context-aware module cannot generate fine-grained details that match the text description, while our complete model is able to generate corresponding details (e.g., texture details, pockets, buttons, etc.).

[0109] Furthermore, we also quantitatively analyze the impact of codebook size on methods without and with text-aware modules. When the codebook size is fixed, FRIES can obtain lower FID and LPIPS values, proving the necessity of text-aware modules for real image synthesis. Comparing the models trained with different codebook sizes, we also observe that larger codebooks tend to lead to lower FID and LPIPS scores. This creates a trade-off between codebook expressiveness and computational cost. For our method, setting the codebook size to 4096 is a better choice.

[0110] The present embodiment provides a cross-modal character clothing design generation method based on a multimodal codebook, which solves the problem that the feature codebook only encodes the local visual features of the image and ignores the corresponding text information. The expressive power of the feature codebook is enhanced by using text information with finer-grained semantics, and the precision of clothing generation combining multimodal information is improved.

[0111] Example 2

[0112] A cross-modal clothing design generation system based on attention mechanism includes an image encoding module, a feature encoding module, a decoding module, a feature codebook construction module and a sequence prediction module; wherein,

[0113] The image encoding module is used to obtain a first target clothing image and encode it to obtain a feature vector image;

[0114] The feature coding module is used to construct a feature coding book, discretize the feature vector graph, select different discretization codes from the feature coding book, and combine them to obtain a discretized coding feature graph;

[0115] The decoding module is used to decode the encoded feature map to obtain the input target clothing image, and perform block loss calculation on the obtained target clothing image;

[0116] The feature codebook construction module is used to learn the correspondence between visual features and word embeddings for training using marked contrast loss to obtain a second feature codebook rich in semantic and image information;

[0117] The sequence prediction module is used to train a neural network based on an attention mechanism by inputting a target image to predict a sequence in an image, sample a sequence value from a second feature codebook, and decode the sequence value with a second decoder to output a final image.

[0118] The present invention realizes multiple operations on character clothing images by constructing a feature codebook and training a neural network based on the attention mechanism. It is a novel and universal cross-modal clothing design framework that can perform clothing operations based on multimodal control signals and utilize an expressive feature codebook to merge fine-grained semantic text information, with high efficiency and flexibility.

[0119] Example 3

[0120] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following method steps are implemented:

[0121] S1: Input the first target clothing image into the image encoder, encode it to obtain a feature vector map, and construct a first feature codebook; after discretizing the feature vector map, select different discretization codes from the first feature codebook, and combine them to obtain a discretized encoding feature map;

[0122] S2: Input the encoded feature map into the decoder for decoding, decode to obtain the restored second target clothing image, perform block loss calculation on the second target clothing image, and update the image encoder and decoder step by step through reverse gradient propagation;

[0123] S3: Use the marked contrast loss to learn the correspondence between visual features and word embeddings for training, and obtain the second feature codebook rich in semantic and image information;

[0124] S4: By inputting the target image, a neural network based on the attention mechanism is trained to predict the sequence in the image, the sequence value is sampled from the second feature code book, and the sequence value is decoded by the second decoder to output the final image.

[0125] Example 4

[0126] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0127] S1: Input the first target clothing image into the image encoder, encode it to obtain a feature vector map, and construct a first feature codebook; after discretizing the feature vector map, select different discretization codes from the first feature codebook, and combine them to obtain a discretized encoding feature map;

[0128] S2: Input the encoded feature map into the decoder for decoding, decode to obtain the restored second target clothing image, perform block loss calculation on the second target clothing image, and update the image encoder and decoder step by step through reverse gradient propagation;

[0129] S3: Use the marked contrast loss to learn the correspondence between visual features and word embeddings for training, and obtain the second feature codebook rich in semantic and image information;

[0130] S4: By inputting the target image, a neural network based on the attention mechanism is trained to predict the sequence in the image, the sequence value is sampled from the second feature code book, and the sequence value is decoded by the second decoder to output the final image.

[0131] The various embodiments of the present invention can be combined arbitrarily to achieve different technical effects.

[0132] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk), etc.

[0133] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal character clothing design generation method based on a multimodal codebook, characterized by: The method comprises the following steps: S1: Input the first target clothing image into the image encoder, encode it to obtain a feature vector map, and construct a first feature codebook; after discretizing the feature vector map, select different discretization codes from the first feature codebook, and combine them to obtain a discretized encoding feature map; S2: Input the encoded feature map into the decoder for decoding, decode to obtain the restored second target clothing image, perform block loss calculation on the second target clothing image, and update the image encoder and decoder step by step through reverse gradient propagation; the details are as follows: S201: construct a decoder based on a multi-layer convolutional network, input the encoded feature map into the decoder for decoding, and obtain a restored second target clothing image; S202: extracting features from the first target clothing image and the restored second target clothing image through a VGG16 network respectively, and calculating a visual loss between the first target clothing image and the restored second target clothing image; S203: construct a discriminator based on a multi-layer convolutional network, divide the restored second target clothing image into blocks, input them into the discriminator to judge whether they are true or false, and combine the output values ​​to obtain the adversarial generation network loss, perform reverse gradient propagation on the loss, and update the parameters of the image encoder, decoder and discriminator step by step; S3: Use the marked contrast loss to learn the correspondence between visual features and word embeddings for training, and obtain the second feature codebook rich in semantic and image information; the details are as follows: S301: In the training phase, given an image-text pair ,image The image encoder composed of a multi-layer convolutional network is encoded into visual features ,text Encoded as text features by the attention-based neural network ; S302: Calculate the similarity between the image and the text, and thereby calculate the image-text tag similarity loss of a training batch and its text-image counterparts, and calculate the total loss function; Calculate the similarity between the image and the text: （3） in, , is the pth target clothing image input, is the qth paragraph of text input, is the transposed value of the feature value of the image sequence, is the text sequence feature value, n 1 represents the length of the image sequence; Formula (3) is and Similarity calculation of ; （4） in, , n 2 represents the length of the text sequence; Formula (4) is and Similarity calculation of ; （5） （6） In the formula, is the pth target clothing image input, is the qth paragraph of text input; Equations (5) and (6) represent the image-text label similarity loss of a training batch and its text-image counterparts; The total loss function is calculated as: （7） S303: Perform gradient back propagation according to the obtained total loss function, update the feature codebook, and obtain a second feature codebook; S4: By inputting the target image, a neural network based on the attention mechanism is trained to predict the sequence in the target image, and the sequence value is sampled from the second feature codebook, and the sequence value is decoded by the second decoder to output the final image, as follows: S401: converting one or more of the text, outline or partial image of the input target image into a text tag sequence and / or an image tag sequence, wherein the text tag sequence is directly obtained by an encoder based on the attention mechanism; for the image sequence, learning a feature codebook for each type of target image and quantizing each target image into a discrete image sequence; S402: combining different text tag sequences and image tag sequences together, and using a special tag [SEP] to indicate the separation between the modes; the combined sequence forms a cross-modal sequence, and serves as a conditional input of a neural network based on an attention mechanism during training, training the neural network based on the attention mechanism to maximize the log-likelihood value of the predicted sequence based on various target images, i.e., minimize the training loss, sample the sequence value from the second feature codebook, and finally output the generated image of the clothing design; Among them, the network loss function used to train the attention-based neural network is expressed as follows: （8） （9） in, represents the predicted sequence, represents the conditional sequence of input, is the probability value that the input sequence is similar to the input sequence.

2. The method for generating cross-modal character clothing design based on a multimodal codebook according to claim 1, characterized in that: Step S1 is as follows: S101: input a first target clothing image, use an image encoder composed of a multi-layer convolutional network, and use the image encoder to encode the first target clothing image to obtain a feature vector image; S102: discretize the obtained feature vector map, divide the feature vector map into blocks, calculate the distance value between each feature vector block and all the codes in the first feature code book, select the closest distance value as the code of the block; select the discretized code value in the first feature code book to construct a code map.

3. A cross-modal clothing design generation system based on attention mechanism, characterized by: It includes an image encoding module, a feature encoding module, a decoding module, a feature encoding book construction module and a sequence prediction module; wherein, The image encoding module is used to obtain a first target clothing image and encode it to obtain a feature vector image; The feature coding module is used to construct a feature coding book, discretize the feature vector graph, select different discretization codes from the feature coding book, and combine them to obtain a discretized coding feature graph; The decoding module is used to decode the encoded feature map to obtain the input target clothing image, and perform block loss calculation on the obtained target clothing image; specifically as follows: S201: construct a decoder based on a multi-layer convolutional network, input the encoded feature map into the decoder for decoding, and obtain a restored second target clothing image; S202: extracting features from the first target clothing image and the restored second target clothing image through a VGG16 network respectively, and calculating a visual loss between the first target clothing image and the restored second target clothing image; S203: construct a discriminator based on a multi-layer convolutional network, divide the restored second target clothing image into blocks, input them into the discriminator to judge whether they are true or false, and combine the output values ​​to obtain the adversarial generation network loss, perform reverse gradient propagation on the loss, and update the parameters of the image encoder, decoder and discriminator step by step; The feature codebook construction module is used to learn the correspondence between visual features and word embeddings by using marked contrast loss for training, so as to obtain a second feature codebook rich in semantic and image information; specifically as follows: S301: In the training phase, given an image-text pair ,image The image encoder composed of a multi-layer convolutional network is encoded into visual features ,text Encoded as text features by the attention-based neural network ; S302: Calculate the similarity between the image and the text, and thereby calculate the image-text tag similarity loss of a training batch and its text-image counterparts, and calculate the total loss function; Calculate the similarity between the image and the text: （3） in, , is the pth target clothing image input, is the qth paragraph of text input, is the transposed value of the feature value of the image sequence, is the text sequence feature value, n 1 represents the length of the image sequence; Formula (3) is and Similarity calculation of ; （4） in, , n 2 represents the length of the text sequence; Formula (4) is and Similarity calculation of ; （5） （6） In the formula, is the pth target clothing image input, is the qth paragraph of text input; Equations (5) and (6) represent the image-text label similarity loss of a training batch and its text-image counterparts; The total loss function is calculated as: （7） S303: Perform gradient back propagation according to the obtained total loss function, update the feature codebook, and obtain a second feature codebook; The sequence prediction module is used to train a neural network based on an attention mechanism to predict the sequence in the image by inputting text, image contour or partial image, sampling from the second feature codebook to obtain the sequence value, and decoding the sequence value with the second decoder to obtain the final image, as follows: S401: converting one or more of the text, outline or partial image of the input target image into a text tag sequence and / or an image tag sequence, wherein the text tag sequence is directly obtained by an encoder based on the attention mechanism; for the image sequence, learning a feature codebook for each type of target image and quantizing each target image into a discrete image sequence; S402: combining different text tag sequences and image tag sequences together, and using a special tag [SEP] to indicate the separation between the modes; the combined sequence forms a cross-modal sequence, and serves as a conditional input of a neural network based on an attention mechanism during training, training the neural network based on the attention mechanism to maximize the log-likelihood value of the predicted sequence based on various target images, i.e., minimize the training loss, sample the sequence value from the second feature codebook, and finally output the generated image of the clothing design; Among them, the network loss function used to train the attention-based neural network is expressed as follows: （8） （9） in, represents the predicted sequence, represents the conditional sequence of input, is the probability value that the input sequence is similar to the input sequence.

4. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images