Method, device and equipment for constructing compliance text graph training set and storage medium

Abstract

The invention discloses a compliance text graph training set construction method and device, equipment and a storage medium, and the method can extract compliance corpus data from an open source training dataset according to compliance evaluation information after the open source training dataset is obtained, and then carries out the reverse deduction of cue words based on compliance pictures in the compliance corpus data. And inputting the cue word into the target text graph model to generate an expanded image through the target text graph model, and constructing a compliance text graph training set according to the cue word, the expanded image and the compliance corpus data. According to the method, the compliance training set adaptive to the specified business scene can be constructed based on the open source training data set, and the construction period of the data set is shortened. And the data volume of the training data set is increased by expanding the image, so that the text graph model can output a more accurate image generation result after the constructed training data set is applied.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for constructing a compliance text graph training set. Background Art

[0002] Text-to-Image Models are artificial intelligence models that generate corresponding images based on text descriptions, such as Generative Adversarial Networks (GANs), Diffusion Models, or Autoregressive Models. Text-to-Image Models can learn the mapping relationship between text and images based on deep learning technology. By constructing a training set and using the training set to train the text-to-image model, the text-to-image model can correctly understand the input text description and output an image that conforms to the text description.

[0003] Since the Wensheng graph model is trained based on a single language corpus, its support for other languages ​​is limited. In addition, when constructing the training corpus for the Wensheng graph model, local values ​​and legal restrictions are mainly considered, resulting in the generated results being biased towards local requirements, which are quite different from the requirements of other regions. Therefore, directly using the trained Wensheng graph model often outputs non-compliant image generation results.

[0004] In order to obtain compliant image generation results, it is necessary to rebuild the training corpus that meets the requirements of the use area for performing text-based graph model training. For example, in business scenarios such as life insurance, health insurance, and property insurance, most of the images used in promotional copy created to maintain customer relationships and expand new customers are local elements and styles. At the same time, local laws, regulations, and ethical standards must also be considered. Therefore, it is necessary to build a corresponding training corpus training model to better meet the needs of such business scenarios. However, the text-based graph model training process requires a large amount of training data sets. If the training corpus is built from scratch, not only will the construction cycle be long, but it will also easily lead to a decrease in the generalization ability of the model and inaccurate output image generation results. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, device, equipment and storage medium for constructing a compliant text image training set to solve the problems of long training data set construction cycle and inaccurate image generation results.

[0006] According to one aspect of the present application, a method for constructing a compliance text graph training set is provided, the method comprising:

[0007] Acquire an open source training data set, wherein the open source training data set includes training corpus data, and the training corpus data includes image corpus and text annotation corpus associated with the image corpus;

[0008] Extracting compliance corpus data from the open source training data set according to the compliance evaluation information, wherein the compliance corpus data includes compliance images and compliance text annotations; the compliance evaluation information includes at least one of copyright information, sensitive word information, and image feature information;

[0009] Inferring prompt words based on the compliance pictures in the compliance corpus data;

[0010] Inputting the prompt word into a target text graph model to generate an extended image through the target text graph model, wherein the target text graph model is a trained text graph model suitable for a target field;

[0011] A compliant text image training set is constructed according to the prompt words, the expanded images and the compliant corpus data.

[0012] According to another aspect of the present application, a device for constructing a compliance text graph training set is provided, the device comprising:

[0013] A data acquisition module, used to acquire an open source training data set, wherein the open source training data set includes training corpus data, and the training corpus data includes image corpus and text annotation corpus associated with the image corpus;

[0014] A compliance detection module, configured to extract compliance corpus data from the open source training data set according to compliance evaluation information, wherein the compliance corpus data includes compliance images and compliance text annotations; the compliance evaluation information includes at least one of copyright information, sensitive word information, and image feature information;

[0015] A prompt word generation module, used for inferring prompt words based on the compliant images in the compliant corpus data;

[0016] An image expansion module, used for inputting the prompt word into a target text graph model to generate an expanded image through the target text graph model, wherein the target text graph model is a trained text graph model suitable for a target field;

[0017] The data set construction module is used to construct a compliant text image training set according to the prompt words, the expanded images and the compliant corpus data.

[0018] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned method for constructing a compliant text graph training set when executing the program.

[0019] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for constructing a compliant text graph training set is implemented.

[0020] By means of the above technical scheme, a method, device, equipment and storage medium for constructing a compliant text graph training set provided by the embodiment of the present application can first extract compliant corpus data from the open source training data set according to compliance evaluation information after obtaining an open source training data set, then infer prompt words based on the compliant images in the compliant corpus data, and input the prompt words into the target text graph model to generate an extended image through the target text graph model, and then construct a compliant text graph training set according to the prompt words, extended images and compliant corpus data. The method can construct a compliant training set adapted to a specified business scenario based on an open source training data set, shortening the construction cycle of the data set. And by expanding the image, the data volume of the training data set is increased, so that after applying the constructed training data set, the text graph model can output a more accurate image generation result. The method can also be quickly applied to other text graph models and open source corpora for image understanding to improve the scalability of the training set. In addition, by fine-tuning the paradigm, it can be easily migrated to different business fields to achieve the purpose of enabling business.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 A schematic diagram of a Wensheng diagram process provided in an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of the training process of the Wensheng graph model provided in an embodiment of the present application is shown;

[0025] Figure 3 A flow chart of a method for constructing a compliance culture graph training set provided in an embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram of the process of constructing a compliance training data set provided in an embodiment of the present application is shown;

[0027] Figure 5 A flowchart of performing compliance detection based on copyright information provided by an embodiment of the present application is shown;

[0028] Figure 6A schematic diagram of a compliance detection process based on a first sensitive word set provided in an embodiment of the present application is shown;

[0029] Figure 7 A schematic diagram of a compliance detection process based on a second sensitive word set provided in an embodiment of the present application is shown;

[0030] Figure 8 A flow chart of performing compliance detection based on image features provided in an embodiment of the present application is shown;

[0031] Fig. 9 A schematic diagram of a process for generating an extended image provided by an embodiment of the present application is shown;

[0032] Fig.10 A schematic diagram of a process for training a text graph model based on a compliant text graph training set provided in an embodiment of the present application is shown;

[0033] Fig.11 A schematic diagram of the structure of a device for constructing a compliance text graph training set provided in an embodiment of the present application is shown;

[0034] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0036] In the embodiment of the present application, text image refers to an artificial intelligence technology that generates corresponding images based on text descriptions. Figure 1 As shown in the figure, the text-generated graph technology can extract key information from text description data based on the deep learning model, and construct the image pixel content according to the key information to obtain the image generation result.

[0037] In some embodiments, the deep learning model can be a combination of one or more of Generative Adversarial Networks (GAN), Variational Autoencoder (VAE), and a diffusion model based on a Transformer model.

[0038] Among them, the generative adversarial network includes a generator and a discriminator. The generator (G) is a deep neural network responsible for generating image data. The generator can receive random noise as input and try to generate image data that looks real. The discriminator (D) is a deep neural network used to distinguish between fake data generated by the generator and real data. The discriminator outputs a probability value, indicating the probability that the input data is real. The generator and the discriminator can be trained in an adversarial manner. The generator generates realistic data samples as much as possible, making it impossible for the discriminator to distinguish between real data and generated fake data. The discriminator distinguishes between real data and generated fake data as accurately as possible.

[0039] The variational autoencoder learns the potential representation of data through the structure of encoder and decoder, and can generate new data samples. The variational autoencoder can include an encoder (Encoder) and a decoder (Decoder). The encoder can map the input data to the distribution parameters of a latent variable. The decoder maps the variables sampled from the latent variable distribution back to the data space to generate new data samples. VAE can introduce the concept of probability distribution between the encoder and decoder, making the generated samples more diverse and continuous. The encoder outputs the distribution of latent variables, which can be sampled from this distribution and sent to the decoder to generate other samples similar to the input samples.

[0040] The diffusion model destroys the training data by continuously adding Gaussian noise to the original data, and then denoises it through a deep network to learn to recover the data. After the training is completed, data can be generated by inputting random noise and passing it to the denoising process. The diffusion model based on Transformers is a deep learning model built on the attention mechanism. Combined with the Transformer architecture, the encoder and decoder are used to process text and image data to generate high-quality images. The Transformer model is able to consider all positions in the input sequence at the same time, allowing the model to assign different attention weights according to different parts of the input sequence, so as to better capture semantic relationships. The Transformer architecture can follow the overall architecture of the encoder and decoder, using stacked self-attention mechanisms and fully connected layers for the encoder and decoder respectively. By introducing the multi-head attention mechanism, the model is allowed to process different information subspaces in parallel and capture different types of relationships.

[0041] A document graph model can be constructed based on any of the above deep learning models. The document graph model needs to be trained before application. The training process of the document graph model may include multiple stages, such as Figure 2As shown, data preparation, model architecture design, training generators and discriminators, adversarial training, loss function optimization, iterative training, evaluation and fine-tuning, etc.

[0042] The data preparation stage requires the collection and preprocessing of training data sets, namely training data sets. Training data sets can include training corpus data collected from various information sources, including image corpus and text annotation corpus. Image corpus refers to a large number of images used for text-based graph model training. Text annotation corpus refers to the annotation of image corpus to add explanatory language information.

[0043] During the model training process, the image corpus can provide visual information to help the model learn how to convert text descriptions into visual content, so that the model can understand and generate images that match the input text description. For example, for the generative adversarial network model, the real images in the image corpus can be used to train the discriminator and improve the discriminator's recognition ability. During the model training phase, the image corpus is paired with the text annotation corpus and input into the text graph model, so that the text graph model can learn the semantic association between text and images. The image corpus can also be combined with text data, and the model can perform multimodal learning, so that the model can better understand and generate images.

[0044] Text annotated corpus can provide conditional input in text-to-graph models. For example, in conditional generative adversarial networks (Conditional GAN), text annotated corpus is used as input to the model, allowing the model to generate images that match the text description. Text annotated corpus can guide image synthesis. In guided generative models (Guided GAN), text annotated corpus can participate in the image synthesis process through auxiliary classifiers, thereby improving the quality and relevance of generated images. Text annotated corpus can also enhance semantic understanding, allowing the model to have a deeper understanding of the semantic content of the text. Text annotated corpus can also help the model capture the nuances in the text, thereby generating more diverse images.

[0045] In order to improve the generalization ability of the text graph model, training datasets can be obtained from different data sources when performing model training. For example, training datasets can come from public datasets, professional datasets, social media platforms, web crawlers, user-generated content, partner and third-party databases, synthetic data, and crowdsourcing platforms.

[0046] After obtaining the training data set, you can design the model architecture. For example, for a generative adversarial network model, you can design the network architecture of the generator and the discriminator, so that the generator is responsible for generating images based on the input text description, while the discriminator is responsible for judging whether the generated image is real.

[0047] Then, the model is trained based on the designed model architecture. That is, the generator and the discriminator are trained. When training the generator, the generator can receive text descriptions (text annotation corpus) as input, convert it into a representation of the latent space through the encoder, and then generate images through the decoder. When training the discriminator, the discriminator can receive real images (picture corpus) and images generated by the generator as input, and learn to distinguish between the picture corpus and the generated images. By performing adversarial training, the generator and the discriminator compete with each other during the training process. The generator continuously learns how to generate more realistic images, while the discriminator continuously learns how to better distinguish between real and fake images.

[0048] During adversarial training, loss functions can be used for parameter optimization. For example, in a generative adversarial network (GAN), the loss function can include the difference between the generated image of the generator and the real image, and the classification error rate of the discriminator. The goal of training is to minimize these losses. In a variational autoencoder (VAE), the loss function includes reconstruction error and KL divergence. The reconstruction error measures the difference between the generated image and the real image, while the KL divergence measures the difference between the potential distribution of the encoder output and the prior distribution. The goal of training is to minimize the reconstruction error and make the potential distribution close to the prior distribution.

[0049] Through multiple iterations of training, the parameters of the generator and discriminator are continuously optimized until the quality of the generated images reaches a satisfactory level. During the training process, the performance of the model, such as the quality and diversity of the images, can be regularly evaluated, and the model can be fine-tuned as needed.

[0050] Through the above model training process, a text graph model with a preset accuracy can be obtained. Since the training data set can affect the loss calculation and parameter optimization during the training of the text graph model, it affects the iterative training process and ultimately affects the model accuracy of the text graph model. Therefore, in order to obtain a text graph model with a set accuracy and generalization ability, it is necessary to improve the quality of the training corpus data in the training data set.

[0051] In some embodiments, the training data set needs to contain clear, diverse, high-quality images, and the images are closely related to the text descriptions. The scale of the training data set should also be large enough to cover more scenes and objects, thereby improving the generalization ability of the model. For example, the Stable Diffusion model uses large-scale data sets such as Laion-2B-en during training to improve the quality of sample data through data screening. The training data set should also contain diverse images and text descriptions to ensure that the model can learn rich visual and language features. In addition, the image corpus in the training data set should have specific image feature requirements such as image size. For example, the Stable Diffusion model can be pre-trained at a resolution of 256×256 during training, and then fine-tuned and optimized at a resolution of 512×512 to improve the resolution of the generated images.

[0052] Therefore, in the process of executing the training of the culture graph model, the output capacity of the culture graph model can be indirectly controlled by constraining the training data set. For example, for a specific region, due to factors such as laws, morals, and values, the images generated by the culture graph model need to comply with specific regulations. Therefore, by adjusting the training corpus data in the training data set, the model can be controlled not to generate image content that does not comply with the regulations, thereby meeting the requirements of a specific region.

[0053] However, due to the huge amount of data in the training dataset, the workload of adjusting the training dataset is huge, which will extend the construction cycle of the training dataset. In addition, in some cases, the amount of data in the training dataset is reduced after adjustment. Using the adjusted training dataset for training will cause the model to fall into local convergence, reduce the generalization ability of the Wensheng graph model, and even affect the accuracy of the image generation results.

[0054] Based on this, in order to improve the quality of the training data set, shorten the construction cycle of the training data set, and improve the generalization ability of the model and the accuracy of the image generation results, a method for constructing a compliant text image training set is provided in this embodiment. Figure 3 , Figure 4 As shown, the method includes:

[0055] S101. Obtain an open source training dataset.

[0056] Among them, open source training data sets are data sets that are composed of public data and can be obtained by any organization and used for training. Similar to typical training data sets, the open source training data sets can also include training corpus data. In addition, the training corpus data includes image corpus and text annotation corpus associated with the image corpus.

[0057] For example, open source training datasets can be laion-400m, LAION-5B, MULAN and other training datasets. These open source training datasets can contain a certain number of "image-text" pairs. Among them, laion-400m is a large-scale "image-text" pair dataset, which can contain about 400 million pairs of images and corresponding text descriptions. It can provide training resources for training and improving multimodal language-vision models.

[0058] LAION-5B is also a large-scale training dataset, containing 585 million CLIP-filtered image-text pairs and 232 million English texts. The LAION-5B training dataset can support the replication and fine-tuning of basic models such as CLIP, GLIDE, and StableDiffusion, and provides nearest neighbor indexing, improved web interfaces for dataset exploration and subset generation, and detection scores for watermarking, NSFW, and content detection.

[0059] MULAN is a multi-layer annotation dataset for controllable text image generation, containing more than 44K multi-layer RGBA decompositions and more than 100K instance images. The MULAN dataset provides high-quality image resources, as well as instance decomposition and occlusion information.

[0060] In order to obtain the open source training data set, the data processing device (such as a client, server, etc.) that performs model training can connect to the network and access the data acquisition interface of the open source training data set through the network connection. After receiving the control instruction for performing model training, the data processing device can request data from the data acquisition interface to obtain the open source training data set.

[0061] S102: extracting compliance corpus data from the open source training data set according to the compliance evaluation information.

[0062] After obtaining the open source training data set, the data processing device can perform compliance detection on the open source training data set, that is, screen out the training data set that meets the preset regulations to perform subsequent model training. To this end, after obtaining the training data set, the data processing device can read the information used for compliance detection from the training data set, that is, the compliance evaluation information, and the compliance evaluation information includes at least one of copyright information, sensitive word information, and image feature information. The data processing device then extracts compliant corpus data from the open source training data set based on the compliance evaluation information, wherein the compliant corpus data includes compliant images and compliant text annotations.

[0063] In some embodiments, in order to extract compliant corpus data, the data processing device can filter out the training corpus data that does not comply with copyright regulations based on the copyright information in the corpus file. Figure 5 As shown, when the data processing device performs the step of extracting compliant corpus data from the open source training data set according to the compliance evaluation information, it reads the copyright information of the training corpus data and obtains a preset authorized copyright identification list. The copyright information can be obtained by reading the license information of the training corpus file in the training data set or identifying the copyright logo pattern in the image corpus. The authorized copyright identification list is an authorized list pre-constructed according to the copyright authorization situation, that is, in the authorized copyright identification list, the types of training corpus data that can be trained by the current data processing device are recorded.

[0064] By comparing the copyright information with the list of authorized copyright identifications, it can be determined whether the training corpus data is copyright-compliant training corpus data or copyright-incompliant training corpus data. If the copyright information is in the list of authorized copyright identifications, the training corpus data corresponding to the copyright information is retained. If the copyright information is not in the list of authorized copyright identifications, the training corpus data corresponding to the copyright information is deleted.

[0065] It should be noted that the copyright information recorded in the authorized copyright identification list is the training corpus data that can be used by the current data processing device to perform training. If the copyright information of the training corpus data is not in the authorized copyright identification list, or the training corpus data does not contain copyright information, the training corpus data will be determined as copyright-incompliant training corpus data.

[0066] For example, for the acquired training data set, the training corpus data contained therein may have the following situations: if there is no license information, the corpus data is determined to be copyright non-compliant data, so the corpus data can be deleted. If the training corpus data contains license information, you can select the corpus that does not have non-commercial (None Commercial, NC) copyright requirements, such as CC-BY, CC-BY-SA and other commercially available training corpus data.

[0067] By comparing copyright information with the list of authorized copyright identifiers, the training corpus data that does not comply with copyright regulations in the training data set can be screened out, so that the remaining training corpus data complies with copyright regulations. For example, more than 95% of the training corpus data that does not comply with copyright regulations can be screened out through copyright information.

[0068] In some embodiments, the data processing device can also perform compliance detection based on text annotations, that is, filter out training corpus data with non-compliant text annotations. Figure 6 As shown, when the data processing device extracts the compliant corpus data from the open source training data set according to the compliance evaluation information, it can also read the text annotation corpus of the training corpus data and extract corpus keywords from the text annotation corpus.

[0069] According to the different forms of the text annotation corpus in the training corpus data, the data processing device can extract the corpus keywords in different ways. When the text annotation corpus in the training corpus data is a word vector, the data processing device can directly read the word vector to obtain the keyword set contained in the word vector.

[0070] When the text annotation corpus in the training corpus data is a natural language text, the data processing device can segment the natural language text according to the preset word library to obtain a segmentation set. The segmentation set is then filtered according to the word part of speech, symbol recognition, stop word word library, etc. to obtain keywords with content meanings in the natural language text.

[0071] In order to better obtain keyword content, the data processing device can also call a natural language processing model to extract keywords from the text annotation corpus. For example, the natural language processing model can extract keywords based on frequency (word frequency or inverse document frequency), statistics (mutual information, point mutual information), co-occurrence relationship graph, machine learning, topic model, part-of-speech tagging and named entity recognition rules, and one or more combinations of attention mechanisms.

[0072] After extracting corpus keywords from the text annotation corpus, the data processing device can obtain a preset first sensitive word set, and compare the extracted corpus keywords with the first sensitive word set to determine whether the corpus keywords contained in the text annotation corpus appear in the first sensitive word set. Among them, the first sensitive word set is used to record non-compliant keywords. If the corpus keyword is not in the first sensitive word set, the corpus keyword is marked as a compliant text annotation, and the training corpus data corresponding to the compliant text annotation is retained. If the corpus keyword is in the first sensitive word set, the corpus keyword is marked as a non-compliant text annotation, and the training corpus data corresponding to the non-compliant text annotation is deleted.

[0073] For example, since the training data set of the Laion series comes from the Internet, its text annotations are mainly published by different users in different regions. The correlation between text annotations and images varies greatly, and there is no restriction on compliance, resulting in some content in the text annotations that does not comply with the regulations of specific regions. Therefore, after obtaining the training data set of the Laion series, the text annotation corpus of each training corpus data in the training data set can be traversed, and the text detection model can be used to detect whether the text annotation corpus contains words in the preset first sensitive word set, thereby detecting the compliance of the text annotation corpus.

[0074] By traversing the keywords of the text annotation corpus, when the text annotation corpus includes keyword A, and the preset first sensitive word set also includes keyword A, it can be determined that the current text annotation corpus contains non-compliant keywords, so the training corpus data containing keyword A can be deleted. Similarly, when the text annotation corpus includes keyword B and keyword C, and both keyword B and keyword C are not in the first sensitive word set, it can be determined that the current text annotation corpus does not contain non-compliant keywords, so the training corpus data containing keyword B and keyword C can be retained.

[0075] In the process of deleting or retaining the training corpus data, since the image corpus and the text annotation corpus in the training corpus data are a pair of data with an associated relationship, when deleting or retaining the training corpus data, the image corpus associated with the text annotation corpus needs to be deleted or retained. To this end, in some embodiments, the data processing device can call a pre-trained neural network, and use the pre-trained neural network to calculate the association between the image corpus and the text annotation corpus in the training data set, and then compare the association with the association threshold to determine the image corpus that has an associated relationship with the text annotation corpus.

[0076] Wherein, pre-trained neural network is a model pre-trained on a large amount of data, and these models can be fine-tuned (fine-tuning) in subsequent tasks to adapt to specific application scenarios. In the present embodiment, the pre-trained neural network can be used to calculate the degree of association between the image corpus and the text annotation corpus. For example, the pre-trained neural network can be a CLIP (Contrastive Language-Image Pre-training) model, and the CLIP model can be pre-trained by a large amount of paired data of images and texts to learn the alignment relationship between images and texts. The CLIP model may include a text encoder (Text Encoder) and an image encoder (Image Encoder). The text encoder is used to convert text into a low-dimensional vector representation, and the image encoder is used to convert an image into a similar vector representation. In the prediction stage, the CLIP model generates a prediction by calculating the cosine similarity between the text and image vectors.

[0077] The CLIP model (open-clip-g) is used to calculate the correlation (i.e., CLIP-score) between the image corpus and the text annotation corpus, thereby determining the image corpus and text annotation corpus that have a correlation relationship. When the text annotation corpus is determined to be a non-compliant text annotation, the non-compliant text annotation corpus can be deleted, and the image corpus with a correlation greater than a threshold with the non-compliant text annotation corpus can be deleted at the same time. Similarly, when the text annotation corpus is determined to be a compliant text annotation, the image corpus with a CLIP-score less than the threshold can be deleted.

[0078] It should be noted that the CLIP-score calculated by the pre-trained neural network is also used to determine whether the image corpus and the text annotation corpus in the training corpus data constitute an image-text pair. Therefore, after calculating the CLIP-score, the training corpus data can be screened according to the CLIP-score, thereby deleting the corpus with a CLIP-score less than a threshold, such as the training corpus data with a CLIP-score less than 0.35.

[0079] In some embodiments, the compliance of the training corpus data can also be checked based on the text length corresponding to the text annotation corpus. That is, the data processing device can read the training corpus data from the training data set and detect the text length of the training corpus data. The text length can be represented by the number of words or characters contained in the text annotation corpus in the training corpus data.

[0080] After the text length is detected, the text length is compared with the preset length interval. If the text length is within the preset length interval, it can be determined that the current text annotation corpus is a compliant text annotation, so the training corpus data corresponding to the compliant text annotation can be retained. If the text length is not within the preset length interval, that is, the text annotation corpus is too long or too short, it can be determined that the current text annotation corpus is a non-compliant text annotation, so the training corpus data corresponding to the non-compliant text annotation can be deleted. For example, the data processing device can delete corpora with text lengths that are too long or too short, and only retain corpora with text lengths of 5-75.

[0081] In some embodiments, for text annotation corpora of different language types, the text annotation corpora can also be translated, and sensitive word detection can be performed on the translated text annotation corpora to extract text annotation corpora that better meets the local sensitive word requirements. Figure 7 As shown, when the data processing device extracts the compliance corpus data from the open source training data set according to the compliance evaluation information, it can read the text annotation corpus of the training corpus data. Then translate the text annotation corpus into text annotation information of the target language type. Among them, the target language type is the language type corresponding to the application scenario of the text graph model. The original language type of the training corpus data is the language type corresponding to the scenario provided by the training data set. For example, the target language type is Chinese, and the original language type is English. After reading the text annotation corpus of the training corpus data, the English-to-Chinese language conversion module can be used to translate the English-type text annotation corpus into Chinese-type text annotation information.

[0082] Then, the target language keywords are extracted from the text annotation information, and a preset second sensitive word set is obtained. The difference between the second sensitive word set and the first sensitive word set is that the second sensitive word set is a sensitive word set constructed based on the target language type, while the first sensitive word set does not consider the specific language type. The first sensitive word set can be constructed based on the original language type or based on a mixed language type.

[0083] After obtaining the second sensitive word set, the data processing device can determine whether the target language keyword appears in the second sensitive word set according to the sensitive word detection method in the above embodiment. If the target language keyword is not in the second sensitive word set, the target language keyword is marked as a compliant text annotation, and the training corpus data corresponding to the compliant text annotation is retained. If the target language keyword is in the second sensitive word set, the target language keyword is marked as a non-compliant text annotation, and the training corpus data corresponding to the non-compliant text annotation is deleted.

[0084] For example, since the training corpus data in the Laion-400m training data set are mostly published by foreign users, their original language type is mostly English. In order to adapt to the Chinese scenario, the data processing device can translate the original English text annotations into Chinese text annotations based on the English-to-Chinese translation model, and perform Chinese sensitive word detection to filter out non-compliant text annotation corpus.

[0085] In some embodiments, the data processing device can also perform compliance detection based on the specific image content corresponding to the image corpus, that is, Figure 8 As shown, extracting compliant corpus data from the open source training data set according to the compliance evaluation information also includes: obtaining picture corpus in the training corpus data, and inputting the picture corpus into a picture detection model to obtain image feature information output by the picture detection model. Wherein, the picture detection model is used to identify image features in the picture corpus. The picture detection model can detect whether the picture corpus contains non-compliant image content based on image detection algorithms such as pixel color values, pixel distribution, graphic features, and object recognition in the image. For example, the image detection model can be an image detection model based on deep learning. By training a deep learning model based on structures such as Ernie-Vil, illegal content is identified through a custom prompt.

[0086] After obtaining the image feature information output by the image detection model, it is possible to determine whether the image feature information contains non-compliant image features. If the image feature information includes non-compliant image features, the image corpus is marked as a non-compliant image, and the training corpus data corresponding to the non-compliant image is deleted. If the image feature information does not include non-compliant image features, the image corpus is marked as a compliant image, and the training corpus data corresponding to the compliant image is retained.

[0087] For example, since the image corpus in the Laion-400m training dataset is published by users, its content has a large degree of freedom and covers a wide range of aspects, resulting in limited compliance of some image corpora in specific application fields. Therefore, an image detection model can be used to detect non-compliant image corpora and delete the corresponding image corpora to make the image corpus meet the compliance requirements of the application field.

[0088] In some embodiments, in the process of compliance detection of the picture corpus, the image parameters of the picture corpus can also be detected, so as to screen out the picture corpus whose image parameters meet the preset requirements. That is, when the data processing device extracts the compliant corpus data from the open source training data set according to the compliance evaluation information, it can read the image parameters of the picture corpus and obtain the preset image parameter interval. If the image parameter is within the image parameter interval, it is determined that the current picture corpus is a compliant picture, so the training corpus data corresponding to the picture corpus can be retained. If the image parameter is not within the image parameter interval, it is determined that the current picture corpus is a non-compliant picture, so the training corpus data corresponding to the picture corpus can be deleted.

[0089] For example, the image parameters may include the image width and image height of the image corpus. During the compliance detection of the image corpus, the image width and image height of the image corpus may be read. If the image width or image height of the image corpus is within the interval [128, 2048], it can be determined that the image size of the image corpus is compliant, so the image corpus with image sizes of 128 to 2048 on either side is retained. Similarly, if the image width and image height of the image corpus are not within the interval [128, 2048], it means that the image size of the image corpus is too large or too small, so the image corpus with image sizes that are too large or too small can be deleted.

[0090] It should be noted that in the above embodiments, the compliance of the training corpus data is tested in terms of compliance evaluation information in terms of copyright, sensitive words, text length, image content, image parameters, etc., so as to extract compliant corpus data from the open source training data set according to the compliance evaluation information. Obviously, the compliance evaluation information during compliance testing is not limited to the aspects listed above, but may also include compliance evaluation information in other aspects, such as corpus source, image style, etc. The compliance evaluation information in other aspects that can be associated by those skilled in the art without creative labor belongs to the protection scope of this application.

[0091] It should be understood that the compliance evaluation information of the above aspects can be used for compliance testing separately or in combination with each other. For example, in the process of compliance testing, the data processing equipment can first perform compliance testing based on copyright information, then perform sensitive word detection based on keywords, and then perform non-compliant image feature detection based on image content, and obtain compliant corpus data that is compliant in copyright, text annotation, and image content through a variety of compliance evaluation information. After multiple compliance testing operations, the training data set can select more than 300,000 compliant corpus data from the 400 million corpus of the laion-400m open source corpus, recorded as laion-400m-nsfw, for subsequent training.

[0092] S103: Infer prompt words based on the compliant images in the compliant corpus data.

[0093] Since compliance detection will filter out a large amount of training corpus data, resulting in a reduction in the number of samples in the training data set, the generalization ability of subsequent model training will be affected. Therefore, after performing compliance detection to obtain compliant corpus data, the data processing device can also infer the prompt words based on the compliant images in the compliant corpus data. In order to infer the prompt words, the data processing device can call a tool or model that generates text based on images, and input the compliant images in the compliant corpus data into the called tool or model, identify and interpret the image content of the compliant images through the called tool or model, and infer the prompt words.

[0094] For example, the corresponding prompts of the pictures in laion-400m-nsfw can be inferred based on the interrogate clip. The interrogate clip is a tool based on OpenAI's CLIP model and Salesforce's BLIP (Bootstrapping Language-Image Pretraining) model, which can analyze images and generate descriptive text or labels to explain the image content through natural language descriptions. The BLIP model can focus on generating basic initial titles for images. The CLIP model can receive the basic description of BLIP and enhance it, adding more details to the description by comparing it with various predefined phrases. This makes the interrogate clip useful for application scenarios that understand or replicate the style and content of an image, helping to identify key elements and provide prompts for creating similar images.

[0095] CLIP Interrogator can first generate a basic title, that is, use the BLIP model to create an initial title for the image to provide a basic description of the image. Then perform content enhancement, that is, add various categories to the basic title, such as specific phrases of objects and styles. Then use the CLIP model to match the image with the enhanced phrases to ensure that the final text is more detailed and closely aligned with the image content.

[0096] S104: input the prompt word into a target text-based graph model to generate an extended image through the target text-based graph model.

[0097] After the prompt word is deduced, the data processing device inputs the prompt word into the target text graph model to generate an extended image through the target text graph model. The target text graph model is a text graph model that is suitable for the target field and has been trained. For example, the target text graph model can be one or more combinations of SDXL, playground2.5, Hunyuan, Taiyi and other models.

[0098] When generating an extended image, the target text graph model can be called according to specific domain requirements, that is, when the data processing device inputs the prompt word into the target text graph model to generate an extended image through the target text graph model, the target domain information can be obtained, and the target text graph model can be extracted according to the target domain information. Then the prompt word is input into the target text graph model to obtain the extended image output by the target text graph model. It can be seen that after determining the target text graph model of the domain based on the target domain information, an extended image that conforms to the target domain information can be generated through the target text graph model based on the prompt word.

[0099] For example, after inferring the prompts corresponding to the pictures in laion-400m-nsfw based on the interrogate clip, you can select models from SDXL, playground2.5, Hunyuan, Taiyi and other models according to the required data field. If you need to generate a cartoon dataset, select the cartoon generation model.

[0100] In some embodiments, Fig. 9 As shown, in order to generate an extended image, the data processing device can input the prompt word into the target text graph model one by one according to a preset number of generation times to obtain multiple candidate images output by the target text graph model. Then calculate the similarity between the candidate image and the compliant image corresponding to the prompt word. If the similarity is within the preset similarity interval, mark the candidate image as the extended image and retain the extended image. If the similarity is not within the preset similarity interval, mark the candidate image as a non-extended image and delete the non-extended image. Through similarity calculation, it can be ensured that the generated extended image has a certain similarity and a certain diversity with the original image corpus.

[0101] For example, 5-15 images can be generated by models such as SDXL. Then, images with a similarity of 0.4-0.6 to the corresponding image clip of laion-400m-nsfw can be selected as extended images, thereby ensuring a certain similarity with the original image and a certain diversity, and obtaining the extended image dataset laion-400m-nsfw-exp1. Similarly, a similar process can be used to further expand the extended image set laion-400m-nsfw-exp1 to obtain laion-400m-nsfw-exp2, laion-400m-nsfw-exp3, ..., laion-400m-nsfw-expN. It can be seen that based on this method, a laion-400m-nsfw-exp dataset of a certain scale can be obtained, and it can be in any field while meeting relevant compliance requirements.

[0102] S105: construct a compliant text image training set according to the prompt words, the expanded images and the compliant corpus data.

[0103] After obtaining the augmented image through the target text graph model, the obtained augmented image can be combined with the prompt word to obtain a new image-text pair. Then, it is combined with the compliant corpus data obtained after screening from the open source training data set to construct a compliant text graph training set. Since the compliant corpus data obtained through screening meets the compliance requirements in terms of copyright, text annotation, and image features, and no non-compliant data is used in the generation process of the augmented image, the augmented image also meets the compliance requirements. Therefore, the training corpus data contained in the compliant text graph training set constructed based on the prompt word, augmented image, and compliant corpus data can meet the compliance requirements.

[0104] By applying the technical solution of this embodiment, the above embodiment can automatically build a compliance training set adapted to a specific business scenario based on open source corpus, which can shorten the dataset construction cycle. In addition, it can be seamlessly and quickly applied to other open source corpora of text, image, and even image understanding to build a compliance training set, which has strong scalability. This fine-tuning paradigm can be easily migrated to different business fields to achieve the purpose of enabling business.

[0105] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a model training method based on a compliance document training set, such as Fig.10 As shown, the method includes:

[0106] S201. Obtain a training set of compliance documents and graphs.

[0107] The compliant text graph training set includes an extended image-text pair and compliant corpus data. The extended image-text pair includes an extended image and a prompt word associated with the extended image. The prompt word is a prompt word obtained by inferring the compliant image in the compliant corpus data. The extended image is an image generated by inputting the prompt word into the target text graph model.

[0108] S202: Input the compliant text graph training set into the trained text graph model to obtain a training generated image output by the trained text graph model.

[0109] The trained Wensheng graph model is one of an unconverged generative adversarial network model, a variational autoencoder model, and a diffusion model based on Transformer. After obtaining the compliant Wensheng graph training set, by inputting the compliant Wensheng graph training set into the trained Wensheng graph model, the trained Wensheng graph model can perform image generation calculations, thereby outputting an image, that is, obtaining a training generated image.

[0110] S203: Calculate training loss according to the training generated image and the sample image.

[0111] The sample image is an augmented image or a compliant image in the compliant text image training set. After the trained text image model outputs the training generated image, the training loss can be calculated based on the training loss algorithm set by the model.

[0112] S204: If the training loss is greater than the loss threshold, modify the model parameters of the trained text graph model according to the training loss.

[0113] After calculating the training loss, the data processing device can obtain the preset training loss threshold, and then compare the training loss with the loss threshold. If the training loss is greater than the loss threshold, it means that the accuracy of the result output by the current trained model does not meet the preset requirements, that is, the trained model has not converged. Therefore, the training loss can be used for back propagation to modify the model parameters of the trained model. Then, iterative training is performed based on the modified model parameters, and the training loss is recalculated until the training loss is less than or equal to the loss threshold.

[0114] S205: If the training loss is less than or equal to the loss threshold, output the model parameters of the trained text graph model.

[0115] After multiple iterations of training, when the training loss is less than or equal to the loss threshold, it means that the output accuracy of the current trained Wensheng graph model has reached the preset accuracy requirement, that is, the trained Wensheng graph model has converged. Therefore, the model parameters of the trained Wensheng graph model can be output to obtain the Wensheng graph model.

[0116] Since the training data sets used in the training process of the document graph model are all compliant training corpus data, the document graph model obtained based on the above training method can output compliant document graph results, thereby improving the problem of inaccurate output image generation results.

[0117] Further, as a specific implementation of the method for constructing a compliance text graph training set described in the above embodiment, the present application embodiment provides a device for constructing a compliance text graph training set, such as Fig.11 As shown, the device comprises:

[0118] A data acquisition module, used to acquire an open source training data set, wherein the open source training data set includes training corpus data, and the training corpus data includes image corpus and text annotation corpus associated with the image corpus;

[0119] A compliance detection module, configured to extract compliance corpus data from the open source training data set according to compliance evaluation information, wherein the compliance corpus data includes compliance images and compliance text annotations; the compliance evaluation information includes at least one of copyright information, sensitive word information, and image feature information;

[0120] A prompt word generation module, used for inferring prompt words based on the compliant images in the compliant corpus data;

[0121] An image expansion module, used for inputting the prompt word into a target text graph model to generate an expanded image through the target text graph model, wherein the target text graph model is a trained text graph model suitable for a target field;

[0122] The data set construction module is used to construct a compliant text image training set according to the prompt words, the expanded images and the compliant corpus data.

[0123] It should be noted that for other corresponding descriptions of the functional units involved in the device for constructing a compliant text graph training set provided in an embodiment of the present application, reference can be made to the corresponding descriptions in the method for constructing a compliant text graph training set provided in the above embodiment, and will not be repeated here.

[0124] By applying the technical solution of this embodiment, the device for constructing a compliant text graph training set can first extract compliant corpus data from the open source training data set according to the compliance evaluation information after obtaining the open source training data set, then infer the prompt words based on the compliant images in the compliant corpus data, and input the prompt words into the target text graph model to generate an extended image through the target text graph model, and then construct a compliant text graph training set based on the prompt words, extended images and compliant corpus data. The device can construct a compliant training set adapted to a specified business scenario based on the open source training data set, shortening the construction cycle of the data set. And by expanding the image, the data volume of the training data set is increased, so that after applying the constructed training data set, the text graph model can output a more accurate image generation result. The device can also be quickly applied to other text graph models and open source corpora for image understanding to improve the scalability of the training set.

[0125] like Fig.12 As shown, the embodiment of the present application also provides a computer device, which can be specifically a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0126] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0127] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a compliance text graph training set, characterized in that: The method comprises: Acquire an open source training data set, wherein the open source training data set includes training corpus data, and the training corpus data includes image corpus and text annotation corpus associated with the image corpus; Extracting compliance corpus data from the open source training data set according to the compliance evaluation information, wherein the compliance corpus data includes compliance images and compliance text annotations; the compliance evaluation information includes at least one of copyright information, sensitive word information, and image feature information; Inferring prompt words based on the compliance pictures in the compliance corpus data; Inputting the prompt word into a target text graph model to generate an extended image through the target text graph model, wherein the target text graph model is a trained text graph model suitable for a target field; A compliant text image training set is constructed according to the prompt words, the expanded images and the compliant corpus data.

2. The method according to claim 1, characterized in that Extracting compliance corpus data from the open source training data set according to the compliance evaluation information includes: Reading copyright information of the training corpus data; Get the preset list of authorized copyright logos; If the copyright information is in the list of authorized copyright identifiers, retaining the training corpus data corresponding to the copyright information; If the copyright information is not in the list of authorized copyright identifiers, the training corpus data corresponding to the copyright information is deleted.

3. The method according to claim 1, characterized in that: Extracting compliance corpus data from the open source training data set according to the compliance evaluation information includes: Read the text annotation corpus of the training corpus data; Extracting corpus keywords from the text annotation corpus; Get the preset first sensitive word set; If the corpus keyword is not in the first sensitive word set, marking the corpus keyword as a compliant text annotation, and retaining the training corpus data corresponding to the compliant text annotation; If the corpus keyword is in the first sensitive word set, the corpus keyword is marked as a non-compliant text annotation, and the training corpus data corresponding to the non-compliant text annotation is deleted.

4. The method according to claim 1, characterized in that: Extracting compliance corpus data from the open source training data set according to the compliance evaluation information includes: Read the text annotation corpus of the training corpus data; Translating the text annotation corpus into text annotation information of a target language type; Extracting target language keywords from the text annotation information; Get the preset second sensitive word set; If the target language keyword is not in the second sensitive word set, marking the target language keyword as a compliant text annotation, and retaining the training corpus data corresponding to the compliant text annotation; If the target language keyword is in the second sensitive word set, the target language keyword is marked as a non-compliant text annotation, and the training corpus data corresponding to the non-compliant text annotation is deleted.

5. The method according to claim 1, characterized in that Extracting compliance corpus data from the open source training data set according to the compliance evaluation information includes: Obtaining image corpus from the training corpus data; Inputting the picture corpus into a picture detection model to obtain image feature information output by the picture detection model, wherein the picture detection model is used to identify image features in the picture corpus; If the image feature information includes a non-compliant image feature, marking the image corpus as a non-compliant image, and deleting training corpus data corresponding to the non-compliant image; If the image feature information does not include non-compliant image features, the picture corpus is marked as a compliant picture, and the training corpus data corresponding to the compliant picture is retained.

6. The method according to claim 1, characterized in that Inputting the prompt word into a target text graph model to generate an augmented image through the target text graph model includes: Obtain target domain information; Extracting the target text graph model according to the target domain information; The prompt word is input into the target text graph model to obtain the extended image output by the target text graph model, wherein the extended image is an image generated by the target text graph model based on the prompt word and conforming to the target domain information.

7. The method according to claim 1, characterized in that Inputting the prompt word into a target text graph model to generate an augmented image through the target text graph model includes: Inputting the prompt words into the target text graph model one by one according to a preset number of generation times to obtain a plurality of candidate images output by the target text graph model; Calculating the similarity between the candidate image and the qualified image corresponding to the prompt word; If the similarity is within a preset similarity interval, marking the candidate image as the extended image, and retaining the extended image; If the similarity is not within a preset similarity interval, the candidate image is marked as a non-expanded image, and the non-expanded image is deleted.

8. A device for constructing a compliant text graph training set, characterized in that: The device comprises: A data acquisition module, used to acquire an open source training data set, wherein the open source training data set includes training corpus data, and the training corpus data includes image corpus and text annotation corpus associated with the image corpus; A compliance detection module, configured to extract compliance corpus data from the open source training data set according to compliance evaluation information, wherein the compliance corpus data includes compliance images and compliance text annotations; the compliance evaluation information includes at least one of copyright information, sensitive word information, and image feature information; A prompt word generation module, used for inferring prompt words based on the compliant images in the compliant corpus data; An image expansion module, used for inputting the prompt word into a target text graph model to generate an expanded image through the target text graph model, wherein the target text graph model is a trained text graph model suitable for a target field; The data set construction module is used to construct a compliant text image training set according to the prompt words, the expanded images and the compliant corpus data.

9. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images