Data processing method and system, device, and storage medium

Abstract

The present invention relates to the technical field of computers, and disclosed are a data processing method and system, a device, and a storage medium. The method comprises: acquiring a plurality of sample prompts for model training; analyzing the correspondence between entities in the sample prompts and entity attributes, and extracting sample prompts in which the correspondence reaches a complexity index, so as to obtain a first data set; extracting, from the first data set, sample prompts satisfying a difference index, so as to obtain a second data set; and performing model training on the basis of the second data set.

Description

Data processing method, system, device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed on March 27, 2024, with application number 202410362148.3 and invention name “Data processing method, system, device and storage medium”. The entire contents of this application are incorporated by reference into this application. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a data processing method, system, device, and storage medium. Background Art

[0004] The text-based graph model uses prompt words as a guide to generate images that match the prompt words. Both initial training and fine-tuning of the text-based graph model require a large number of prompt words as training samples. The accuracy and rationality of the training samples significantly impact the accuracy of the text-based graph model.

[0005] At present, there are generally unreasonable problems in the selected training samples, which leads to low accuracy of the trained text-based graph model and distortion of the generated images.

[0006] Therefore, a method to improve the accuracy of the model is urgently needed. Summary of the Invention

[0007] In view of this, embodiments of the present disclosure provide a data processing method, a data processing system, an electronic device, and a computer-readable storage medium.

[0008] In one aspect, the present disclosure provides a data processing method, the method comprising:

[0009] Obtain multiple sample prompt words for model training;

[0010] parsing the correspondence between entities and entity attributes in the sample prompt words, and extracting sample prompt words whose correspondences meet the complexity index to obtain a first data set;

[0011] Extracting sample prompt words that meet the difference index from the first data set to obtain a second data set;

[0012] Model training is performed based on the second data set.

[0013] In one aspect, the present disclosure provides a data processing system, the system comprising:

[0014] A data acquisition module is used to obtain multiple sample prompt words for model training;

[0015] A first data extraction module is configured to analyze the correspondence between entities and entity attributes in the sample prompt words, and extract the sample prompt words whose correspondences meet the complexity index to obtain a first data set;

[0016] A second data extraction module is used to extract sample prompt words that meet the difference index from the first data set to obtain a second data set;

[0017] A training module is used to perform model training based on the second data set.

[0018] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0019] On the other hand, the present disclosure further provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0021] Figure 1 shows a schematic diagram of some cultural graph models generating images;

[0022] FIG2 is a schematic diagram showing a flow chart of a data processing method provided by an embodiment of the present application;

[0023] FIG3 shows a schematic diagram of a parse tree provided by an embodiment of the present application;

[0024] FIG4 shows a schematic diagram of the process of the first stage training provided by an embodiment of the present application;

[0025] FIG5 shows a schematic diagram of module interaction of the first phase training in FIG4 ;

[0026] FIG6 shows a schematic diagram of module interaction in the first phase of training provided by another embodiment of the present application;

[0027] FIG7 shows a schematic diagram of the process of the second stage training provided by an embodiment of the present application;

[0028] FIG8 shows a schematic diagram of module interaction in the second phase of training in FIG7 ;

[0029] FIG9 shows a schematic diagram of module interaction in the second phase of training provided by another embodiment of the present application;

[0030] FIG10 shows a module diagram of a data processing system provided by an embodiment of the present application;

[0031] FIG11 shows a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0033] In the technical solutions of some embodiments of the present application, after obtaining sample prompt words for model training, the sample prompt words are screened according to a complexity index to eliminate sample prompt words with low semantic complexity. Sample prompt words are also screened according to a difference index to reduce redundancy and expand the data distribution range of the sample prompt words. Training the model based on sample prompt words screened by complexity and difference can achieve better training results, thereby improving the accuracy of the trained model.

[0034] Refer to Figure 1 for a schematic diagram of image generation using some text graph models. In Figure 1, the text graph model can be a diffusion model. Prompt words are used to guide the text graph model in generating the desired image. For example, if the text graph model is required to generate an image of three children sitting on a couch, the prompt word could be "Three children on a couch, full shot." For another example, if the text graph model is required to generate a night scene of the moon and a park, the prompt word could be "The moon is very bright in the night sky, casting a gentle light over the abandoned amusement park."

[0035] The principle of the Wensheng graph model to generate images is: using the prompt word as a guide, the initial graph X including random noise is processed in multiple time steps. t Perform noise reduction to obtain an image that matches the prompt word. For example, in Figure 1, X t-1 Represents the initial graph X t The image obtained after the first denoising, X t-2 Indicates that in X t-1 Based on the initial graph X tThe image obtained after the second denoising is performed. The image X0 obtained after multiple denoising can be used as the image matching the prompt word.

[0036] It is understandable that based on different prompt words, the initial graph X is analyzed in multiple time steps. t The denoising methods used for denoising can be different, so that the text graph model can generate different images for different prompt words. The training process of the text graph model is to provide a large number of prompt words as training samples, so that the text graph model can learn the denoising method that matches the prompt words. After the text graph model is trained, the text graph model can use the input prompt words and the learned denoising knowledge to denoise the initial image X according to the denoising method that matches the prompt words. t Perform noise reduction to generate an image that matches the prompt word.

[0037] During the training of a text graph model, the rationality of the choice of prompt words significantly impacts the model's accuracy. However, currently, prompt words for text graph model training are commonly collected randomly. This method can lead to problems such as insufficient complexity, a narrow distribution range, and high redundancy, resulting in low accuracy of the trained text graph model.

[0038] In view of this, the present application provides a data processing method that can solve the above problems and improve the accuracy of the model. The data processing method can be applied to electronic devices. Electronic devices include but are not limited to tablet computers, laptop computers, desktop computers, servers, etc. Referring to Figure 2, a flow chart of a data processing method provided in one embodiment of the present application is shown. In Figure 2, the data processing method includes:

[0039] Step S21: Acquire multiple sample prompt words for model training.

[0040] In this embodiment, the sample prompt words obtained in step S21 may be selected from prompt words input by online users. Online users may be users who use the text graph model to generate images on the Internet. The sample prompt words obtained here may have problems such as insufficient complexity, narrow distribution range, and high redundancy.

[0041] In some other embodiments, the sample prompt words may be obtained based on other channels, such as obtaining multiple sample prompt words from a public prompt word dataset. This application does not limit the method of obtaining the sample prompt words in step S21.

[0042] Step S22 , parsing the correspondence between entities and entity attributes in the sample prompt words, and extracting the sample prompt words whose correspondence reaches the complexity index to obtain a first data set.

[0043] Specifically, entities can be objects that form the basic structure of a sample prompt word. For example, the initiator of an action, the recipient of an action, and the modified object can all serve as entities. Entity attributes are used to characterize the characteristics of an entity. For example, the action initiated by the initiator can serve as an entity attribute of the initiator; the action received by the recipient can serve as an entity attribute of the recipient; and the words used to modify the initiator can serve as an entity attribute of the initiator.

[0044] For ease of understanding, let's take the example prompt "A black cat is chasing a small brown bird." In this prompt, "cat" and "bird" can be considered entities, "black" and "chasing" can be considered entity attributes of "cat," and "small," "brown," and "chasing" can be considered entity attributes of "bird."

[0045] The correspondence between entities and entity attributes can be represented by a parse tree. For example, after parsing the sample prompt word "A black cat is chasing a small brown bird", the resulting parse tree can be shown in Figure 3. In the parse tree, the entity attribute that has a connection relationship with an entity is the entity attribute corresponding to the entity. For example, in Figure 3, with the entity "bird" as the center, there are 3 entity attributes connected to the entity "bird", indicating that there are 3 entity attributes and the entity "bird" have a corresponding relationship; with the entity "cat" as the center, there are 2 entity attributes connected to the entity "cat", indicating that there are 2 entity attributes and the entity "cat" have a corresponding relationship. In this application, the number of entity attributes corresponding to an entity is referred to as the number of corresponding relationships between the entity and the entity attributes. For example, the entity "bird" corresponds to 3 entity attributes, and the number of corresponding relationships between the entity "bird" and the entity attributes is 3.

[0046] Complexity metrics can be used to define the complexity requirements for the correspondences between entities and entity attributes in sample prompt words. For example, these metrics include the total number of correspondences, the number of correspondences between a single entity and entity attributes, and the part of speech of the entity attributes connected to the entity (e.g., verb, noun, adjective, etc.). Complexity metrics can be specified based on actual needs.

[0047] In this embodiment, the complexity index includes the minimum number of corresponding relationships between entities and entity attributes. Based on the complexity index, the sample prompt words whose corresponding relationships meet the complexity index in step S21 may include:

[0048] In response to the presence of an entity satisfying the following conditions in a sample prompt word, the sample prompt word is extracted into the first data set:

[0049] The number of correspondences between entities and entity attributes reaches the minimum number of correspondences.

[0050] For example, assume the minimum number of correspondences is 3. Take the sample prompt "A black cat is chasing a small brown bird" as an example, as shown in Figure 3. In this sample prompt, the number of correspondences between the entity "cat" and the entity attribute is 2, and the number of correspondences between the entity "bird" and the entity attribute is 3. Since the number of correspondences between the entity "bird" and the entity attribute reaches the minimum number of correspondences, the sample prompt "A black cat is chasing a small brown bird" can be extracted into the first dataset.

[0051] For another example, if the minimum number of correspondences is 4, the sample prompt "A black cat is chasing a small brown bird" does not have the minimum number of correspondences between entities and entity attributes, so the sample prompt "A black cat is chasing a small brown bird" cannot be extracted into the first data set.

[0052] After filtering sample prompt words according to the complexity index, we can eliminate sample prompt words with low semantic complexity and retain sample prompt words with semantic complexity that meets the model training requirements. Training the model based on sample prompt words that meet the semantic complexity requirements can greatly improve the model's accuracy.

[0053] Step S23: extracting sample prompt words that meet the difference index from the first data set to obtain a second data set.

[0054] Specifically, the difference index can be used to indicate the required difference between any two sample prompt words in the second data set. For example, the difference between any two sample prompt words in the second data set needs to reach 80%.

[0055] In this embodiment, the difference index can be represented by the similarity between the sample prompt words. For example, requiring the difference between the sample prompt words to reach 80% means that the similarity between the sample prompt words needs to be less than 20%. In view of this, the difference index can include a first similarity threshold (such as 20%) that the similarity between the sample prompt words needs to be lower than. Based on the first similarity threshold, the sample prompt words that meet the difference index are extracted from the first data set, which may include:

[0056] Extract features of each sample prompt word to obtain a feature vector of each sample prompt word;

[0057] Finding a first sample prompt word and a second sample prompt word whose feature vector distance is within a preset distance range, and determining a first similarity between the first sample prompt word and the second sample prompt word;

[0058] In response to the first similarity being higher than a first similarity threshold, deleting the second sample prompt word or the first sample prompt word from the first data set;

[0059] The first data set after deleting the sample prompt words is used as the second data set.

[0060] Specifically, the distance of the feature vectors may include the cosine distance between the feature vectors. For any first sample prompt word A in the first data set, a preset number (for example, 3) of second sample prompt words closest to the first sample prompt word A can be searched in the first data set based on the cosine distance between the sample prompt words. Furthermore, the similarity between the first sample prompt word A and each second sample prompt word can be calculated. If the similarity between the first sample prompt word A and any second sample prompt word B is higher than the first similarity threshold, it means that the first sample prompt word A and the second sample prompt word B do not meet the difference index. In this case, the first sample prompt word A or the second sample prompt word B can be deleted from the first data set. In this way, after performing the above operation on each sample prompt word that has not been deleted in the first data set, in the obtained data set, the difference index between any two sample prompt words is met. Therefore, this obtained data set can be used as the second data set.

[0061] The data in the second dataset can have a wider distribution and lower redundancy in the sample prompt words. By screening the sample prompt words in the first dataset based on the difference index, the problem of narrow data distribution and high redundancy in the sample prompt words can be solved. Training the model based on sample prompt words with a wider data distribution and low redundancy can significantly improve the model's accuracy.

[0062] Furthermore, in some embodiments, before extracting the second data set from the first data set based on the first similarity, the data processing method of the present application may further include:

[0063] In the first data set, searching for multiple groups of first sample prompt words and second sample prompt words whose feature vector distances are within a preset distance range, and calculating the second similarity of each group of first sample prompt words and second sample prompt words respectively;

[0064] In response to an average of second similarities between the plurality of groups of first sample prompt words and second sample prompt words being lower than a second similarity threshold, the first data set is used as the second data set.

[0065] Specifically, if the average value of the second similarities of multiple groups of first sample prompt words and second sample prompt words is lower than the second similarity threshold, it can be said that the distribution range and redundancy of the sample prompt words in the first data set can meet the requirements of model training. In this case, the first data set can be directly used as the second data set, thereby reducing the amount of data processing in the data screening process.

[0066] Step S24: Perform model training based on the second data set.

[0067] In summary, in the technical solutions of some embodiments of the present application, after obtaining sample prompt words for model training, the sample prompt words are screened according to a complexity index to eliminate sample prompt words with low semantic complexity. Furthermore, the sample prompt words are screened according to a difference index to reduce the redundancy of the sample prompt words and expand the data distribution range of the sample prompt words. Training the model based on the sample prompt words screened by complexity and difference can achieve better training results, thereby improving the accuracy of the trained model.

[0068] In some embodiments, after training the text graph model based on the second data set, fine-tuning training can also be performed on the text graph model. Fine-tuning training can include first-stage training and second-stage training. Refer to Figures 4 and 5 in conjunction. Figure 4 is a flow chart of the first-stage training provided by one embodiment of the present application. Figure 5 is a schematic diagram of the module interaction of the first-stage training in Figure 4. In Figure 4, the first-stage training includes the following steps:

[0069] Step S41 : obtaining a first sample image and performing noise reduction processing on the first sample image to obtain a second sample image.

[0070] Specifically, the first sample image may be a standard image that needs to be generated by a Wensheng graph model. After obtaining the first sample image, random noise may be added to the first sample image to obtain a second sample image.

[0071] Step S42 : inputting the second sample image into the Wensheng graph model to convert the second sample image into a third sample image through the Wensheng graph model.

[0072] Specifically, the third sample image is an image obtained by denoising the second sample image based on the denoising knowledge learned by the Wensheng graph model during the initial training process.

[0073] It is understandable that if the Vincent graph model is highly accurate and noise reduction is accurate for the second sample image, the third sample image should be consistent with the first sample image, or the difference between the third sample image and the first sample image should be within an acceptable threshold. However, if the Vincent graph model is not highly accurate and noise reduction is inaccurate for the second sample image, the third sample image may differ significantly from the first sample image. Therefore, the difference between the third sample image and the first sample image can be used to evaluate the accuracy of the Vincent graph model. Simply put, the smaller the difference between the third sample image and the first sample image, the higher the accuracy of the Vincent graph model; and the larger the difference between the third sample image and the first sample image, the lower the accuracy of the Vincent graph model.

[0074] Step S43 : comparing the first sample image and the third sample image to obtain quality difference values ​​between the first sample image and the third sample image in various quality dimensions representing image quality.

[0075] Specifically, the quality dimension is used to determine the image quality of the third sample image. The image quality of the third sample image represents the degree of similarity between the third sample image and the first sample image. A higher degree of similarity between the third sample image and the first sample image indicates better image quality, while a lower degree of similarity between the third sample image and the first sample image indicates worse image quality. When comparing the third sample image and the first sample image for similarity, the third sample image and the first sample image can be compared along multiple dimensions. The multiple dimensions used herein can be referred to as quality dimensions.

[0076] In this embodiment, the quality dimension may include an instance dimension and an image style dimension. Comparing the similarity between the third sample image and the first sample image from the instance dimension may involve comparing the similarity between the instance segmentation results of the third sample image and the first sample image. Comparing the similarity between the third sample image and the first sample image from the image style dimension may involve comparing the similarity between the image styles of the third sample image and the first sample image. It is understood that the selection can be based on actual needs. In this embodiment, dividing the quality dimension into the instance dimension and the image style dimension does not constitute a limitation to this application.

[0077] The difference between the third sample image and the first sample image in each quality dimension can be represented by a quality difference value. Specifically, if the difference between the third sample image and the first sample image in a quality dimension is large, then the quality difference value between the third sample image and the first sample image in that quality dimension can be large. Conversely, if the difference between the third sample image and the first sample image in a quality dimension is small, then the quality difference value between the third sample image and the first sample image in that quality dimension can be correspondingly small.

[0078] In this embodiment, the quality difference value includes an instance difference value between the third sample image and the first sample image in the instance dimension, and a style difference value between the third sample image and the first sample image in the image style dimension.

[0079] The following takes the instance dimension and the image style dimension as an example to illustrate how to compare the third sample image with the first sample image to obtain the quality difference value between the third sample image and the first sample image in each quality dimension.

[0080] Specifically, in some embodiments, after obtaining the first sample image in step S41, the first sample image may be instance-labeled so that the first sample image has instance segmentation annotations. Based on this, from the instance dimension, the above-mentioned comparison of the first sample image and the third sample image may include:

[0081] Performing instance segmentation on the third sample image to obtain an instance segmentation result of the third sample image;

[0082] The difference value between the instance segmentation annotation and the instance segmentation result is used as the instance difference value between the third sample image and the first sample image in the instance dimension.

[0083] Referring to FIG5 , performing instance segmentation on the third sample image and determining the difference between the instance segmentation result of the third sample image and the instance segmentation annotation of the first sample image can be achieved by using a trained instance segmentation model. Performing instance segmentation on an image is a technique that should be known to those skilled in the art and will not be described in detail here. In this application:

[0084] Use m I (x′0) represents the instance segmentation result of the third sample image, where m I represents an instance segmentation network for performing instance segmentation on the third sample image, and x′0 represents the third sample image;

[0085] Use GT(x0) to represent the instance segmentation annotation of the first sample image, where x0 represents the first sample image;

[0086] Use L instance (m I (x′0),GT(x0)) represents the quality difference value (i.e., instance difference value) between the third sample image and the first sample image in the instance dimension. If the instance segmentation result of the third sample image is significantly different from the instance segmentation annotation of the first sample image, L instance (m I The value of (x′0),GT(x0)) can be large; if the difference between the instance segmentation result of the third sample image and the instance segmentation annotation of the first sample image is small, Linstance (m I The value of (x′0),GT(x0)) can be smaller.

[0087] Furthermore, in some embodiments, from the perspective of image style, the comparison of the first sample image and the third sample image may include:

[0088] extracting a first style value representing a style of the first sample image from the first sample image, and extracting a second style value representing a style of the third sample image from the third sample image;

[0089] The difference between the first style value and the second style value is used as the style difference between the first sample image and the third sample image in the image style dimension.

[0090] Specifically, image styles can be divided according to actual needs, for example, image styles can be divided into retro style and modern style. Different image styles can have different style values. With reference to FIG3 , extracting the style values ​​of the first sample image and the third sample image, and calculating the style difference value between the first sample image and the third sample image can be achieved by a trained image style extraction model. Image style extraction is a technology that should be known to those skilled in the art and will not be described in detail here. In this application:

[0091] Gram(V(x0)) is used to represent the first style value of the first sample image style, where x0 represents the first sample image, V represents the image style extraction network for extracting image style features, and Gram represents the use of the Gram matrix to calculate the extracted image style features. The calculated result can be used as the second style value;

[0092] Gram(V(x′0)) is used to represent the second style value of the third sample image style, wherein x′0 represents the third sample image. V and Gram can refer to the relevant description of Gram(V(x0)), which is not repeated here.

[0093] ||Gram(V(x′0))-Gram(V(x0))||2 is used to represent the quality difference value (i.e., style difference value) between the third sample image and the first sample image in the image style dimension. If the image style of the third sample image is significantly different from that of the first sample image, the value of ||Gram(V(x′0))-Gram(V(x0))||2 can be larger; if the image style of the third sample image is slightly different from that of the first sample image, the value of ||Gram(V(x′0))-Gram(V(x0))||2 can be smaller.

[0094] Step S44 : in response to the quality difference value indicating that the first sample image and the third sample image do not match, performing a first stage training on the Vincent graph model according to the quality difference value.

[0095] Specifically, in this embodiment, whether the first sample image and the third sample image match can be determined separately according to the quality difference value of each quality dimension. For example, for the instance dimension, if the instance difference value of the first sample image and the third sample image in the instance dimension is less than a first threshold value (such as 0.5), it can be indicated that the first sample image and the third sample image match in the instance dimension; if the instance difference value of the first sample image and the third sample image in the instance dimension is not less than the first threshold value, it can be indicated that the first sample image and the third sample image do not match in the instance dimension. For the image style dimension, if the style difference value of the first sample image and the third sample image in the image style dimension is less than a second threshold value (such as 0.3), it can be indicated that the first sample image and the third sample image match in the image style dimension; if the instance difference value of the first sample image and the third sample image in the image style dimension is not less than the second threshold value, it can be indicated that the first sample image and the third sample image do not match in the image style dimension.

[0096] The first phase of training for the Vincent graph model can be performed based on the quality difference values ​​for each quality dimension. For example, if the instance difference value for the instance dimension determines that the first sample image and the second sample image do not match, the first phase of training for the Vincent graph model can be performed based on the instance difference value to ensure that the first sample image and the third sample image match in the instance dimension. Similarly, if the style difference value for the image style dimension determines that the first sample image and the second sample image do not match, the first phase of training for the Vincent graph model can be performed based on the style difference value to ensure that the first sample image and the third sample image match in the image style dimension.

[0097] Furthermore, if the first sample image and the second sample image match in one quality dimension, then the first-stage training of the Vincent graph model based on the quality difference value of that quality dimension can be eliminated. For example, if the first sample image and the second sample image are determined to be mismatched based on the instance difference value of the instance dimension, but are matched based on the style difference value of the image style dimension, then the first-stage training of the Vincent graph model based on the instance difference value can be eliminated.

[0098] In this embodiment, a loss function as shown in Expression (1) can be constructed based on the instance difference value to train the text graph model from the instance dimension:

[0099] Where E represents the expectation, D represents the first sample image set, represents the third sample image set.

[0100] Furthermore, based on the style difference value, a loss function as shown in Expression (2) can be constructed to train the text image model from the image style dimension:

[0101] It is understandable that after training the Vincent graph model separately from different quality dimensions, the trained Vincent graph model can generate a third sample image that matches the first sample image according to each quality dimension. Compared to some technologies that compare the first and third sample images as a whole and train the Vincent graph model based on the results of the overall comparison, the present application splits and compares the first and third sample images according to each quality dimension, and based on the results of the split comparison, trains the Vincent graph model separately from each quality dimension. In this way, the training of the Vincent graph model is more refined, so that the trained Vincent graph model can have higher accuracy, which in turn can improve the accuracy of the generated image.

[0102] Referring to FIG6 , which is a schematic diagram of module interaction for the first phase of training provided in another embodiment of the present application, in some embodiments, the first phase of training may further include:

[0103] Inputting the third sample image into the sub-scoring model, so as to score the third sample image according to multiple aesthetic dimensions representing the beauty of the image through the sub-scoring model, thereby obtaining an aesthetic score of the third sample image in each aesthetic dimension;

[0104] If the aesthetic score does not exceed the score threshold, the text graph model is trained based on the aesthetic score.

[0105] Specifically, the aesthetic dimension includes, but is not limited to, image lighting effects, image color, image layout, etc. This application does not impose any restrictions on the division of the aesthetic dimension. It should be noted that the aesthetic dimension is a dimensional division of the third sample image from the perspective of its aesthetics, while the above-mentioned quality dimension is a dimensional division of the third sample image from the perspective of evaluating the similarity between the third sample image and the first sample image. Therefore, the aesthetic dimension and the quality dimension may be the same or different.

[0106] With reference to FIG6 , in this embodiment, each aesthetic dimension may correspond to a trained sub-scoring model for scoring the aesthetic quality of the third sample image in that aesthetic dimension. For example, in FIG6 , the lighting scoring model may be used to score the lighting effects of the third sample image, and the color scoring model may be used to score the colors of the third sample image.

[0107] For any sub-scoring model, the higher the aesthetic score output by that sub-scoring model, the better the aesthetic quality of the third sample image in the corresponding aesthetic dimension. Based on the aesthetic scores output by each sub-scoring model, the Vincent graph model can be trained separately so that the aesthetic scores of the third sample images generated by the trained Vincent graph model meet the scoring threshold in each aesthetic dimension. This improves the aesthetic quality of the third sample images.

[0108] Specifically, based on the aesthetic scores output by each sub-scoring model, a loss function as shown in Expression (3) can be constructed to train the text graph model from the aesthetic dimension:

[0109] Where c represents the prompt word used to guide the Wensheng graph model to generate the third sample image, x′0 represents the third sample image, and r d represents the sub-scoring model, r d (x′0,c) represents the beauty score of the third sample image, α d Represents the scoring threshold, and the summation brackets represent scoring of multiple aesthetic dimensions of the third sample image. In expression (3), for any aesthetic dimension, the aesthetic score r of the third sample image in the aesthetic dimension is d (x′0,c) reaches the score threshold α d When , the aesthetic rating r based on the aesthetic dimension can be stopped d (x′0,c) is used to train the Wensheng graph model.

[0110] After completing the first phase of training, the Wensheng graph model has already achieved relatively good performance in terms of image quality and image aesthetics. Based on the Wensheng graph model obtained after the first phase of training, the second phase of training can be continued to further improve the inference speed of the Wensheng graph model.

[0111] Specifically, refer to Figures 7 and 8. Figure 7 is a flow chart of the second phase training provided by an embodiment of the present application. Figure 8 is a schematic diagram of the module interaction of the second phase training in Figure 7. In Figure 7, the second phase training may specifically include the following steps:

[0112] Step S71 : inputting the first sample image and the third sample image into a trained total score model, so as to output a first image score of the first sample image and a second image score of the third sample image through the total score model.

[0113] Specifically, the image score can represent the quality of the sample image. The overall score model can be a reward model. A higher image score output by the overall score model indicates a better image, and a lower image score output by the overall score model indicates a worse image.

[0114] In this embodiment, the image score can be a score obtained using the first sample image as a benchmark. Simply put, the first sample image is considered the best image, meaning that the first sample image's first image score can be the highest. The higher the similarity between the third sample image and the first sample image, the higher the second image score can be. Conversely, the lower the similarity between the third sample image and the first sample image, the lower the second image score can be.

[0115] In this application, the use of r a (x0) represents the first image score output by the total score model for the first sample image, and the use of r a (x′0,c) represents the second image score output by the total score model for the third sample image, where r a represents the total score model, x0 represents the first sample image, x′0 represents the third sample image, and c represents the prompt word that guides the text-based graph model to generate the third sample image.

[0116] Step S72 : Based on the second image score, the second stage training is performed on the Wensheng graph model to increase the second image score of the third sample image output by the Wensheng graph model.

[0117] Specifically, based on the second image score, a loss function as shown in Expression (4) can be constructed to train the Wensheng graph model:

[0118] Among them, in expression (4), in r d1 The meaning of adding a negative sign before (x′0,c) is: a The higher (x′0,c) is, the smaller the loss of the Wensheng graph model is.

[0119] Step S73: Based on the first image score and the second image score, the total score model is trained so that the first image score output by the total score model is increased and the second image score is decreased.

[0120] It can be understood that after training the Wensheng graph model based on expression (4), the third sample image generated by the Wensheng graph model will become better and better, that is, the second image score r d(x′0,c) will become higher and higher. In this case, in order to make the third sample image generated by the Vincent graph model better and better, the total scoring model can be trained based on the first image score and the second image score, so that the total scoring model has more and more stringent requirements for the third sample image. Correspondingly, under stricter standards, the second image score of the third sample image by the total scoring model should be lower. In this way, the Vincent graph model can continue to be trained under a lower second image score. This training process can also be called an adversarial training process. Specifically, in the adversarial training process, the total scoring model can regard the first sample image as a good image, output a high first image score for the first sample image, and regard the third sample image as a bad image, output a low second image score for the third sample image. In this way, the accuracy of the Vincent graph model can be improved through such adversarial training.

[0121] Specifically, in this embodiment, a loss function as shown in Expression (5) can be constructed based on the first image score and the second image score to train the total score model:

[0122] Among them, σ is the adjustment parameter.

[0123] In the second stage of training, the third sample image with a low number of denoising steps can also receive sufficient feedback optimization, so that after completing the second stage of training, the third sample image obtained by the Vincent graph model using fewer denoising steps can also have better image quality, which is equivalent to speeding up the inference speed of the Vincent graph model.

[0124] Please refer to Figure 9 for a schematic diagram of the module interactions during the second phase of training, provided in another embodiment of the present application. In Figure 9, while the document graph model is trained based on the overall scoring model, it can also be trained based on multiple sub-scoring models. This further improves the accuracy of the document graph model. The principles of the sub-scoring models can be found in Figure 6 and will not be further elaborated here.

[0125] This concludes the first and second phases of training. The Wensheng graph model, after the first and second phases of training, can achieve higher accuracy and inference speed.

[0126] Referring to FIG10 , which is a block diagram of a data processing system according to an embodiment of the present application, the image generation system includes:

[0127] A data acquisition module is used to obtain multiple sample prompt words for model training;

[0128] A first data extraction module is configured to analyze the correspondence between entities and entity attributes in the sample prompt words, and extract the sample prompt words whose correspondences meet the complexity index to obtain a first data set;

[0129] A second data extraction module is used to extract sample prompt words that meet the difference index from the first data set to obtain a second data set;

[0130] A training module is used to perform model training based on the second data set.

[0131] 11 is a schematic diagram of an electronic device according to an embodiment of the present application. The electronic device includes a processor and a memory, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the above method is implemented.

[0132] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0133] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the methods in the aforementioned method embodiments.

[0134] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] One embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the above method is implemented.

[0136] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method, comprising: Obtain multiple sample prompt words for model training; parsing the correspondence between entities and entity attributes in the sample prompt words, and extracting sample prompt words whose correspondences meet the complexity index to obtain a first data set; Extracting sample prompt words that meet the difference index from the first data set to obtain a second data set; Model training is performed based on the second data set.

2. The method of claim 1, wherein the complexity indicator comprises a minimum number of corresponding relationships between entities and entity attributes; The extraction of sample prompt words whose corresponding relationships meet the complexity index includes: In response to the presence of an entity satisfying the following conditions in a sample prompt word, the sample prompt word is extracted into the first data set: The number of corresponding relationships between entities and entity attributes reaches the minimum number of corresponding relationships.

3. The method of claim 1, wherein the difference indicator comprises a first similarity threshold below which the similarity between the sample prompt words must be lower than; The step of extracting sample prompt words that meet the difference index from the first data set includes: Performing feature extraction on each of the sample prompt words to obtain a feature vector of each of the sample prompt words; Searching for a first sample prompt word and a second sample prompt word whose feature vector distance is within a preset distance range, and determining a first similarity between the first sample prompt word and the second sample prompt word; In response to the first similarity being higher than the first similarity threshold, deleting the second sample prompt word or the first sample prompt word from the first data set; The first data set after deleting the sample prompt words is used as the second data set.

4. The method according to claim 3, wherein before extracting the second data set from the first data set according to the first similarity, the method further comprises: In the first data set, searching for multiple groups of first sample prompt words and second sample prompt words whose feature vector distances are within a preset distance range, and calculating the second similarity of each group of first sample prompt words and second sample prompt words respectively; In response to an average of second similarities between the multiple groups of first sample prompt words and second sample prompt words being lower than a second similarity threshold, the first data set is used as the second data set.

5. The method of claim 1, wherein the trained model comprises a text-graph model, wherein the text-graph model is used to generate an image based on a prompt word; After the document graph model is trained based on the second data set, fine-tuning training is performed on the document graph model based on the following method: Acquire a first sample image, and perform noise processing on the first sample image to obtain a second sample image; inputting the second sample image into the Vincent graph model to convert the second sample image into a third sample image through the Vincent graph model; Comparing the first sample image with the third sample image to obtain quality difference values ​​between the first sample image and the third sample image in various quality dimensions representing image quality; In response to the quality difference value indicating that the first sample image and the third sample image do not match, a first stage training is performed on the Vincent graph model according to the quality difference value.

6. The method of claim 5, wherein the quality dimension comprises an instance dimension, the quality difference value comprises an instance difference value between the first sample image and the third sample image in the instance dimension, and the first sample image has an instance segmentation annotation; The comparing the first sample image and the third sample image includes: Performing instance segmentation on the third sample image to obtain an instance segmentation result of the third sample image; The difference value between the instance segmentation annotation and the instance segmentation result is used as the instance difference value between the first sample image and the third sample image in the instance dimension.

7. The method according to claim 5, wherein the quality dimension comprises an image style dimension, and the quality difference value comprises a style difference value between the first sample image and the third sample image in the image style dimension; The comparing the first sample image and the third sample image includes: extracting a first style value representing a style of the first sample image from the first sample image, and extracting a second style value representing a style of the third sample image from the third sample image; The difference value between the first style value and the second style value is used as the style difference value between the first sample image and the third sample image in the image style dimension.

8. The method of claim 5, wherein after completing the first stage of training, the method further comprises: Inputting the first sample image and the third sample image into a trained total score model, so as to output a first image score of the first sample image and a second image score of the third sample image through the total score model; performing a second-stage training on the Wensheng graph model based on the first image score, so as to increase the first image score of the third sample image output by the Wensheng graph model; The total score model is trained based on the first image score and the second image score, so that the first image score output by the total score model is reduced and the second image score is increased.

9. The method of claim 8, wherein in the first phase training and the second phase training, at least one phase training further comprises: Inputting the third sample image into a sub-scoring model, so as to score the third sample image according to a plurality of aesthetic dimensions representing the beauty of the image by the sub-scoring model, thereby obtaining an aesthetic score of the third sample image in each of the aesthetic dimensions; If the aesthetic score does not exceed the score threshold, the text graph model is trained according to the aesthetic score.

10. A data processing system comprising: A data acquisition module is used to obtain multiple sample prompt words for model training; A first data extraction module is configured to analyze the correspondence between entities and entity attributes in the sample prompt words, and extract the sample prompt words whose correspondences meet the complexity index to obtain a first data set; A second data extraction module is used to extract sample prompt words that meet the difference index from the first data set to obtain a second data set; A training module is used to perform model training based on the second data set.

11. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. An electronic device, wherein the electronic device comprises a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

Images