Vegetation graph model training method, image generation method, device and medium

Abstract

The invention provides a text graph model training method and device, an image generation method and device and a medium, relates to the technical field of artificial intelligence such as computer vision, deep learning and the like, can be applied to scenes such as AIGC content generation based on artificial intelligence, and comprises the following steps: obtaining a candidate text graph model and a sample high-quality image set; obtaining a candidate user preference data set corresponding to the user set, and obtaining a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set; constructing a sample image-text pair set corresponding to the sample high-quality preference image set, and based on the sample preference image-text pair set, obtaining a target fine tuning sample required by the candidate text graph model; and based on the target fine adjustment sample, performing model fine adjustment on the candidate text graph model to obtain a target text graph model.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to artificial intelligence fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Background Art

[0002] With the development of technology, people have higher and higher requirements for the quality of images generated by artificial intelligence. In related technologies, the text image model can be trained based on a general data set to improve the image quality of the model output. However, the user preference adaptation is poor, which affects the user experience. Summary of the invention

[0003] The present disclosure proposes a training method, an image generation method, a device and a medium for a text-based graph model.

[0004] According to a first aspect of the present disclosure, a training method for a text graph model is proposed, comprising: obtaining a candidate text graph model and a sample high-quality image set; obtaining a candidate user preference data set corresponding to a user set, and obtaining a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set; constructing a sample image-text pair set corresponding to the sample high-quality preference image set, and obtaining a target fine-tuning sample required for the candidate text graph model based on the sample preference image-text pair set; performing model training on the candidate text graph model based on the target fine-tuning sample to obtain a target text graph model.

[0005] According to a second aspect of the present disclosure, an image generation method is proposed, comprising: obtaining a fine-tuned target text graph model, wherein the target text graph model is obtained based on the training method of the text graph model proposed in the first aspect above; obtaining an image generation requirement input by a user terminal, and obtaining a corresponding image generation requirement prompt text based on the image generation requirement; inputting the image generation requirement prompt text into the target text graph model, and obtaining a target image required by the user terminal through the model capability of the target text graph model.

[0006] According to a third aspect of the present disclosure, a training device for a text graph model is proposed, comprising: a first acquisition module, used to acquire a candidate text graph model and a sample high-quality image set; a second acquisition module, used to acquire a candidate user preference data set corresponding to a user set, and acquire a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set; a construction module, used to construct a sample image-text pair set corresponding to the sample high-quality preference image set, and obtain a target fine-tuning sample required for the candidate text graph model based on the sample preference image-text pair set; and a training module, used to perform model training on the candidate text graph model based on the target fine-tuning sample to obtain a target text graph model.

[0007] According to a fourth aspect of the present disclosure, an image generation device is proposed, comprising: a third acquisition module, used to acquire a fine-tuned target Wensheng graph model, wherein the target Wensheng graph model is obtained based on the training device of the Wensheng graph model proposed in the third aspect; a fourth acquisition module, used to acquire an image generation requirement input by a user terminal, and obtain a corresponding image generation requirement prompt text based on the image generation requirement; an image generation module, used to input the image generation requirement prompt text into the target Wensheng graph model, and obtain the target image required by the user terminal through the model capability of the target Wensheng graph model.

[0008] According to a fifth aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the Vincent graph model proposed in the first aspect and / or the image generation method proposed in the second aspect.

[0009] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the training method of the Vincent graph model proposed in the first aspect and / or the image generation method proposed in the second aspect.

[0010] According to a seventh aspect of the present disclosure, a computer program product is proposed, including a computer program, which, when executed by a processor, implements the training method of the Vincent graph model proposed in the first aspect and / or the image generation method proposed in the second aspect.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 A flowchart of a method for training a text graph model according to an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of a flow chart of a training method for a text graph model according to another embodiment of the present disclosure;

[0015] Figure 3A schematic diagram of a flow chart of a training method for a text graph model according to another embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of a training method and an image generation method of a Vincent graph model according to another embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of a flow chart of a training method for a text graph model according to another embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of a flow chart of an image generating method according to an embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram of the structure of a training device for a text graph model according to an embodiment of the present disclosure;

[0020] Figure 8 is a schematic structural diagram of an image generating device in one embodiment of the present disclosure;

[0021] Fig. 9 A schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Data processing is a basic part of system engineering and automatic control. Data is a form of expression of facts, concepts or instructions, which can be processed by manual or automatic devices. After data is interpreted and given a certain meaning, it becomes information. Data processing is the collection, storage, retrieval, processing, transformation and transmission of data. The basic purpose of data processing is to extract and derive valuable and meaningful data for certain specific people from a large amount of data that may be disorganized and difficult to understand.

[0024] Computer vision refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. The information here refers to the information defined by Shannon that can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.

[0025] Deep learning refers specifically to machine learning based on deep neural network models and methods. It is developed based on statistical machine learning, artificial neural network and other algorithmic models, combined with the development of contemporary big data and large computing power. The most important technical feature of deep learning is the ability to automatically extract features. The extracted features are also called deep features or deep feature representations. Compared with artificially designed features, deep features have stronger and more robust representation capabilities. Therefore, the essence of deep learning is feature representation learning. Deep neural networks are the model basis for deep learning to automatically extract features. Deep neural networks are essentially a nested series of nonlinear transformations.

[0026] Artificial Intelligence (AI) is an important driving force for a new round of scientific and technological revolution and industrial transformation. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0027] Artificial intelligence is an important part of the intelligent discipline. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is a very broad science, including robotics, language recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.

[0028] Early Stopping is a commonly used machine learning technique, mainly used to prevent model overfitting. By monitoring the performance of the model on the validation set, the training process is terminated early when the performance no longer improves. This method not only helps improve the generalization ability of the model, but also saves computing resources and time.

[0029] Figure 1 FIG. 1 is a flow chart of a training method of a text graph model according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0030] S101, obtaining candidate text graph models and sample high-quality image sets.

[0031] In the embodiments of the present disclosure, the generation of artificial intelligence images can be achieved through the Wensheng graph model, wherein the Wensheng graph model may be obtained by model training based on general samples. In this scenario, the model training based on the general samples makes it possible that the quality of the image generated by the trained Wensheng graph model may not meet the quality requirements of the user end. In this scenario, the general data set may not be able to introduce user preference data, resulting in poor adaptation between the image generated by the Wensheng graph model and the user preference.

[0032] Optionally, the document graph model may be further fine-tuned, wherein the document graph model that needs to be fine-tuned may be determined as a candidate document graph model to be fine-tuned.

[0033] Optionally, a corresponding high-quality image set may be obtained for the candidate text graph model, and the candidate text graph model may be fine-tuned using the high-quality image set, so that the quality of images generated by the text graph model is improved.

[0034] Among them, the high-quality image set used when fine-tuning the candidate text-generated graph model can be determined as the sample high-quality image set.

[0035] Among them, the sample high-quality image set can be understood as an image set consisting of high-quality images whose parameters such as clarity, resolution, color expression, contrast, brightness, detail level, noise, artifacts and dynamic range meet the preset image quality conditions.

[0036] S102, obtaining a candidate user preference data set corresponding to the user set, and obtaining a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set.

[0037] In the disclosed embodiment, while fine-tuning the candidate text-based graph model in the dimension of image quality, the candidate text-based graph model can also be fine-tuned in the dimension of user preference, so that the high-quality image generated by the fine-tuned text-based graph model can meet the user's quality requirements while also being adapted to the user's preferences.

[0038] In this scenario, the preference data of each user in the user set can be obtained to form a preference data set corresponding to the user set, which is marked as a candidate user preference data set. Furthermore, user preferences are annotated for the sample high-quality image set based on the candidate user preference data set, and the annotated image set is determined as the sample high-quality preference image set.

[0039] Among them, for any sample high-quality image, user preference data matching the sample high-quality image can be obtained from the candidate user preference data set, and the user preference data can be annotated on the sample high-quality image based on the image annotation method in the relevant technology, and the annotated image is determined as the sample high-quality preference image of the sample high-quality image. Furthermore, based on the sample high-quality preference image of each sample high-quality image, a sample high-quality preference image set is obtained.

[0040] S103, constructing a sample image-text pair set corresponding to the sample high-quality preference image set, and obtaining target fine-tuning samples required for the candidate text-generated image model based on the sample preference image-text pair set.

[0041] In the disclosed embodiment, the input text of the candidate text-generated graph model can be obtained through a sample high-quality preference image set, wherein the description text of each sample high-quality preference image in the sample high-quality preference image set can be obtained, and for any description text, the sample high-quality preference image set corresponding to the description text is obtained, and a picture-text pair consisting of the description text and the sample high-quality preference image is constructed and determined as a sample picture-text pair.

[0042] Furthermore, according to the sample image-text pairs of each sample high-quality preference image, a sample image-text pair set of the sample high-quality preference image set is composed.

[0043] In this scenario, samples used for fine-tuning the candidate text-graph model can be obtained based on the sample text-graph pair set, and determined as target fine-tuning samples for the candidate text-graph model.

[0044] Optionally, some sample image-text pairs used by the candidate text-generated graph model in the current round of fine-tuning can be obtained from the sample image-text pair set, and sample construction can be performed based on the partial sample image-text pairs based on the sample construction method in the relevant technology, so as to obtain the target fine-tuning samples of the candidate text-generated graph model input into the model in the current round of fine-tuning.

[0045] S104, performing model training on the candidate text-generated graph model based on the target fine-tuning sample to obtain a target text-generated graph model.

[0046] In the disclosed embodiment, for the current round of model fine-tuning, the corresponding target fine-tuning sample may be input into the candidate text graph model, and an image generated by the candidate text graph model based on the target fine-tuning sample may be obtained and output.

[0047] Optionally, the fine-tuning loss of the model is obtained based on the image output by the candidate text graph model in the current model fine-tuning round, and the model parameters of the candidate text graph model are adjusted and optimized based on the fine-tuning loss until the adjusted and optimized model meets the end conditions of the model fine-tuning. The model fine-tuning training of the candidate text graph model can be ended, and the fine-tuned model is determined as the target text graph model.

[0048] The training method of the text graph model proposed in the present invention obtains a candidate text graph model and a sample high-quality image set and a candidate user preference data set of a user set, obtains a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set, constructs a sample image-text pair set corresponding to the sample high-quality preference image set to obtain a target fine-tuning sample of the candidate text graph model, and fine-tunes the candidate text graph model through the target fine-tuning sample, thereby obtaining a fine-tuned target text graph model. In the present disclosure, a target fine-tuning sample is obtained based on a sample high-quality preference image set, and then a target text graph model is obtained by fine-tuning the model through the target fine-tuning sample, thereby improving the image quality of the image generated by the target text graph model and improving the degree of adaptation between the image generated by the target text graph model and the user preference. The sample high-quality image set introduces a candidate preference data set of a user set composed of multiple users to obtain a sample high-quality preference image set, so that the target text graph model realizes the preference alignment of multiple users, and there is no need to perform separate model training for each user among the multiple users, thereby reducing the acquisition cost of the text graph model. The model is fine-tuned based on the trained text graph model, thereby improving the model performance of the fine-tuned target text graph model. In the scenario of image generation through the target text graph model, the user experience is optimized and the user stickiness is improved.

[0049] In the above embodiment, the acquisition of the target fine-tuning sample of the target cultural graph model can be combined with Figure 2 Further understanding, Figure 2 FIG. 1 is a flow chart of a training method of a text graph model according to another embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0050] S201, obtaining a preset high-quality image screening strategy, wherein the high-quality image screening strategy is obtained based on at least a preset image aesthetic screening strategy, an image quality screening strategy, and an image style screening strategy.

[0051] In the disclosed embodiment, a sample high-quality image set required for a candidate text graph model can be obtained from a source database, wherein a screening strategy corresponding to the sample high-quality image set can be constructed and determined as a high-quality image screening strategy, and then the high-quality image set is screened from the source database using the obtained high-quality image screening strategy, thereby obtaining the sample high-quality image set required for the candidate text graph model.

[0052] Optionally, a corresponding image screening strategy based on the aesthetic dimension of the image can be constructed and marked as an image aesthetic screening strategy, a corresponding image screening strategy based on the quality dimension of the image can be constructed and marked as an image quality screening strategy, and a corresponding image screening strategy based on the style dimension of the image can be constructed and marked as an image style screening strategy.

[0053] S202, performing image screening on the pre-acquired source database based on the high-quality image screening strategy, and obtaining a sample high-quality image set matching the image screening strategy from the source database.

[0054] In the disclosed embodiment, after obtaining the high-quality image screening strategy, the pre-acquired source database may be screened to obtain a screened high-quality image set as a sample high-quality image set.

[0055] As an example, Figure 3 As shown, high-quality image filtering conditions can be used to select Figure 3 The source database 31 shown is used to screen high-quality images, thereby obtaining a sample high-quality image set.

[0056] The source database 31 may include an open source image database, a sample image data set used when fine-tuning a candidate text graph model, or other image databases, which are not specifically limited here.

[0057] In the embodiment of the present disclosure, the user end of the cultural graph model may have a demand for generating images within a specific concept range, wherein the images within the specific concept range may include images within a pet breed range, images within a car brand range, and images within other specific concept ranges.

[0058] In this scenario, after obtaining the sample high-quality image set, further image screening can be performed based on a specific concept-limited range, wherein a preset specific concept image screening strategy can be obtained, and a sample specific concept image set that matches the specific concept image screening strategy in the sample high-quality image set can be obtained.

[0059] In the disclosed embodiments, the screening strategy corresponding to the images within a specific concept-limited range can be determined as the specific concept image screening strategy. For example, for the images within a pet breed-limited range, the specific concept image screening strategy corresponding to the corresponding pet breed can be constructed based on the main feature images of each pet breed limited in the range, and the screening of high-quality images of each pet breed can be achieved based on the strategy.

[0060] Optionally, in response to the sample specific concept image set not satisfying the fine-tuning sample conditions of the candidate text graph model, specific concept image screening is performed based on a specific concept image screening strategy to obtain a new sample specific concept image set, and the sample high-quality image set is updated based on the new sample specific concept image set to obtain a new sample high-quality image set.

[0061] In the disclosed embodiment, after the source database is screened for high-quality images, the number of images in the screened sample high-quality image set that fall within the specified range of a specific concept can be counted to obtain the number of images within the specified range of the specific concept.

[0062] Optionally, model fine-tuning of the candidate text graph model has preset requirements for the sample size of fine-tuning samples, and the fine-tuning sample conditions of the candidate text graph can be set based on the requirements. In this scenario, the number of images acquired within the specified range of a specific concept can be compared with the fine-tuning sample conditions. When the comparison result indicates that the number of images matches the fine-tuning sample conditions, it can be determined that the image set composed of images within the specified range of the specific concept in the current sample high-quality image set can meet the fine-tuning requirements of the candidate text graph model for image generation within the specified range of the specific concept, and the model fine-tuning of the candidate text graph model can be achieved based on the current sample high-quality image set.

[0063] Accordingly, when the number of images acquired within the specified range of a specific concept does not match the fine-tuning sample conditions, it can be determined that the image set consisting of images within the specified range of a specific concept in the current sample high-quality image set cannot meet the fine-tuning requirements of the candidate text graph model for generating images within the specified range of a specific concept.

[0064] In this scenario, high-quality images within the specific concept limit can be further screened based on the specific concept image screening strategy, and the new high-quality image set obtained by screening can be determined as a new sample specific concept image set.

[0065] Furthermore, the new sample specific concept image set can be deduplicated and integrated with the sample high-quality image set screened from the source database, thereby realizing the update of the sample specific concept image set to the sample high-quality image set, thereby obtaining a new sample high-quality image set.

[0066] S203, obtaining candidate user preference data of each user in the user set to obtain a candidate user preference data set composed of the preference data of each candidate user.

[0067] In the embodiment of the present disclosure, preference data of each user in the user set can be determined as candidate user preference data of each user, wherein a user portrait of each user in the user set can be obtained, thereby realizing the extraction of candidate user preference data of each user.

[0068] Furthermore, a set of candidate preference data of each user is determined as a candidate user preference data set corresponding to the user set.

[0069] S204, for any sample high-quality image, obtain target user preference data that matches the sample high-quality image from the candidate user preference data set, and perform preference alignment between the target user preference data and the sample high-quality image to obtain a sample high-quality preference image corresponding to the sample high-quality image.

[0070] In the disclosed embodiment, the candidate user preference data set may be introduced into the sample high-quality image set, thereby obtaining the sample high-quality preference image set required for fine-tuning the candidate text graph model.

[0071] Optionally, for any sample high-quality image, algorithmic processing can be performed on the sample high-quality image and each candidate user preference data in the candidate user preference data set based on a matching algorithm of related technology to obtain an algorithm result of the degree of matching between each candidate user preference data and the sample high-quality image, and then based on the algorithm result, the preference data matching the sample high-quality image is filtered out from the candidate user preference data set and marked as the target user preference data.

[0072] In this scenario, preference data can be annotated on the corresponding sample high-quality image based on the target user's preference data, thereby achieving preference alignment between the target user's preference data and the corresponding sample high-quality image. Furthermore, the image after the preference alignment processing is determined as the sample high-quality preference image corresponding to the sample high-quality image.

[0073] S205 , obtaining a sample high-quality preference image set based on the sample high-quality preference images of each sample high-quality image.

[0074] Optionally, a set construction method may be used to construct a sample high-quality preference image for each sample high-quality image, and the constructed set may be determined as a corresponding sample high-quality preference image set.

[0075] As an example, Figure 3 As shown, in Figure 3 In the target fine-tuning sample construction 33 scenario shown in FIG. 1 , the candidate user preference data set corresponding to the user preference 34 can be introduced into the sample high-quality image set obtained by screening the source database, thereby obtaining Figure 3 A sample high quality preference image set 32 ​​is shown.

[0076] S206, obtaining a sample target description text of each sample high-quality preference image in the sample high-quality preference image set.

[0077] As a possible implementation method, based on the image description information extraction method in the relevant technology, description information can be extracted for each sample high-quality preference image, and then the description text of each sample high-quality preference image is obtained according to the extracted description information, and marked as the target description text of each sample high-quality preference image.

[0078] As another possible implementation method, the target description text can be acquired through the model capabilities of a multimodal large language model (MLLM), wherein a multimodal large language model can be acquired and the sample initial description text of each sample high-quality preference image can be generated through the model capabilities of the multimodal large language model.

[0079] In the disclosed embodiment, each sample high-quality preference image can be input into a multimodal large language model to obtain an output text of the multimodal large language model, and the text can be determined as a sample initial description text of each sample high-quality preference image.

[0080] Optionally, description correction is performed on the initial description text of each sample to obtain a sample candidate description text of each sample high-quality preference image.

[0081] In the disclosed embodiment, the sample initial description text output by the multimodal large language model may differ to a certain extent from the image information of the sample high-quality preference image. In this scenario, the sample initial description text of each sample high-quality preference image may be subjected to description correction, wherein the image information of each sample high-quality preference image may be manually compared with the sample initial description text of each sample high-quality preference image, and the description information of each sample initial description text may be modified according to the comparison result, so that the sample description text may be aligned with the image information of its corresponding sample high-quality preference image, thereby achieving description correction of each sample initial description text.

[0082] Furthermore, the corrected text obtained after the description correction is determined as the sample candidate description text of each sample initial description text.

[0083] Optionally, through the model capability of the multimodal large language model, preference text embedding is performed on each sample candidate description text based on the candidate user preference data set to obtain a sample target description text of each sample high-quality preference image.

[0084] In the disclosed embodiment, user preference alignment can be performed on candidate description texts of each sample based on the candidate user preference data set of the user set, and the aligned description text is determined as the sample target description text of each sample high-quality preference image.

[0085] Optionally, for any sample high-quality preference image, obtain the target user preference text aligned with the sample high-quality preference image, and through the model capabilities of the multimodal large language model, align the target user preference text to the sample candidate description text corresponding to the sample high-quality preference image to obtain the sample target description text of the sample high-quality preference image.

[0086] In the disclosed embodiment, for any sample high-quality preference image, preference data used when aligning the high-quality image with the user preference can be obtained from the candidate user preference data set and determined as the target user preference data for the sample high-quality preference image.

[0087] Furthermore, text generation is performed based on the target user preference data to obtain the target user preference text of the sample high-quality preference image corresponding to the target user preference data.

[0088] In this scenario, the sample candidate description text of any sample high-quality preference image is obtained, and the target user preference text of the sample high-quality preference image is embedded into the sample candidate description text by calling the model capability of the multimodal large language model. The embedded description text is integrated with text dimensions such as semantic dimension and word order dimension, so as to achieve preference alignment between the target user preference text and the sample candidate description text, and thus obtain the sample target description text of the sample high-quality preference image corresponding to the sample candidate description text.

[0089] S207, obtaining sample image-text pairs of each sample high-quality preference image according to each sample high-quality preference image and the sample target description text of each sample high-quality preference image, so as to obtain a sample image-text pair set composed of each sample image-text pair.

[0090] In the disclosed embodiments, for any sample high-quality preference image, a sample target description text of the sample high-quality preference image can be obtained, and a picture-text pair constructed by the sample high-quality preference image and the sample target description text can be determined as the sample picture-text pair of the sample high-quality preference image.

[0091] Furthermore, a set of sample image-text pairs is determined as a sample image-text pair set.

[0092] As an example, Figure 3 As shown, it can be Figure 3 In the scenario of target fine-tuning sample build 33 shown in the figure, Figure 3 The user preference 34 shown is aligned with the description text of each sample high-quality preference image in the sample high-quality preference atlas 32, thereby obtaining Figure 3 A set 35 of sample image-text pairs is shown.

[0093] S208, sampling the sample image-text pair set to obtain target fine-tuning samples required by the candidate text-image model.

[0094] As a possible implementation method, sample collection of sample image-text pairs can be performed based on sampling methods such as stratified sampling or systematic sampling in related technologies, and the target fine-tuning text used in the current round of model fine-tuning of the candidate text-image model can be constructed based on the collected samples.

[0095] As another possible implementation method, a random sampling strategy for a set of sample image-text pairs may be obtained, and the set of sample image-text pairs may be sampled based on the random sampling strategy to obtain target fine-tuning samples required for the candidate text-graph model.

[0096] In the disclosed embodiment, a strategy for randomly sampling samples may be marked as a random sampling strategy, and a set of sample image-text pairs may be sampled based on the random sampling strategy.

[0097] Among them, the sample image-text pair set can be loaded through a preset candidate data loader to obtain a loaded target data loader, and the sample image-text pair set in the target data loader can be polled and sampled based on a random sampling strategy to obtain a candidate fine-tuning sample set.

[0098] In the disclosed embodiment, a corresponding data loader may be constructed and marked as a candidate data loader, and a set of sample image and text pairs may be loaded in the candidate data loader, and the loaded data loader may be determined as a target data loader.

[0099] Furthermore, based on the random sampling strategy, polling sampling is performed from the set of sample image pairs loaded in the target data loader, and each sample obtained by the polling sampling is determined as a candidate fine-tuning sample, thereby obtaining a candidate fine-tuning sample set composed of each candidate fine-tuning sample.

[0100] Optionally, a target fine-tuning sample required for the candidate text graph model in the current fine-tuning round is obtained from the candidate fine-tuning sample set.

[0101] In this scenario, samples used for fine-tuning the candidate text graph model in the current round of model fine-tuning can be obtained from the candidate fine-tuning sample set as target fine-tuning samples.

[0102] As an example, Figure 3 As shown, it can be Figure 3 In the model fine-tuning 36 scenario shown in FIG. 1 , random sampling 37 is performed based on a random sampling strategy to poll and sample from a target data loader loaded with a set of sample image-text pairs to obtain a corresponding set of candidate fine-tuning samples, and then obtain a corresponding set of candidate fine-tuning samples from the set of candidate fine-tuning samples. Figure 3The candidate text graph model 38 shown is a target fine-tuning sample used for model fine-tuning in the current round, and then the model fine-tuning of the candidate text graph model is implemented based on the target fine-tuning sample.

[0103] As another example, Figure 4 As shown, in Figure 4 In the fine-tuning phase shown in the figure, Figure 4 The user preference 41 shown is introduced into the sample high-quality image set, resulting in Figure 4 A sample high quality preference image set 42 is shown, in Figure 4 In the illustrated text graph model optimization 43 module, model fine-tuning 44 of candidate text graph models is implemented based on a sample high-quality preference image set 42 .

[0104] Among them, it can be Figure 4 The parameter initialization step 45 shown in the figure obtains the model parameters of the candidate text graph model to achieve model fine-tuning of the candidate text graph model, and then the model parameters obtained after the fine-tuning are updated to Figure 4 In the illustrated Wensheng graph model, a finely tuned target Wensheng graph model 47 is obtained.

[0105] The training method of the text graph model proposed in the present invention performs user preference alignment on a sample high-quality image set based on a candidate user preference data set of a user set to obtain a sample high-quality preference image set, and then implements model fine-tuning of the candidate text graph model based on the sample high-quality preference image set, thereby improving the image generation quality of the fine-tuned target text graph model, so that the target text graph model achieves multi-user preference alignment without the need to perform separate model training for each of the multiple users, obtains a sample initial description text through the model capability of a multimodal large language model, and performs description correction and user preference alignment processing on the sample initial description text, thereby improving the precision and accuracy of the processed sample target description text, and improving the matching degree between the sample target description text and the sample high-quality preference image, thereby improving the sample quality of the target fine-tuning sample obtained based on the sample image-text pair set, optimizing the model fine-tuning effect of the text graph model, and improving the model performance of the target text graph model.

[0106] In the above embodiment, the model fine-tuning of the target document graph model can also be combined with Figure 5 Further understanding, Figure 5 FIG. 1 is a flow chart of a training method of a text graph model according to another embodiment of the present disclosure. Figure 5 As shown, the method includes:

[0107] S501, obtaining a sample prompt text corresponding to a target fine-tuning sample, and inputting the sample prompt text into a candidate text-generated graph model to obtain an output image generated by the candidate text-generated graph model based on the sample prompt text.

[0108] In the disclosed embodiment, the input text of the candidate text graph model may be a prompt text (promot text). In this scenario, the promote text may be extracted from the target fine-tuning sample as the input text of the candidate text graph model in the current model fine-tuning round, and marked as the sample prompt text corresponding to the target fine-tuning sample.

[0109] Optionally, the sample prompt text can be input into the candidate text graph model, and the relevant features of the sample prompt text can be extracted through the candidate text graph model, and then the corresponding image can be generated according to the extracted features, which can be used as the candidate text graph model in the current fine-tuning process.

[0110] S502, obtaining a label image corresponding to the target fine-tuning sample, and obtaining a fine-tuning loss of the candidate text-generated graph model based on the output image and the label image.

[0111] Optionally, a high-quality preference image carried in the label information of the target fine-tuning sample may be obtained, and the image may be determined as a label image corresponding to the target fine-tuning sample.

[0112] Furthermore, based on the fine-tuning loss acquisition algorithm in the relevant technology, the output image and the label image of the candidate text graph model are algorithmically processed, and the loss value of the candidate text graph model in the current fine-tuning round is obtained according to the result of the algorithm processing, which is determined as the fine-tuning loss of the candidate text graph model.

[0113] As an example, Figure 3 As shown, it can be Figure 3 The target fine-tuning sample input obtained by random sampling 37 in the model fine-tuning 36 scenario is shown Figure 3 In the candidate text graph model 38 shown in FIG, the output image of the candidate text graph model 38 is obtained, and then based on the label image and the output image of the target fine-tuning sample in this example, the output image is obtained. Figure 3 The fine-tuning loss of the candidate text graph model is shown as 39.

[0114] S503, adjusting the model parameters of the candidate text graph model based on the fine-tuning loss, and returning to obtain the next target fine-tuning sample to continue fine-tuning the adjusted candidate text graph model until the fine-tuning is completed to obtain the target text graph model.

[0115] In the disclosed embodiment, the model parameters of the candidate text graph model may be adjusted and optimized based on the obtained fine-tuning loss, and the next target fine-tuning sample may be returned to continue model fine-tuning the candidate text graph model after the parameter adjustment and optimization.

[0116] Optionally, during the model fine-tuning process of the candidate text graph model, it is possible to identify whether the candidate text graph model obtained after each round of fine-tuning meets the fine-tuning end condition. For the current model fine-tuning round, when it is identified that the candidate text graph model after the end of the round meets the preset fine-tuning end condition, the model fine-tuning of the candidate text graph model can be stopped, and the model after fine-tuning in the current round can be determined as the target text graph model.

[0117] Accordingly, when it is identified that the candidate text graph model after the end of this round does not meet the preset fine-tuning end condition, it is possible to return to obtain the next target fine-tuning sample to continue model fine-tuning the candidate text graph model after parameter adjustment and optimization until the end condition of model fine-tuning is met, and then the model fine-tuning of the candidate text graph model can be stopped to obtain the target text graph model.

[0118] In the disclosed embodiment, the fine-tuning end condition of the candidate text graph model can be set based on the early stopping strategy, wherein the early stopping strategy of the candidate text graph model can be constructed according to the model training early stopping technology in the related technology to reduce the probability of overfitting of the candidate text graph model, wherein the early stopping strategy can terminate the fine-tuning process of the candidate text graph model when the performance is no longer improved, thereby improving the generalization ability of the fine-tuned target text graph model.

[0119] Among them, the model fine-tuning of the candidate text graph model based on the early stopping strategy can be understood in combination with the following content:

[0120] Optionally, an early stopping strategy of the candidate text graph model and a corresponding early stopping strategy verification set are obtained, and performance verification is performed on the candidate text graph model in the process of model fine-tuning based on the early stopping strategy verification set to obtain a performance verification result of the candidate text graph model.

[0121] In the embodiment of the present disclosure, a corresponding verification data set is provided for the early stopping strategy corresponding to the fine-tuning of the candidate text graph model, and the verification data set can be determined as the early stopping strategy verification set corresponding to the early stopping strategy.

[0122] Among them, the process of obtaining the early stopping strategy verification set can be understood in conjunction with the method for obtaining the sample high-quality preference image set proposed in the above embodiment, and will not be repeated here.

[0123] Optionally, for the current fine-tuning round, after the model fine-tuning of the fine-tuning round is completed, the performance of the candidate text graph model obtained after the fine-tuning of the round is verified, wherein the verification data currently used for the performance verification can be obtained from the early stopping strategy verification set, and the verification data can be input into the candidate text graph model obtained after the fine-tuning round, and then the model performance of the candidate text graph model obtained after the fine-tuning round is verified according to the output result of the candidate text graph model to obtain a corresponding verification result.

[0124] Among them, the verification result can be marked as the performance verification result of the candidate text graph model after this round of fine-tuning.

[0125] Optionally, based on the performance verification result, it is identified whether the candidate text graph model triggers an early stopping strategy, wherein, in response to the performance verification result indicating that performance optimization of the candidate text graph model is stagnant during the model fine-tuning process, it is determined that the candidate text graph model is identified to trigger the early stopping strategy.

[0126] In the disclosed embodiment, when it is identified based on the performance verification results that the candidate text graph model cannot continue to achieve performance optimization during the model fine-tuning process, it can be determined that the candidate text graph model has experienced performance optimization stagnation during the model fine-tuning process. In this scenario, it can be determined that the candidate text graph model meets the triggering conditions for the early stopping strategy.

[0127] Optionally, in response to identifying that the candidate text graph model triggers an early stopping strategy, model fine-tuning of the candidate text graph model is terminated to obtain a target text graph model.

[0128] In the disclosed embodiment, when it is identified that a candidate text graph model triggers an early stopping strategy, it can be determined that the candidate text graph model satisfies the fine-tuning end condition of model fine-tuning. In this scenario, the model fine-tuning of the candidate text graph model can be ended, and the text graph model obtained after the current last round of fine-tuning is determined as the target text graph model.

[0129] It should be noted that the candidate text graph model has a preset iteration round, such as Figure 3 As shown, the total number of iterations of the candidate text graph model 38 can be Figure 3 The number of iterations N shown is less than or equal to 10,000.

[0130] In this scenario, the end condition of model fine-tuning can also be set based on the number of iterations. It can be understood that when the candidate text graph model is not identified to trigger the early stopping strategy and the total number of iterations of the candidate text graph model has reached the upper limit of 10,000 iterations, it can be determined that the candidate text graph model in this scenario has met the end condition of model fine-tuning. Furthermore, the model fine-tuning of the candidate text graph model can be ended, and the text graph model obtained after the 10,000th round of fine-tuning is determined as the target text graph model.

[0131] The training method of the Wensheng graph model proposed in the present invention iteratively optimizes the candidate Wensheng graph model through fine-tuning loss, and sets the fine-tuning end condition of the candidate Wensheng graph model based on the early stopping strategy, thereby reducing the probability of overfitting of the candidate Wensheng graph model during the model fine-tuning process, reducing unnecessary iterations, reducing the consumption of resources for model fine-tuning, improving the stability and robustness of the model performance of the target Wensheng graph model after fine-tuning, and optimizing the model fine-tuning method and fine-tuning effect.

[0132] The present disclosure also proposes an image generation method, which can be combined with Figure 6 understand, Figure 6 FIG. 1 is a flow chart of an image generating method according to an embodiment of the present disclosure. Figure 6 As shown, the method includes:

[0133] S601, obtaining a finely tuned target text graph model.

[0134] Among them, the target text graph model is based on the above Figures 1 to 5 The training method of the cultural graph model proposed in the embodiment is obtained.

[0135] In the embodiment of the present disclosure, the fine-tuned document graph model can be determined as the target document graph model, wherein the fine-tuning training process of the target document graph model can be combined with the above Figures 1 to 5 The method proposed by the embodiment is understood.

[0136] S602, obtaining an image generation requirement input by the user terminal, and obtaining a corresponding image generation requirement prompt text based on the image generation requirement.

[0137] In the embodiment of the present disclosure, relevant requirements for generating an image may be input through a user terminal, wherein the requirement may be marked as an image generation requirement input through the user terminal, wherein the image generation requirement may be input in a text-based form or in a voice-based form, which is not specifically limited here.

[0138] In the disclosed embodiment, the target text-based graph model can perform feature extraction on the prompt text input therein, and then generate a corresponding image based on the extracted features. In this scenario, after obtaining the image generation requirements input by the user end, the image generation requirements can be subjected to text extraction based on the prompt text acquisition method in the related technology, and then the prompt text corresponding to the image generation requirements is obtained and determined as the image generation requirement prompt text.

[0139] The image generation requirement prompt text may be a prompt text, or other types of text that meet the input requirements of the target text-generated graph model, which is not specifically limited here.

[0140] S603, inputting the image generation requirement prompt text into the target text-based graph model, and obtaining the target image required by the user end through the model capability of the target text-based graph model.

[0141] In the disclosed embodiment, the target text graph model is obtained by fine-tuning the model based on the trained candidate text graph model. In this scenario, the image generation requirement text corresponding to the image generation requirement can be input into the fine-tuned target text graph model, and relevant features of the image generation requirement prompt text can be extracted based on the model capability of the target text graph model.

[0142] Furthermore, based on the model capability of the target text graph model, an image satisfying the image generation requirement is generated based on the extracted features and output as the target image required by the user end.

[0143] As an example, Figure 4 As shown, set Figure 4 The prompt text 48 shown is a prompt text for the user to generate the requirement. Figure 4 In the application phase scenario shown, Figure 4 The text graph model shown is the finely tuned target text graph model 47. In this scenario, the prompt text 48 can be input into Figure 4 In the fine-tuned target text graph model 47 shown, the model capability output of the target text graph model 47 is Figure 4 The generated image 49 is shown, and the generated image is determined as the target image corresponding to the user-generated requirement prompt text in this example.

[0144] The image generation method proposed in the present disclosure obtains the image generation requirement prompt text corresponding to the image generation requirement of the user end, and inputs the image generation requirement prompt text into the fine-tuned target text-based graph model, and generates the target image corresponding to the image generation requirement prompt text through the target text-based graph model. Figures 1 to 5The training method of the text-generated graph model proposed in the embodiment obtains the target text-generated graph model, improves the degree of adaptation between the target image generated based on the target text-generated graph model and the image generation requirements corresponding to the user end, improves the image generation quality of the target image, and optimizes the user's image generation experience.

[0145] An embodiment of the present disclosure further proposes a training device for a culture graph model. Since the training device for a culture graph model proposed in the embodiment of the present disclosure corresponds to the training methods for a culture graph model proposed in the above-mentioned embodiments, the implementation methods of the above-mentioned training methods for a culture graph model are also applicable to the training device for a culture graph model proposed in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0146] Figure 7 FIG. 1 is a schematic diagram of a training device for a text graph model according to an embodiment of the present disclosure. Figure 7 As shown, the training device 700 of the Wensheng graph model includes a first acquisition module 71, a second acquisition module 72, a construction module 73 and a fine-tuning module 74, wherein:

[0147] A first acquisition module 71 is used to acquire candidate text graph models and sample high-quality image sets;

[0148] A second acquisition module 72 is used to acquire a candidate user preference data set corresponding to the user set, and acquire a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set;

[0149] A construction module 73 is used to construct a sample image-text pair set corresponding to the sample high-quality preference image set, and obtain a target fine-tuning sample required for the candidate text-generated image model based on the sample preference image-text pair set;

[0150] The fine-tuning module 74 is used to perform model training on the candidate text graph model based on the target fine-tuning sample to obtain the target text graph model.

[0151] In the disclosed embodiment, the second acquisition module 72 is further used to: obtain the candidate user preference data of each user in the user set to obtain a candidate user preference data set composed of the preference data of each candidate user; for any sample high-quality image, obtain the target user preference data matching the sample high-quality image from the candidate user preference data set, and perform preference alignment between the target user preference data and the sample high-quality image to obtain a sample high-quality preference image corresponding to the sample high-quality image; based on the sample high-quality preference images of each sample high-quality image, obtain a sample high-quality preference image set.

[0152] In the disclosed embodiment, the construction module 73 is also used to: obtain the sample target description text of each sample high-quality preference image in the sample high-quality preference image set; obtain the sample image-text pairs of each sample high-quality preference image according to each sample high-quality preference image and the sample target description text of each sample high-quality preference image, so as to obtain a sample image-text pair set composed of each sample image-text pair; sample the sample image-text pair set to obtain the target fine-tuning samples required for the candidate text-generated graph model.

[0153] In the disclosed embodiment, the construction module 73 is also used to: obtain a multimodal large language model, and generate a sample initial description text of each sample high-quality preference image through the model capability of the multimodal large language model; perform description correction on each sample initial description text to obtain a sample candidate description text of each sample high-quality preference image; and perform preference text embedding on each sample candidate description text based on a candidate user preference data set through the model capability of the multimodal large language model to obtain a sample target description text of each sample high-quality preference image.

[0154] In the disclosed embodiment, the construction module 73 is also used to: for any sample high-quality preference image, obtain the target user preference text aligned with the sample high-quality preference image; through the model capability of the multimodal large language model, align the target user preference text to the sample candidate description text corresponding to the sample high-quality preference image, and obtain the sample target description text of the sample high-quality preference image.

[0155] In the disclosed embodiment, the construction module 73 is further used to: obtain a random sampling strategy for a set of sample image-text pairs; and sample the set of sample image-text pairs based on the random sampling strategy to obtain target fine-tuning samples required for the candidate text-image model.

[0156] In the disclosed embodiment, the construction module 73 is also used to: load the sample image-text pair set through a preset candidate data loader to obtain a loaded target data loader; poll and sample the sample image-text pair set in the target data loader based on a random sampling strategy to obtain a candidate fine-tuning sample set; and obtain the target fine-tuning sample required for the candidate text-graph model in the current fine-tuning round from the candidate fine-tuning sample set.

[0157] In the disclosed embodiment, the first acquisition module 71 is also used to: obtain a preset high-quality image screening strategy, wherein the high-quality image screening strategy is obtained based on at least a preset image aesthetic screening strategy, an image quality screening strategy, and an image style screening strategy; perform image screening on a pre-acquired source database based on the high-quality image screening strategy, and obtain a set of sample high-quality images that match the image screening strategy from the source database.

[0158] In the disclosed embodiment, the first acquisition module 71 is further used to: obtain a preset specific concept image screening strategy; obtain a sample specific concept image set that matches the specific concept image screening strategy in the sample high-quality image set; in response to the sample specific concept image set not satisfying the fine-tuning sample conditions of the candidate text graph model, perform specific concept image screening based on the specific concept image screening strategy to obtain a new sample specific concept image set, and update the sample high-quality image set based on the new sample specific concept image set to obtain a new sample high-quality image set.

[0159] In the disclosed embodiment, the fine-tuning module 74 is further used to: obtain the sample prompt text corresponding to the target fine-tuning sample, and input the sample prompt text into the candidate text graph model to obtain the output image generated by the candidate text graph model based on the sample prompt text; obtain the label image corresponding to the target fine-tuning sample, and obtain the fine-tuning loss of the candidate text graph model based on the output image and the label image; adjust the model parameters of the candidate text graph model based on the fine-tuning loss, and return to obtain the next target fine-tuning sample to continue fine-tuning the adjusted candidate text graph model until the fine-tuning is completed to obtain the target text graph model.

[0160] In the disclosed embodiment, the fine-tuning module 74 is further used to: obtain the early stopping strategy of the candidate text graph model and the corresponding early stopping strategy verification set; perform performance verification on the candidate text graph model in the model fine-tuning process based on the early stopping strategy verification set to obtain the performance verification result of the candidate text graph model; and identify whether the candidate text graph model triggers the early stopping strategy based on the performance verification result.

[0161] In the embodiment of the present disclosure, the fine-tuning module 74 is further configured to: in response to the performance verification result indicating that the performance optimization of the candidate text graph model is stagnant during the model fine-tuning process, determine that the candidate text graph model is identified to trigger an early stopping strategy.

[0162] In the disclosed embodiment, the fine-tuning module 74 is further configured to: in response to identifying that the candidate text graph model triggers the early stopping strategy, terminate the model fine-tuning of the candidate text graph model to obtain the target text graph model.

[0163] The training device of the text graph model proposed in the present invention obtains a candidate text graph model and a sample high-quality image set and a candidate user preference data set of a user set, obtains a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set, constructs a sample image-text pair set corresponding to the sample high-quality preference image set to obtain a target fine-tuning sample of the candidate text graph model, and fine-tunes the candidate text graph model through the target fine-tuning sample, thereby obtaining a fine-tuned target text graph model. In the present disclosure, a target fine-tuning sample is obtained based on a sample high-quality preference image set, and then a target text graph model is obtained by fine-tuning the model through the target fine-tuning sample, thereby improving the image quality of the image generated by the target text graph model and improving the degree of adaptation between the image generated by the target text graph model and the user preference. The sample high-quality image set introduces a candidate preference data set of a user set composed of multiple users to obtain a sample high-quality preference image set, so that the target text graph model realizes the preference alignment of multiple users, and there is no need to perform separate model training for each user among the multiple users, thereby reducing the acquisition cost of the text graph model. The model is fine-tuned based on the trained text graph model, thereby improving the model performance of the fine-tuned target text graph model. In the scenario of image generation through the target text graph model, the user experience is optimized and the user stickiness is improved.

[0164] An embodiment of the present disclosure further proposes an image generating device. Since the image generating device proposed in the embodiment of the present disclosure corresponds to the image generating methods proposed in the above-mentioned embodiments, the implementation methods of the above-mentioned image generating methods are also applicable to the image generating device proposed in the embodiment of the present disclosure and will not be described in detail in the following embodiments.

[0165] Figure 8 FIG. 1 is a schematic diagram of the structure of an image generating device according to an embodiment of the present disclosure. Figure 8 As shown, the image generating device 800 includes a third acquiring module 81, a fourth acquiring module 82 and an image generating module 83, wherein:

[0166] A third acquisition module 81 is used to acquire a finely tuned target document graph model, wherein the target document graph model is obtained based on the document graph model training device proposed in any one of claims 15 to 27;

[0167] The fourth acquisition module 82 is used to acquire the image generation requirement input by the user terminal, and obtain the corresponding image generation requirement prompt text based on the image generation requirement;

[0168] The image generation module 83 is used to input the image generation requirement prompt text into the target text-based graph model, and obtain the target image required by the user end through the model capability of the target text-based graph model.

[0169] The image generation device proposed in the present disclosure obtains the image generation requirement prompt text corresponding to the image generation requirement of the user end, and inputs the image generation requirement prompt text into the fine-tuned target text-based graph model, and generates the target image corresponding to the image generation requirement prompt text through the target text-based graph model. Figure 7 The training device of the text graph model proposed in the embodiment obtains the target text graph model, improves the degree of adaptation between the target image generated based on the target text graph model and the image generation requirements corresponding to the user end, improves the image generation quality of the target image, and optimizes the user's image generation experience.

[0170] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a readable storage medium, and a computer program product.

[0171] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0172] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 909 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0173] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 906, such as various types of displays, speakers, etc.; a storage unit 909, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0174] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method and / or image generation method of the Wensheng graph model. For example, in some embodiments, the training method and / or image generation method of the Wensheng graph model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 909. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method and / or image generation method of the Wensheng graph model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the training method and / or the image generation method of the text graph model in any other appropriate manner (for example, by means of firmware).

[0175] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0177] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0178] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).

[0179] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0180] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0181] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0182] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a text graph model, wherein: The method comprises: Obtain candidate text graph models and sample high-quality image sets; Acquire a candidate user preference data set corresponding to the user set, and acquire a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set; Constructing a sample image-text pair set corresponding to the sample high-quality preference image set, and obtaining a target fine-tuning sample required for the candidate text-generated image model based on the sample preference image-text pair set; The candidate document graph model is trained based on the target fine-tuning sample to obtain a target document graph model.

2. The method according to claim 1, wherein: The step of obtaining a candidate user preference data set corresponding to the user set, and obtaining a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set, includes: Acquire the candidate user preference data of each user in the user set to obtain the candidate user preference data set composed of the preference data of each candidate user; For any sample high-quality image, obtain the target user preference data that matches the sample high-quality image from the candidate user preference data set, and perform preference alignment between the target user preference data and the sample high-quality image to obtain a sample high-quality preference image corresponding to the sample high-quality image; Based on the sample high-quality preference images of each sample high-quality image, the sample high-quality preference image set is obtained.

3. The method according to claim 1, wherein: The step of constructing a sample image-text pair set corresponding to the sample high-quality preference image set, and obtaining a target fine-tuning sample required for the candidate text-generated image model based on the sample image-text pair set, includes: Obtaining a sample target description text of each sample high-quality preference image in the sample high-quality preference image set; According to each sample high-quality preference image and the sample target description text of each sample high-quality preference image, a sample image-text pair of each sample high-quality preference image is obtained to obtain the sample image-text pair set composed of each sample image-text pair; The sample image-text pair set is sampled to obtain the target fine-tuning sample required by the candidate text-image model.

4. The method according to claim 3, wherein: The obtaining of the sample target description text of each sample high-quality preference image in the sample high-quality preference image set includes: Acquire a multimodal large language model, and generate sample initial description text of each sample high-quality preference image through the model capability of the multimodal large language model; Perform description correction on the initial description text of each sample to obtain the sample candidate description text of the high-quality preference image of each sample; By utilizing the model capability of the multimodal large language model, preference text embedding is performed on each sample candidate description text based on the candidate user preference data set to obtain a sample target description text of each sample high-quality preference image.

5. The method according to claim 4, wherein: The method of embedding the preference text of each sample candidate description text based on the candidate user preference data set by using the model capability of the multimodal large language model to obtain the sample target description text of each sample high-quality preference image includes: For any sample high-quality preference image, obtaining the target user preference text aligned with the sample high-quality preference image; By utilizing the model capability of the multimodal large language model, the target user preference text is aligned to the sample candidate description text corresponding to the sample high-quality preference image to obtain the sample target description text of the sample high-quality preference image.

6. The method according to claim 3, wherein: The step of sampling the sample image-text pair set to obtain the target fine-tuning sample required by the candidate text-generated image model includes: Obtaining a random sampling strategy for the sample image-text pair set; The sample image-text pair set is sampled based on the random sampling strategy to obtain the target fine-tuning sample required by the candidate text-image model.

7. The method according to claim 6, wherein: The step of sampling the sample image-text pair set based on the random sampling strategy to obtain the target fine-tuning sample required by the candidate text-generated image model includes: Loading the sample image-text pair set through a preset candidate data loader to obtain a loaded target data loader; Based on the random sampling strategy, polling and sampling the sample image-text pair set in the target data loader to obtain a candidate fine-tuning sample set; The target fine-tuning samples required for the candidate text graph model in the current fine-tuning round are obtained from the candidate fine-tuning sample set.

8. The method according to claim 1, wherein: The step of obtaining a sample high-quality image set comprises: Acquire a preset high-quality image screening strategy, wherein the high-quality image screening strategy is obtained based on at least a preset image aesthetic screening strategy, an image quality screening strategy, and an image style screening strategy; The pre-acquired source database is screened for images based on the high-quality image screening strategy, and the sample high-quality image set matching the image screening strategy is acquired from the source database.

9. The method according to claim 8, wherein: The method further comprises: Get preset specific concept image screening strategies; Acquire a sample specific concept image set matched by the specific concept image screening strategy in the sample high-quality image set; In response to the sample specific concept image set not satisfying the fine-tuning sample condition of the candidate text graph model, specific concept image screening is performed based on the specific concept image screening strategy to obtain a new sample specific concept image set, and the sample high-quality image set is updated based on the new sample specific concept image set to obtain a new sample high-quality image set.

10. The method according to claim 1, wherein: The performing model training on the candidate document graph model based on the fine-tuning sample to obtain a target document graph model includes: Acquire a sample prompt text corresponding to the target fine-tuning sample, and input the sample prompt text into the candidate text-generated graph model to obtain an output image generated by the candidate text-generated graph model based on the sample prompt text; Acquire a label image corresponding to the target fine-tuning sample, and obtain a fine-tuning loss of the candidate text graph model based on the output image and the label image; The model parameters of the candidate text graph model are adjusted based on the fine-tuning loss, and the next target fine-tuning sample is returned to continue fine-tuning the adjusted candidate text graph model to obtain the target text graph model.

11. The method according to claim 1, wherein: The method further comprises: Obtaining an early stopping strategy of the candidate text graph model and a corresponding early stopping strategy verification set; Performing performance verification on the candidate Wensheng graph model in the model fine-tuning process based on the early stopping strategy verification set to obtain a performance verification result of the candidate Wensheng graph model; According to the performance verification result, it is identified whether the candidate text graph model triggers the early stopping strategy.

12. The method according to claim 11, wherein: The step of identifying, according to the performance verification result, whether the candidate document graph model triggers the early stopping strategy includes: In response to the performance verification result indicating that the candidate text graph model is stagnant in performance optimization during the model fine-tuning process, it is determined that the candidate text graph model is identified to trigger the early stopping strategy.

13. The method according to claim 12, wherein: The method further comprises: In response to identifying that the candidate text graph model triggers the early stopping strategy, model fine-tuning of the candidate text graph model is terminated to obtain the target text graph model.

14. A method for generating an image, wherein: The method comprises: Acquire a finely tuned target Wensheng graph model, wherein the target Wensheng graph model is obtained based on the training method of the Wensheng graph model proposed in any one of claims 1 to 13 above; Acquire an image generation requirement input by a user terminal, and obtain a corresponding image generation requirement prompt text based on the image generation requirement; The image generation requirement prompt text is input into the target text-based graph model, and the target image required by the user end is obtained through the model capability of the target text-based graph model.

15. A training device for a text graph model, wherein: The device comprises: The first acquisition module is used to acquire candidate text graph models and sample high-quality image sets; A second acquisition module is used to acquire a candidate user preference data set corresponding to the user set, and acquire a sample high-quality preference image set corresponding to the sample high-quality image set based on the candidate user preference data set; A construction module, used to construct a sample image-text pair set corresponding to the sample high-quality preference image set, and obtain a target fine-tuning sample required for the candidate text-generated image model based on the sample preference image-text pair set; The fine-tuning module is used to perform model training on the candidate text graph model based on the target fine-tuning sample to obtain a target text graph model.

16. The device according to claim 15, wherein: The second acquisition module is further used for: Acquire the candidate user preference data of each user in the user set to obtain the candidate user preference data set composed of the preference data of each candidate user; For any sample high-quality image, obtain the target user preference data that matches the sample high-quality image from the candidate user preference data set, and perform preference alignment between the target user preference data and the sample high-quality image to obtain a sample high-quality preference image corresponding to the sample high-quality image; Based on the sample high-quality preference images of each sample high-quality image, the sample high-quality preference image set is obtained.

17. The device according to claim 15, wherein: The building blocks are also used to: Obtaining a sample target description text of each sample high-quality preference image in the sample high-quality preference image set; According to each sample high-quality preference image and the sample target description text of each sample high-quality preference image, a sample image-text pair of each sample high-quality preference image is obtained to obtain the sample image-text pair set composed of each sample image-text pair; The sample image-text pair set is sampled to obtain the target fine-tuning sample required by the candidate text-image model.

18. The device according to claim 17, wherein: The building blocks are also used to: Acquire a multimodal large language model, and generate sample initial description text of each sample high-quality preference image through the model capability of the multimodal large language model; Perform description correction on the initial description text of each sample to obtain the sample candidate description text of the high-quality preference image of each sample; By utilizing the model capability of the multimodal large language model, preference text embedding is performed on each sample candidate description text based on the candidate user preference data set to obtain a sample target description text of each sample high-quality preference image.

19. The device according to claim 18, wherein: The building blocks are also used to: For any sample high-quality preference image, obtaining the target user preference text aligned with the sample high-quality preference image; By utilizing the model capability of the multimodal large language model, the target user preference text is aligned to the sample candidate description text corresponding to the sample high-quality preference image to obtain the sample target description text of the sample high-quality preference image.

20. The device according to claim 17, wherein: The building blocks are also used to: Obtaining a random sampling strategy for the sample image-text pair set; The sample image-text pair set is sampled based on the random sampling strategy to obtain the target fine-tuning sample required by the candidate text-image model.

21. The device according to claim 20, wherein: The building blocks are also used to: Loading the sample image-text pair set through a preset candidate data loader to obtain a loaded target data loader; Based on the random sampling strategy, polling and sampling the sample image-text pair set in the target data loader to obtain a candidate fine-tuning sample set; The target fine-tuning samples required for the candidate text graph model in the current fine-tuning round are obtained from the candidate fine-tuning sample set.

22. The device according to claim 15, wherein: The first acquisition module is further used for: Acquire a preset high-quality image screening strategy, wherein the high-quality image screening strategy is obtained based on at least a preset image aesthetic screening strategy, an image quality screening strategy, and an image style screening strategy; The pre-acquired source database is screened for images based on the high-quality image screening strategy, and the sample high-quality image set matching the image screening strategy is acquired from the source database.

23. The device according to claim 22, wherein: The first acquisition module is further used for: Get preset specific concept image screening strategies; Acquire a sample specific concept image set matched by the specific concept image screening strategy in the sample high-quality image set; In response to the sample specific concept image set not satisfying the fine-tuning sample condition of the candidate text graph model, specific concept image screening is performed based on the specific concept image screening strategy to obtain a new sample specific concept image set, and the sample high-quality image set is updated based on the new sample specific concept image set to obtain a new sample high-quality image set.

24. The device according to claim 15, wherein: The fine adjustment module is also used for: Acquire a sample prompt text corresponding to the target fine-tuning sample, and input the sample prompt text into the candidate text-generated graph model to obtain an output image generated by the candidate text-generated graph model based on the sample prompt text; Acquire a label image corresponding to the target fine-tuning sample, and obtain a fine-tuning loss of the candidate text graph model based on the output image and the label image; The model parameters of the candidate text graph model are adjusted based on the fine-tuning loss, and the next target fine-tuning sample is returned to continue fine-tuning the adjusted candidate text graph model to obtain the target text graph model.

25. The device according to claim 15, wherein: The fine adjustment module is also used for: Obtaining an early stopping strategy of the candidate text graph model and a corresponding early stopping strategy verification set; Performing performance verification on the candidate Wensheng graph model in the model fine-tuning process based on the early stopping strategy verification set to obtain a performance verification result of the candidate Wensheng graph model; According to the performance verification result, it is identified whether the candidate text graph model triggers the early stopping strategy.

26. The device according to claim 25, wherein The fine adjustment module is also used for: In response to the performance verification result indicating that the candidate text graph model is stagnant in performance optimization during the model fine-tuning process, it is determined that the candidate text graph model is identified to trigger the early stopping strategy.

27. The device according to claim 26, wherein: The fine adjustment module is also used for: In response to identifying that the candidate text graph model triggers the early stopping strategy, model fine-tuning of the candidate text graph model is terminated to obtain the target text graph model.

28. An image generating device, wherein: The device comprises: A third acquisition module is used to acquire a finely tuned target document graph model, wherein the target document graph model is obtained based on the document graph model training device proposed in any one of claims 15 to 27; A fourth acquisition module, used to acquire an image generation requirement input by a user terminal, and obtain a corresponding image generation requirement prompt text based on the image generation requirement; The image generation module is used to input the image generation requirement prompt text into the target text-generated graph model, and obtain the target image required by the user terminal through the model capability of the target text-generated graph model.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 13 and / or claim 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13 and / or claim 14.

31. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13 and / or claim 14.

Images