Text-graph conversion correction method and device and storage medium

Abstract

The invention provides a text-graph conversion correction method and device and a storage medium. The method comprises the steps of obtaining an original text and inputting the original text into a preset text-graph model to generate a visual image; inputting the visual image into a preset image-to-text model to obtain a contrast text; analyzing the original text and the contrast text to obtain a difference noun pair; extracting a corresponding image slice from the visual image based on the difference noun pair; correcting the image slices according to the difference noun pairs to generate corrected slices; and correcting the visual image through the image slice and the correction slice to obtain a target image. The image with the difference is subjected to slice correction in a text comparison mode, it can be effectively ensured that the final image accurately reflects the original text, and the accuracy of generating the image through the text is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a text-to-image conversion correction method, device and storage medium. Background Art

[0002] Text-generated image technology, also known as "text image", is an artificial intelligence technology that automatically generates corresponding images through natural language descriptions. This technology combines the two fields of natural language processing and computer vision, with the aim of enabling machines to understand text information and convert it into visual content. Text-generated image technology can generate high-quality images in a short period of time by combining different text descriptions and image features. It can also be adjusted and optimized according to different application scenarios. Compared with traditional manual design and image generation methods, text-generated image technology can significantly reduce manpower and time costs.

[0003] The existing text-to-image technology has the problem that the generated image is inconsistent with the input text, which is called hallucination. In order to reduce the occurrence of hallucinations and ensure that the model output follows the content of the input text, the existing technology mostly uses the pre-trained CLIP model as a text encoder. However, this method is mainly trained based on short texts. When processing long texts, especially dense texts describing multiple objects and their complex relationships, hallucinations still exist, resulting in inaccurate generated images. In particular, when the text describes multiple objects or contains detailed attributes, the generated image is often inconsistent with the original text. This hallucination problem is mainly manifested in object omissions, incorrect object positions, or confusion of object attributes, which seriously affects the accuracy of the generated results. Summary of the invention

[0004] The present application provides a text-to-image conversion correction method, device and storage medium, which are used to solve the technical problem of inaccurate image generation from text in the prior art.

[0005] In a first aspect, the present application provides a text-to-image conversion correction method, comprising:

[0006] Acquire original text and input the original text into a preset text-generated graph model to generate a visual image;

[0007] Inputting the visual image into a preset image-to-text model to obtain a reference text;

[0008] By analyzing the original text and the control text, different noun pairs are obtained;

[0009] extracting corresponding image slices from the visual image based on the difference noun pairs;

[0010] Correcting the image slice according to the difference noun pair to generate a corrected slice;

[0011] The visual image is corrected by the image slices and the correction slices to obtain a target image.

[0012] In a possible design, the visual image is input into a preset image-to-text model to obtain a reference text, including:

[0013] Inputting the visual image into a preset image-to-text model multiple times to obtain multiple text descriptions;

[0014] Input the text descriptions into a preset multimodal large language model to calculate the semantic entropy corresponding to each text description;

[0015] Extracting a preset number of text descriptions from the plurality of text descriptions as target texts according to the semantic entropy;

[0016] The target text is semantically determined and reorganized to obtain a control text.

[0017] In a possible design, the difference noun pairs are obtained by analyzing the original text and the control text, including:

[0018] Performing natural language processing on the original text and the comparison text to obtain a corresponding plurality of original nouns and a corresponding plurality of comparison nouns;

[0019] Pairing the original nouns with the control nouns to obtain a plurality of noun pairs;

[0020] By performing similarity analysis on the noun pairs, similarity data corresponding to each noun pair is obtained;

[0021] The noun pairs are screened according to the similarity data and a preset similarity threshold to obtain difference nouns, where the difference nouns are noun pairs whose similarity data is less than the similarity threshold.

[0022] In a possible design, extracting corresponding image slices from the visual image based on the difference noun pairs includes:

[0023] Extracting features from the difference noun pairs and the visual image to obtain text features and image features;

[0024] Performing enhanced fusion processing on the text features and the image features to obtain comprehensive image-text features;

[0025] Based on the comprehensive image-text feature, identifying the image boundary corresponding to the difference noun from the visual image;

[0026] In a possible design, the text feature and the image feature are enhanced and fused to obtain a comprehensive image-text feature, including:

[0027] The image features and the text features are enhanced through a self-attention mechanism;

[0028] The enhanced image features and text features are combined through a cross-attention mechanism to obtain comprehensive image and text features.

[0029] In a possible design, modifying the image slice according to the difference noun pair to generate a corrected slice includes:

[0030] extracting an original noun from the differential noun pair;

[0031] Inputting the original noun and the image slice into a preset multimodal large language model for comparative analysis to obtain a modified image feature;

[0032] The image slice is adjusted according to the modified image feature to obtain a corrected slice.

[0033] In a possible design, the visual image is corrected by the image slice and the correction slice to obtain a target image, including:

[0034] obtaining a correction region by comparing the image slice with the visual image;

[0035] Replacing the image in the correction area with the corrected slice to obtain a corrected image;

[0036] The corrected area in the corrected image is fused to obtain a target image.

[0037] In a second aspect, the present application provides a text-to-image conversion correction device, comprising:

[0038] An acquisition module, used for acquiring original text and inputting the original text into a preset text-generated graph model to generate a visual image;

[0039] A comparison module, used for inputting the visual image into a preset image-to-text model to obtain a comparison text;

[0040] An analysis module, used for obtaining difference noun pairs by analyzing the original text and the comparison text;

[0041] an extraction module, configured to extract corresponding image slices from the visual image based on the difference noun pairs;

[0042] An image module, used for modifying the image slice according to the difference noun pair to generate a corrected slice;

[0043] A correction module is used to correct the visual image through the image slice and the correction slice to obtain a target image.

[0044] In a possible design, the control module is specifically used to:

[0045] Inputting the visual image into a preset image-to-text model multiple times to obtain multiple text descriptions;

[0046] Input the text descriptions into a preset multimodal large language model to calculate the semantic entropy corresponding to each text description;

[0047] Extracting a preset number of text descriptions from the plurality of text descriptions as target texts according to the semantic entropy;

[0048] The target text is semantically determined and reorganized to obtain a control text.

[0049] In a possible design, the analysis module is specifically used to:

[0050] Performing natural language processing on the original text and the comparison text to obtain a corresponding plurality of original nouns and a corresponding plurality of comparison nouns;

[0051] Pairing the original nouns with the control nouns to obtain a plurality of noun pairs;

[0052] By performing similarity analysis on the noun pairs, similarity data corresponding to each noun pair is obtained;

[0053] The noun pairs are screened according to the similarity data and a preset similarity threshold to obtain difference nouns, where the difference nouns are noun pairs whose similarity data is less than the similarity threshold.

[0054] In a possible design, the extraction module is specifically used to:

[0055] Extracting features from the difference noun pairs and the visual image to obtain text features and image features;

[0056] Performing enhanced fusion processing on the text features and the image features to obtain comprehensive image-text features;

[0057] Based on the comprehensive image-text feature, identifying the image boundary corresponding to the difference noun from the visual image;

[0058] Corresponding image slices are extracted from the visual image according to the image boundaries.

[0059] In a possible design, the extraction module is further used to:

[0060] The image features and the text features are enhanced through a self-attention mechanism;

[0061] The enhanced image features and text features are combined through a cross-attention mechanism to obtain comprehensive image and text features.

[0062] In a possible design, the image module is specifically used to:

[0063] extracting an original noun from the differential noun pair;

[0064] Inputting the original noun and the image slice into a preset multimodal large language model for comparative analysis to obtain a modified image feature;

[0065] The image slice is adjusted according to the modified image feature to obtain a corrected slice.

[0066] In a possible design, the correction module is specifically used to:

[0067] obtaining a correction region by comparing the image slice with the visual image;

[0068] Replacing the image in the correction area with the corrected slice to obtain a corrected image;

[0069] The corrected area in the corrected image is fused to obtain a target image.

[0070] In a third aspect, the present application provides a text-to-image conversion and correction device, comprising: a processor, and a memory in communication with the processor;

[0071] Memory stores computer-executable instructions;

[0072] The processor executes the computer-executable instructions stored in the memory, so that the path query device accessing the network resource executes any one of the text-to-image conversion correction methods in the first aspect.

[0073] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the text-to-image conversion correction method as described in any one of the first aspects.

[0074] The text-to-image conversion correction method, device and storage medium provided by the present application generate a visual image by acquiring an original text and inputting the original text into a preset text-to-image model; inputting the visual image into a preset image-to-text model to obtain a control text; obtaining a difference noun pair by analyzing the original text and the control text; extracting corresponding image slices from the visual image based on the difference noun pair; correcting the image slices according to the difference noun pair to generate a correction slice; and correcting the image with differences by slicing the visual image using image slices and correction slices to obtain a target image, which can effectively ensure that the final image accurately reflects the original text and improve the accuracy of the image generated by the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0076] Figure 1 This is a flowchart of the text-to-image conversion correction method provided in the embodiment of the present application. Figure 1 ;

[0077] Figure 2 This is a flowchart of the text-to-image conversion correction method provided in the embodiment of the present application. Figure 2 ;

[0078] Figure 3 This is a flowchart of the text-to-image conversion correction method provided in the embodiment of the present application. Figure 3 ;

[0079] Figure 4 This is a flowchart of the text-to-image conversion correction method provided in the embodiment of the present application. Figure 4 ;

[0080] Figure 5 This is a flowchart of the text-to-image conversion correction method provided in the embodiment of the present application. Figure 5 ;

[0081] Figure 6 This is a flowchart of the text-to-image conversion correction method provided in the embodiment of the present application. Figure 6 ;

[0082] Figure 7 It is a structural schematic diagram of a text-to-image conversion and correction device provided in an embodiment of the present application;

[0083] Figure 8 It is a schematic diagram of the hardware structure of the text-to-image conversion and correction device provided in an embodiment of the present application.

[0084] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0085] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0087] Text-generated image technology, also known as "text image", is an artificial intelligence technology that automatically generates corresponding images through natural language descriptions. This technology combines the two fields of natural language processing and computer vision, with the aim of enabling machines to understand text information and convert it into visual content. Text-generated image technology can generate high-quality images in a short period of time by combining different text descriptions and image features. It can also be adjusted and optimized according to different application scenarios. Compared with traditional manual design and image generation methods, text-generated image technology can significantly reduce manpower and time costs.

[0088] The existing text-to-image technology has the problem that the generated image is inconsistent with the input text, which is called hallucination. In order to reduce the occurrence of hallucinations and ensure that the model output follows the content of the input text, the existing technology mostly uses the pre-trained CLIP model as a text encoder. However, this method is mainly trained based on short texts. When processing long texts, especially dense texts describing multiple objects and their complex relationships, hallucinations still exist, resulting in inaccurate generated images. In particular, when the text describes multiple objects or contains detailed attributes, the generated image is often inconsistent with the original text. This hallucination problem is mainly manifested in object omissions, incorrect object positions, or confusion of object attributes, which seriously affects the accuracy of the generated results.

[0089] The data transmission method provided in this application is intended to solve the above technical problems in the prior art.

[0090] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0091] Figure 1 Flowchart of the text-to-image conversion correction method according to an embodiment of the present invention Figure 1 ,like Figure 1 As shown, this embodiment provides a text-to-image conversion correction method, and the process includes the following steps:

[0092] Step S101: obtaining original text and inputting the original text into a preset text-generated graph model to generate a visual image.

[0093] Specifically, by using a pre-trained text-to-image model, the original text is used as input to generate a visual image. The preset text-to-image model can be based on deep learning, such as GANs (Generative Adversarial Networks) or other types of neural networks. Inputting the original text into the text-to-image model can generate an intuitive visual image, allowing the content expressed in the original text to be presented in the form of an image, which is convenient for subsequent analysis and processing.

[0094] Step S102: input the visual image into a preset image-to-text model to obtain a reference text.

[0095] Specifically, by using another pre-trained image-to-text model, the generated visual image is used as input to obtain a control text. The preset image-to-text model can be based on a deep learning model, such as a CNN-RNN structure. By using the image-to-text model to analyze the visual image, the generated control text describes the model's understanding of the image content, which is convenient for subsequent comparison to see whether there is an illusion.

[0096] Step S103: obtaining difference noun pairs by analyzing the original text and the comparison text.

[0097] Specifically, noun pairs are formed by finding inconsistencies or deviations that occur in the conversion process from text to image and then to text. The noun pairs reflect the potential inconsistency between the text description and the image content. The difference noun pairs are the parts that need to be corrected in the subsequent visual image.

[0098] Step S104: extracting corresponding image slices from the visual image based on the difference noun pairs.

[0099] Specifically, by combining the multimodal features of text and image, the image parts corresponding to the differential nouns are identified and extracted from the visual image, providing a basis for further image correction and optimization.

[0100] Step S105: modifying the image slices according to the difference nouns to generate corrected slices.

[0101] Specifically, by accurately adjusting and correcting the image slices, it is ensured that the generated corrected slices are consistent with the text description, thereby improving the accuracy of image generation and understanding.

[0102] Step S106: Correct the visual image by image slicing and correction slicing to obtain a target image.

[0103] Specifically, by integrating the corrected slices into the visual image, the visual consistency and accuracy of the final image is ensured.

[0104] The text-to-image conversion correction method provided in the embodiment of the present invention performs slice correction on images with differences by means of text comparison, which can effectively ensure that the final image accurately reflects the original text and improve the accuracy of the image generated by the text.

[0105] Figure 2 A flowchart of the text-to-image conversion correction method provided in the embodiment of the present application Figure 2 .like Figure 2 As shown, this embodiment describes in detail the process of inputting a visual image into a preset image-to-text model to obtain a reference text in the above embodiment, and the specific implementation method of the process includes the following steps:

[0106] Step S201: Input the visual image into the preset image-to-text model multiple times to obtain multiple text descriptions.

[0107] Specifically, the same visual image is input into the preset image-to-text model multiple times. The image-to-text model will generate multiple different text descriptions based on the image. The text descriptions will differ due to the randomness and diversity of the model.

[0108] Step S202: input the text descriptions into a preset multimodal large language model to calculate the semantic entropy corresponding to each text description.

[0109] Specifically, the big language model used here is an existing open source model. The big language model can calculate the corresponding semantic entropy by analyzing the semantic content of each text. Semantic entropy can reflect the uncertainty and diversity of information in the text description. A text description with high semantic entropy indicates that the text contains richer or more complex semantic information.

[0110] Step S203: extracting a preset number of text descriptions from the plurality of text descriptions as target texts according to the semantic entropy.

[0111] Specifically, by sorting the text descriptions according to semantic entropy, a preset number of text descriptions with higher semantic entropy are extracted from the sequence as target texts. Texts with high semantic entropy can better capture the details and complexity of the image.

[0112] Step S204: semantically determine and reorganize the target text to obtain a reference text.

[0113] Specifically, semantic judgment and reorganization of the target text can be carried out by using a large language model to further analyze the semantic structure of the text to identify key information, and the key information can be integrated and optimized to generate a more accurate and coherent comparison text.

[0114] The embodiment of the present invention improves the accuracy and consistency of text descriptions by extracting and optimizing the text description that best represents the image content from multiple generated text descriptions, helps reduce the problem of information mismatch, and provides a more reliable text basis for subsequent comparison and analysis.

[0115] Figure 3 A flowchart of the text-to-image conversion correction method provided in the embodiment of the present application Figure 3 .like Figure 3 As shown, this embodiment describes in detail the process of obtaining difference noun pairs by analyzing the original text and the comparison text in the above embodiment. The specific implementation method of the process includes the following steps:

[0116] Step S301: performing natural language processing on the original text and the comparison text to obtain a corresponding plurality of original nouns and a corresponding plurality of comparison nouns.

[0117] Specifically, natural language processing (NLP) is performed on the original text and the control text to extract the nouns therein. The noun extraction process can be achieved by using technologies such as named entity recognition (NER) or part-of-speech tagging (POS tagging). Through natural language processing technology, key noun components can be extracted from the text, which helps to clarify the specific concepts and objects involved in the original text and the control text.

[0118] Step S302: Pair the original nouns with the reference nouns to obtain a plurality of noun pairs.

[0119] Specifically, the pairing can be based on the position of the noun in the text, semantic similarity or other relevant features. Each noun pair contains an original noun and a control noun. By establishing noun pairs, a direct association relationship can be established between related nouns from two different texts, which facilitates the subsequent analysis of the differences between the nouns.

[0120] Step S303: Obtain similarity data corresponding to each noun pair by performing similarity analysis on the noun pairs.

[0121] Specifically, similarity analysis can be performed using word vectors, such as Word2Vec and GloVe, or embedding models, such as BERT (Bidirectional Encoder Representations from Transformers). The similarity value is usually between 0 and 1, and the higher the value, the more similar the semantics of the two nouns. By calculating the similarity, the similarity between noun pairs can be quantified, and their similarities or differences can be represented by specific data for more accurate judgment.

[0122] Step S304: screening the noun pairs according to the similarity data and a preset similarity threshold to obtain difference nouns, where the difference nouns are noun pairs whose similarity data is less than the similarity threshold.

[0123] Specifically, differential nouns refer to noun pairs whose similarity data is less than the similarity threshold, and these noun pairs have large differences between the original text and the control text. Differential nouns are parts that have undergone large changes or differences during the text conversion process and require special attention and processing for better improvement and optimization.

[0124] The embodiment of the present invention can effectively identify the difference nouns between the original text and the comparison text through noun location and similarity analysis. The difference nouns indicate potential problems in the text generation or understanding process, and help further analyze and optimize the semantic consistency between text and image.

[0125] Figure 4 A flowchart of the text-to-image conversion correction method provided in the embodiment of the present application Figure 4 .like Figure 4 As shown, this embodiment describes in detail the process of extracting corresponding image slices from visual images based on difference nouns in the above embodiment, and the specific implementation method of the process includes the following steps:

[0126] Step S401: extracting features from the difference noun pairs and visual images to obtain text features and image features.

[0127] Specifically, text feature extraction is performed on the differential noun pairs. This can be achieved by using natural language processing techniques, such as word embedding (such as BERT, Word2Vec), to obtain the semantic representation of nouns. Image feature extraction is performed on visual images. Convolutional neural networks (CNNs) are usually used to extract the visual features of images, which can represent information such as shape, color, texture, etc. in the image. Feature extraction is to obtain text-related features such as semantics contained in the differential noun pairs, as well as features of visual images in terms of content, structure, etc., so as to analyze and process them from different angles.

[0128] Step S402: Enhance and fuse the text features and image features to obtain comprehensive image and text features.

[0129] Specifically, the extracted text features and image features can be enhanced and fused through multimodal learning techniques, such as using attention mechanisms to emphasize relevant features. The purpose of feature fusion is to combine the information of text and image to generate a comprehensive multimodal feature representation that can better capture the relationship between text and image. Through fusion processing, the features of text and image can be organically combined, and the correlation and complementarity between the two can be comprehensively considered, making the subsequent analysis and operations based on these features more comprehensive and accurate.

[0130] Step S403: Identify image boundaries corresponding to differential nouns from the visual image based on the comprehensive image-text features.

[0131] Specifically, we can use target detection algorithms (such as YOLO and Faster R-CNN) to locate objects in the image that are related to the text description. By using the fused image and text features, we can more accurately locate the boundaries of specific areas related to the difference nouns in the visual image, thereby clarifying the image range that needs further processing.

[0132] Step S404: extracting corresponding image slices from the visual image according to the image boundary.

[0133] Specifically, after the image boundary is determined, the partial image slices corresponding to the difference nouns can be accurately captured from the visual image, so that subsequent slices can be corrected.

[0134] In some optional implementations, the above step S402 includes:

[0135] Step a1: Enhance image features and text features through the self-attention mechanism.

[0136] Specifically, the self-attention mechanism allows the model to focus more on the important parts of the features themselves, enhance the image features and text features separately, highlight the key information and relevance within each, and improve the expressiveness and quality of the features.

[0137] In step a2, the enhanced image features and text features are combined through the cross-attention mechanism to obtain comprehensive image and text features.

[0138] Specifically, the cross-attention mechanism can interact and associate the enhanced image features and text features, so that the image features and text features can be better integrated together, thus forming a comprehensive image-text feature that contains rich information of the image and text and is interrelated, providing a more comprehensive and in-depth basis for subsequent various analyses and processing based on this feature. The feature combination makes full use of the advantages of both image and text modalities to achieve more effective information integration and utilization.

[0139] The embodiments of the present invention combine the multimodal features of text and image to identify and extract image parts corresponding to differential nouns from visual images, thereby enhancing the model's ability to process complex semantic relationships and providing a basis for further image correction and optimization.

[0140] Figure 5 A flowchart of the text-to-image conversion correction method provided in the embodiment of the present application Figure 5 .like Figure 5 As shown, this embodiment describes in detail the process of correcting the image slices to generate the corrected slices according to the difference nouns in the above embodiment, and the specific implementation of the process includes the following steps:

[0141] Step S501: extracting original nouns from differential noun pairs.

[0142] Specifically, the original noun is obtained from the original text, represents a concept or entity that is different from the control text, and is the part of the original text that is inconsistent with the image content.

[0143] Step S502: input the original noun and the image slice into a preset multimodal large language model for comparative analysis to obtain modified image features.

[0144] Specifically, the multimodal large language model is an existing open source model that can process and understand the complex relationship between text and images, and identify features in image slices that do not match the original nouns through comparative analysis. The multimodal large language model can use the model's own multimodal understanding capabilities to analyze the relationship between text and images and generate corrected image features that point out the parts of the visual image that need to be adjusted to be consistent with the text description.

[0145] Step S503: adjusting the image slice according to the modified image feature to obtain a corrected slice.

[0146] Specifically, adjustment includes modifying the shape, color, position or other visual features of the object in the slice to ensure that it matches the description of the original noun. The corrected slice generated by adjustment is visually consistent with the original text description and eliminates any mismatch or hallucination phenomenon.

[0147] The embodiment of the present invention uses the capability of a multimodal large language model to accurately adjust and correct image slices, thereby ensuring that the generated corrected slices are consistent with the text description, thereby improving the accuracy of image generation and understanding.

[0148] Figure 6 A flowchart of the text-to-image conversion correction method provided in the embodiment of the present application Figure 6 .like Figure 6 As shown, this embodiment describes in detail the process of correcting the visual image by image slicing and correction slicing to obtain the target image in the above embodiment, and the specific implementation method of the process includes the following steps:

[0149] Step S601: Obtain a correction area by comparing the image slice with the visual image.

[0150] Specifically, by comparing image segmentation, edge detection, etc., the parts of the visual image that are inconsistent with the original text description are identified.

[0151] Step S602: replacing the image in the correction area with the corrected slice to obtain a corrected image.

[0152] Specifically, the correction slice is replaced into the identified correction area, ensuring that the correction slice accurately covers and replaces the inconsistent image portion.

[0153] Step S603: performing fusion processing on the corrected area in the corrected image to obtain a target image.

[0154] Specifically, the corrected area in the corrected image is fused to ensure a seamless visual transition. The fusion process includes: using a smoothing filter or gradient transition technology to eliminate the obvious boundaries of the slice edge; adjusting the hue, brightness and contrast of the slice to make it consistent with the surrounding image area; and ensuring that the texture of the slice matches the texture of the original image through texture matching to avoid visual inconsistency.

[0155] After the fusion process, the final target image is generated, which is visually consistent and accurately reflects the intention of the original text description.

[0156] The embodiment of the present invention integrates the corrected slices into the visual image by combining image processing and editing, thereby ensuring the visual consistency and accuracy of the final image.

[0157] Figure 7 The structure diagram of the text-to-image conversion correction device provided by the embodiment of the present invention is shown in FIG. Figure 7 As shown, the text-to-image conversion and correction device 70 includes: an acquisition module 701, a comparison module 702, an analysis module 703, an extraction module 704, an image module 705, and a correction module 706. Among them:

[0158] An acquisition module 701 is used to acquire original text and input the original text into a preset text-generated graph model to generate a visual image;

[0159] A comparison module 702, used for inputting the visual image into a preset image-to-text model to obtain a comparison text;

[0160] An analysis module 703 is used to obtain difference noun pairs by analyzing the original text and the comparison text;

[0161] An extraction module 704, for extracting corresponding image slices from the visual image based on the difference noun pairs;

[0162] An image module 705 is used to modify the image slices according to the difference nouns to generate corrected slices;

[0163] The correction module 706 is used to correct the visual image by image slicing and correction slicing to obtain a target image.

[0164] In one possible design, the control module 702 is specifically configured to:

[0165] Input the visual image into the preset image-to-text model multiple times to obtain multiple text descriptions;

[0166] Input the text descriptions into the preset multimodal large language model to calculate the semantic entropy corresponding to each text description;

[0167] Extracting a preset number of text descriptions from multiple text descriptions as target texts according to semantic entropy;

[0168] The target text is semantically judged and reorganized to obtain the control text.

[0169] In a possible design, the analysis module 703 is specifically configured to:

[0170] Performing natural language processing on the original text and the comparison text to obtain a corresponding plurality of original nouns and a corresponding plurality of comparison nouns;

[0171] Pairing the original nouns with the control nouns obtains multiple noun pairs;

[0172] By performing similarity analysis on the noun pairs, the similarity data corresponding to each noun pair is obtained;

[0173] The noun pairs are screened according to the similarity data and a preset similarity threshold to obtain difference nouns, where the difference nouns are noun pairs whose similarity data is less than the similarity threshold.

[0174] In one possible design, the extraction module 704 is specifically configured to:

[0175] Extract features from difference noun pairs and visual images to obtain text features and image features;

[0176] Enhance and fuse text features and image features to obtain comprehensive image and text features;

[0177] Identify the image boundaries corresponding to the difference nouns from the visual image based on the comprehensive features of the image and text;

[0178] Corresponding image slices are extracted from the visual image according to the image boundaries.

[0179] In one possible design, the extraction module 704 is further configured to:

[0180] Enhance image and text features through self-attention mechanism;

[0181] The enhanced image features and text features are combined through the cross-attention mechanism to obtain comprehensive image and text features.

[0182] In a possible design, the image module 705 is specifically configured to:

[0183] Extracting original nouns from differential noun pairs;

[0184] Input the original noun and the image slice into a preset multimodal large language model for comparative analysis to obtain the corrected image features;

[0185] The image slices are adjusted according to the corrected image features to obtain corrected slices.

[0186] In one possible design, the correction module 706 is specifically configured to:

[0187] The correction area is obtained by comparing the image slices with the visual image;

[0188] The corrected slice is used to replace the image in the correction area to obtain a corrected image;

[0189] The corrected area in the corrected image is fused to obtain the target image.

[0190] Figure 8 A schematic diagram of the hardware structure of the text-to-image conversion and correction device provided by an embodiment of the present invention, such as Figure 8 As shown, the text-to-image conversion and correction device 80 includes: at least one processor 801 and a memory 802. Optionally, the text-to-image conversion and correction device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0191] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above text-to-image conversion correction method.

[0192] The communication component 803 can exchange data with the server.

[0193] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0194] In the above Figure 8 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0195] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0196] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0197] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned text-to-image conversion correction method is implemented.

[0198] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0199] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0200] The division of the above units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0201] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0203] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0204] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0205] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0206] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0207] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0208] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0209] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

Claims

1. A text-to-image conversion correction method, characterized in that: include: Acquire original text and input the original text into a preset text-generated graph model to generate a visual image; Inputting the visual image into a preset image-to-text model to obtain a reference text; By analyzing the original text and the control text, different noun pairs are obtained; extracting corresponding image slices from the visual image based on the difference noun pairs; Correcting the image slice according to the difference noun pair to generate a corrected slice; The visual image is corrected by the image slices and the correction slices to obtain a target image.

2. The method according to claim 1, characterized in that Inputting the visual image into a preset image-to-text model to obtain a reference text includes: Inputting the visual image into a preset image-to-text model multiple times to obtain multiple text descriptions; Input the text descriptions into a preset multimodal large language model to calculate the semantic entropy corresponding to each text description; Extracting a preset number of text descriptions from the plurality of text descriptions as target texts according to the semantic entropy; The target text is semantically determined and reorganized to obtain a control text.

3. The method according to claim 1, characterized in that By analyzing the original text and the control text, the difference noun pairs are obtained, including: Performing natural language processing on the original text and the comparison text to obtain a corresponding plurality of original nouns and a corresponding plurality of comparison nouns; Pairing the original nouns with the control nouns to obtain a plurality of noun pairs; By performing similarity analysis on the noun pairs, similarity data corresponding to each noun pair is obtained; The noun pairs are screened according to the similarity data and a preset similarity threshold to obtain difference nouns, where the difference nouns are noun pairs whose similarity data is less than the similarity threshold.

4. The method according to claim 1, characterized in that: Extracting corresponding image slices from the visual image based on the difference noun pairs includes: Extracting features from the difference noun pairs and the visual image to obtain text features and image features; Performing enhanced fusion processing on the text features and the image features to obtain comprehensive image-text features; Based on the comprehensive image-text feature, identifying the image boundary corresponding to the difference noun from the visual image; Corresponding image slices are extracted from the visual image according to the image boundaries.

5. The method according to claim 4, characterized in that The text features and the image features are enhanced and fused to obtain comprehensive image and text features, including: The image features and the text features are enhanced through a self-attention mechanism; The enhanced image features and text features are combined through a cross-attention mechanism to obtain comprehensive image and text features.

6. The method according to claim 1, characterized in that Correcting the image slice according to the difference noun pair to generate a corrected slice includes: extracting an original noun from the differential noun pair; Inputting the original noun and the image slice into a preset multimodal large language model for comparative analysis to obtain a modified image feature; The image slice is adjusted according to the modified image feature to obtain a corrected slice.

7. The method according to claim 1, characterized in that Correcting the visual image by using the image slice and the correction slice to obtain a target image includes: obtaining a correction region by comparing the image slice with the visual image; Replacing the image in the correction area with the corrected slice to obtain a corrected image; The corrected area in the corrected image is fused to obtain a target image.

8. A text-image conversion and correction device, characterized in that: include: An acquisition module, used for acquiring original text and inputting the original text into a preset text-generated graph model to generate a visual image; A comparison module, used for inputting the visual image into a preset image-to-text model to obtain a comparison text; An analysis module, used for obtaining difference noun pairs by analyzing the original text and the comparison text; an extraction module, configured to extract corresponding image slices from the visual image based on the difference noun pairs; An image module, used for modifying the image slice according to the difference noun pair to generate a corrected slice; A correction module is used to correct the visual image through the image slice and the correction slice to obtain a target image.

9. A text-image conversion and correction device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images