Image generation method, device, electronic device and storage medium

Abstract

The present invention relates to the field of image processing technology, and provides an image generation method, device, electronic device and storage medium. The method includes: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and general knowledge respectively; performing knowledge retrieval from a domain knowledge base based on the domain knowledge to obtain a reference image; dynamically updating the domain knowledge base; generating a target image based on the reference image, general knowledge, and the routing weights corresponding to the domain knowledge and general knowledge respectively, where the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image. By utilizing the advantages of dynamic knowledge retrieval for knowledge retrieval in a specific domain, it can provide a reference for the generation of knowledge in a specific domain. At the same time, by means of a dynamic routing strategy for image generation, it not only ensures the accuracy of the generated image, but also enhances the diversity and detail richness.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image generation method, apparatus, electronic device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, text-to-image generation technology has become an important tool in multiple fields such as image content creation, advertising design, education, and scientific research. The core advantage of such generation models lies in the ability to automatically generate corresponding images according to the text description input by users, thereby improving the efficiency and creative space of image creation.

[0003] However, current text-to-image generation technologies rely on large-scale general training data to ensure generation diversity and generality. This training method limits the generation ability of the model when dealing with knowledge in specific fields. Specifically, since most of the training data comes from text-image pairs in general scenarios, it is difficult for the model to accurately capture and convey professional concepts and details when generating images in specific fields, resulting in insufficient accuracy and richness of the generated results.

[0004] In addition, with the accelerating speed of knowledge update, knowledge concepts in some rapidly changing or emerging fields are often not fully covered by the model, which further exacerbates the limitations of the model when generating images in these fields, affecting the accuracy and diversity of the generated results. Summary of the Invention

[0005] The present invention provides an image generation method, apparatus, electronic device, and storage medium to solve the defects in the prior art that there are limitations in generating images in specific fields, and the accuracy and richness of the generated images are insufficient, and to achieve fast and accurate image generation and ensure the diversity and rich details of the generated images.

[0006] The present invention provides an image generation method, including:

[0007] Obtain an image description text, and determine the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and the general knowledge respectively;

[0008] Based on the domain knowledge, perform knowledge retrieval from a domain knowledge base to obtain a reference image; the domain knowledge base is dynamically updated;

[0009] Based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, perform image generation to obtain a generated target image;

[0010] Wherein, the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0011] An image generation method provided by the present invention, which generates a target image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, includes:

[0012] Extract features from the reference image, and determine reference generated image features based on the reference image features obtained by feature extraction;

[0013] Extract features from the general knowledge, and determine text generated image features based on the general text features obtained by feature extraction;

[0014] Generate an image based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, to obtain the target image.

[0015] An image generation method provided by the present invention, which generates a target image based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, includes:

[0016] Perform feature interaction based on the reference image features and the reference generated image features to obtain first generated image features;

[0017] Perform feature interaction based on the general text features and the text generated image features to obtain second generated image features;

[0018] Generate an image based on the first generated image features and the second generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, to obtain the target image.

[0019] An image generation method provided by the present invention, which performs feature interaction based on the reference image features and the reference generated image features to obtain first generated image features; and performs feature interaction based on the general text features and the text generated image features to obtain second generated image features, includes:

[0020] Perform cross-attention interaction based on the reference image features and the reference generated image features to obtain first generated image features;

[0021] Perform cross - attention interaction based on the general text features and the text - generated image features, perform self - attention interaction based on the text - generated image features, and perform feature fusion based on the features obtained from the self - attention interaction and the cross - attention interaction to obtain the second generated image features.

[0022] According to an image generation method provided by the present invention, the generating the target image based on the first generated image features, the second generated image features, and the routing weights respectively corresponding to the domain knowledge and the general knowledge includes:

[0023] Based on the routing weights, perform weighted fusion on the first generated image features and the second generated image features to obtain generated image features;

[0024] Based on the generated image features, perform image generation to obtain the target image.

[0025] According to an image generation method provided by the present invention, the determining the domain knowledge and general knowledge in the image description text, and the routing weights respectively corresponding to the domain knowledge and the general knowledge includes:

[0026] Perform text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text;

[0027] Based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, determine the routing weights respectively corresponding to the domain knowledge and the general knowledge.

[0028] According to an image generation method provided by the present invention, the domain knowledge base contains multiple domain knowledge pairs, and each domain knowledge pair contains a domain knowledge text and a corresponding domain knowledge image;

[0029] The performing knowledge retrieval from the domain knowledge base based on the domain knowledge to obtain a reference image includes:

[0030] Perform text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when arranged in descending order of text similarity and / or semantic similarity with the domain knowledge;

[0031] Determine the target domain knowledge pair where the target knowledge text is located in the domain knowledge base, and use the domain knowledge image in the target domain knowledge pair as the reference image.

[0032] The present invention also provides an image generation device, including:

[0033] A text parsing unit, configured to obtain an image description text, and determine domain knowledge and general knowledge in the image description text, as well as routing weights corresponding to the domain knowledge and the general knowledge respectively;

[0034] A knowledge retrieval unit, configured to perform knowledge retrieval from a domain knowledge base based on the domain knowledge to obtain a reference image; the domain knowledge base is dynamically updated;

[0035] An image generation unit, configured to generate a target image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively;

[0036] Wherein, the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, the image generation method as described in any one of the above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the image generation method as described in any one of the above is implemented.

[0039] The image generation method, device, electronic device, and storage medium provided by the present invention perform knowledge retrieval from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used for knowledge retrieval in a specific domain, which can provide a reference for the generation of knowledge in a specific domain. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image accurately reflects the knowledge in a specific domain, guarantee timeliness and text consistency, and can flexibly respond to different knowledge requirements, not only ensuring the accuracy of the generated image, but also enhancing diversity and detail richness. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 It is a schematic flowchart of the image generation method provided by the present invention;

[0042] Figure 2It is the overall flowchart of the image generation method provided by the present invention;

[0043] Figure 3 It is the structural schematic diagram of the image generation device provided by the present invention;

[0044] Figure 4 It is the structural schematic diagram of the electronic device provided by the present invention. Detailed implementation manners

[0045] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] Currently, text-to-image generation technology mainly relies on large-scale general training data to ensure the diversity and generality of generation. Common generation schemes include text-to-image generation methods based on a single model, such as DALL-E, Stable Diffusion, etc. Such schemes convert the input text description into the corresponding image through a single generation model and perform excellently in general scenarios. However, in the generation of knowledge in professional fields, it is limited by the coverage of the training data. That is, since the training data is mostly text-image data in general scenarios and the coverage is limited, when the model processes knowledge in specific fields, especially in complex and highly professional fields such as medicine and mechanical engineering, its generation ability will be limited. This limitation of knowledge stems from the lack of targeted knowledge retrieval and dynamic update mechanisms in existing models. Especially when it comes to knowledge in some rapidly changing or emerging fields, the accuracy and richness of the model generation are particularly insufficient. Therefore, combining knowledge retrieval and generation enhancement to better support multi-field and high-precision knowledge generation has become a major research hotspot in the current text-to-image generation field.

[0047] To address this challenge, in recent years, knowledge enhancement means such as introducing knowledge graphs and external knowledge bases into the generation model have been proposed, hoping to make the generation model perform better in the generation of specific knowledge through knowledge expansion and supplementation. Such enhancement schemes have improved the professionalism of the model to a certain extent and improved the generation effect in specific fields. However, due to the untimely update of the knowledge base or the single knowledge embedding method, the generation results are still inaccurate and incomplete. In addition, if overly relying on real images in the knowledge base, it will seriously affect the imagination of the model in image generation, resulting in the final generated images being affected by the retrieved content and the richness decreasing.

[0048] In this regard, the present invention provides an image generation method, which aims to utilize the advantages of dynamic knowledge retrieval to solve the current difficulties in knowledge generation in specific fields. At the same time, with the help of dynamic routing strategies, it is possible to flexibly respond to different knowledge requirements in the image generation process, which not only ensures the accuracy of the generated images, but also enhances the diversity and richness of details. Figure 1 It is a flow chart of the image generation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0049] Step 110, obtaining the image description text, and determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and general knowledge respectively; wherein the routing weights are used to reflect the presentation degree of the corresponding knowledge on the target image;

[0050] Step 120, based on the domain knowledge, knowledge retrieval is performed from the domain knowledge base to obtain a reference image; the domain knowledge base is dynamically updated;

[0051] Step 130 , performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge, to obtain a generated target image.

[0052] Specifically, considering the limitations of current text-image generation schemes in knowledge coverage and image generation, that is, the model shows generation missing / bias when processing knowledge concepts outside the training data, especially in the generation of knowledge in a specific field, it is difficult to accurately convey professional concepts and details, and when it comes to knowledge in rapidly changing or emerging fields, the generation quality and diversity are low, and the generation results are difficult to fully and accurately present the diverse information contained in the text. In an embodiment of the present invention, it is proposed that image generation can be performed based on dynamic knowledge retrieval, so that knowledge retrieval can be performed through a domain knowledge base that is maintained and dynamically updated in real time, and accurate and real-time reference knowledge in a specific field can be obtained. On this basis, weight distribution is performed with the help of a dynamic routing mechanism, so that the final generated image can meet the needs, which can not only accurately reflect the knowledge in a specific field, but also satisfy the overall image description.

[0053] It is understandable that in the actual application process, before generating an image and performing the text-to-image generation task, the first thing to determine is the description of the image to be generated, that is, the image description text. The image to be generated described by the image description text here can be an image in the general field, such as a street view image, a portrait image, etc., or an image in a specific field, such as a human body structure diagram in the medical field, an instrument structure diagram in the field of mechanical engineering, etc. The embodiments of the present invention do not make specific limitations in this regard. Considering that the current text-to-image generation technology has achieved good results in the general field but has limitations in the specific field, in the embodiments of the present invention, the text-to-image generation is mainly targeted at the specific field, and the image description text can be the description of an image in the specific field, such as "generate an image containing the general structure and internal structure of the human heart, requiring vividness and rich colors".

[0054] After determining the image description text, in the embodiments of the present invention, the image description text can be parsed to identify the knowledge in the general field and the knowledge in the specific field contained therein; that is, the image description text can be accurately parsed to distinguish the general knowledge and the domain knowledge therein. Here, specifically, the powerful learning and understanding capabilities of the large language model can be used to identify the knowledge in the specific field and the knowledge in the general field, and the information obtained by recognition can be regularized into the domain knowledge corresponding to the specific field and the general knowledge corresponding to the general field through natural language processing means.

[0055] Furthermore, after determining the domain knowledge and general knowledge in the image description text, in the embodiments of the present invention, a weight evaluation is also required to determine the impact of the domain knowledge on image generation, so as to obtain the routing weights corresponding to the domain knowledge and general knowledge respectively. Specifically, due to the different information contained in different image description texts, as well as the different proportions and importance of the domain knowledge and general knowledge therein, in order to perform image generation better and more accurately for the image description text, in the embodiments of the present invention, after determining the domain knowledge and general knowledge, it is also necessary to evaluate their guiding role and importance proportion in the current image generation task, and thus the routing weights corresponding to the domain knowledge and general knowledge can be obtained respectively.

[0056] Here, the process of weight evaluation can also be implemented with the help of large language models. Specifically, it can evaluate the impact of domain knowledge and general knowledge on the image generation task based on the relevance, importance of domain knowledge, and the degree of matching with other information in the image description text, so as to obtain their respective routing weights. This weight determines the prominence of domain knowledge in the generated image during subsequent image generation. That is, the routing weight can better reflect the importance / influence of the corresponding knowledge in the image generation process and its presentation on the finally generated image. Generating images in this way can make the generated images accurately reflect domain knowledge and maintain overall consistency with the original image description text.

[0057] After parsing the image description text to obtain domain knowledge and general knowledge, for domain knowledge, in the embodiments of the present invention, further knowledge retrieval can be performed to ensure the timeliness and integrity of knowledge generation in this specific field. That is, considering the limited knowledge coverage of the current model and the limitations in knowledge generation in a specific field, in the embodiments of the present invention, it is proposed that after determining domain knowledge, retrieval enhancement means can be introduced to perform knowledge retrieval on this domain knowledge, so as to retrieve from the domain knowledge base corresponding to this specific field to obtain an image corresponding to this domain knowledge, and subsequent image generation tasks can be carried out with this image as a reference, thereby obtaining the finally generated image.

[0058] Specifically, here it can be based on domain knowledge to perform knowledge retrieval in the domain knowledge base under this specific field to retrieve an image associated with and matching this domain knowledge as the reference image corresponding to this domain knowledge for subsequent image generation.

[0059] Among them, the domain knowledge base under a specific field contains a large number of organized and structured knowledge images and their corresponding text information. Therefore, when performing knowledge retrieval, it can be guided by domain knowledge to retrieve the matching text information, and thus obtain the corresponding knowledge image based on this, and determine the reference image corresponding to domain knowledge on the basis of this knowledge image.

[0060] Moreover, it is worth noting that the domain knowledge base here is maintained in real time and updated dynamically. Its content can be dynamically updated according to domain requirements and user feedback to maintain the timeliness and integrity of the data in the library. Here, the content in the dynamically updated domain knowledge base can accumulate and update over time. This dynamic update mechanism can keep the content in the library at the forefront of technology and in sync with the latest knowledge, so that the finally generated images have better accuracy, timeliness, and rich details. Here, there are various ways to dynamically update the domain knowledge base, such as the knowledge update and maintenance method based on the RAG (Retrieval-Augmented Generation) model, which can achieve the dynamic update of a large-scale knowledge base while ensuring the quality of the content in the domain knowledge base.

[0061] In the embodiments of the present invention, a domain knowledge base maintained in real time is constructed, and knowledge retrieval in a specific domain is carried out based on this, which can provide a reference for the generation of knowledge in a specific domain, ensure that the generated images can accurately reflect the knowledge or concepts in a specific domain, and guarantee timeliness and knowledge synchronization.

[0062] After that, the image generation can be carried out according to the retrieved reference images, general knowledge, and the routing weights corresponding to the domain knowledge and general knowledge respectively, so as to obtain the image to be generated, that is, the target image corresponding to the image description text. Here, specifically, based on the knowledge content in the specific domain represented by the reference image, the general knowledge and the routing weights corresponding to the two are combined for image generation, so as to provide the precise visual features of the specific domain knowledge content required for image generation through the reference image, provide the relevant information of the image description text through the general knowledge, and combine the importance / influence of the two in the image generation process represented by the routing weights and the presentation degree on the generated image to generate the final target image.

[0063] Specifically, since the routing weight can represent the influence intensity of the domain knowledge and general text on the target image, when generating an image based on the routing weight, if the routing weight of the domain knowledge is higher, the generated target image will be more biased towards the domain knowledge, and when the weight tends to the general knowledge, the generated target image will be more in line with the description of the general knowledge. In this way, the dynamic adjustment of the target image between the domain knowledge and general knowledge is realized, so that the generated target image can not only accurately reflect the domain knowledge but also conform to the overall image description.

[0064] The image generation method provided by the present invention retrieves knowledge from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image. The domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific domain, which can provide a reference for the generation of knowledge in a specific domain. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image can accurately reflect the knowledge in a specific domain, ensure timeliness and text consistency, and can flexibly respond to different knowledge requirements, not only ensuring the accuracy of the generated image, but also enhancing diversity and detail richness.

[0065] Based on the above embodiments, step 130 includes:

[0066] Extract features from the reference image, and determine the reference generated image features based on the reference image features obtained by feature extraction;

[0067] Extract features from the general knowledge, and determine the text generated image features based on the general text features obtained by feature extraction;

[0068] Based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, perform image generation to obtain the target image.

[0069] Specifically, the process of performing image generation according to the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the generated target image may specifically include the following steps:

[0070] In detail, the image generation process in the embodiments of the present invention is implemented using a dual-expert model architecture. Specifically, two expert models are introduced, namely an image reference generation model and a text reference generation model, which are used to process the knowledge images and general texts in a specific domain respectively, so as to accurately generate the knowledge in a specific domain through the image reference generation model, and at the same time use the text reference generation model to ensure the overall quality and diversity of the generated images.

[0071] Specifically, here, the general knowledge can be processed first to extract the knowledge features helpful for image generation from it, so as to obtain the text features of the general knowledge, that is, the general text features. Here, specifically, the general knowledge can be feature-extracted through the text reference generation model to obtain the general text features; then, based on this general text feature, feature construction can be performed to construct the feature representation of the image to be generated, that is, this general text feature can be used as the initial image representation, and feature construction can be gradually performed from this. In each step, new image features are generated according to the current image representation and the general text features, and finally the feature representation of the image to be generated, that is, the text generated image features, can be obtained.

[0072] Meanwhile, the reference image can be processed to extract knowledge features in a specific field that are helpful for image generation, so as to obtain the image features of the reference image, that is, the reference image features. Here, specifically, the reference image features can be obtained by extracting features from the reference image through an image reference generation model. Similarly, based on the reference image features, feature construction can be performed to construct the feature representation of the image to be generated, that is, the reference generated image features.

[0073] It should be noted that to ensure the accuracy of image generation, both expert models in the embodiments of the present invention can be pre-trained through corresponding sample data. That is, on the basis of the current text-to-image expert model and image-to-image expert model, two initial models can be constructed respectively, and then they are trained using text data, image data, etc. to obtain two expert models with better image generation capabilities.

[0074] After that, image generation can be performed according to the text-generated image features and the reference generated image features to obtain the target image. That is, based on the reference generated image features and the text-generated image features, image generation can be performed by combining the routing weights corresponding to the domain knowledge and the general knowledge respectively, so as to ensure that the generated image can accurately reflect the knowledge in a specific field while conforming to the overall image description to obtain the final target image.

[0075] In the embodiments of the present invention, a dual-expert model is used to process the general knowledge and the reference image respectively, and on this basis, image generation is performed by combining the routing weights, which can achieve the dynamic balance between the domain knowledge and the general knowledge, thereby ensuring the accuracy and richness of the generated image.

[0076] Based on the above embodiments, image generation is performed based on the reference image features and the reference generated image features, the general text features and the text-generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the target image, including:

[0077] Performing feature interaction based on the reference image features and the reference generated image features to obtain the first generated image features;

[0078] Performing feature interaction based on the general text features and the text-generated image features to obtain the second generated image features;

[0079] Performing image generation based on the first generated image features, the second generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the target image.

[0080] Specifically, the process of generating a target image according to the routing weights corresponding to the reference image features, reference generated image features, general text features, text generated image features, as well as domain knowledge and general knowledge may specifically include:

[0081] First, the reference image features and reference generated image features can be interacted to capture the correlation between them, thereby obtaining new features, which are herein referred to as the first generated image features. Herein, specifically, an attention mechanism can be adopted to perform cross-sequence interaction and feature fusion on the reference generated image features and reference image features, thereby obtaining the first generated image features.

[0082] Meanwhile, the general text features and text generated image features can be interacted to also capture the correlation between them, thereby obtaining new features, namely the second generated image features. Herein, specifically, an attention mechanism can be introduced to perform feature interaction on the text generated image features and general text features. The feature interaction herein may include cross-modal interaction, self-attention interaction in the image modality, etc. Finally, the second generated image features can be obtained through feature fusion.

[0083] After that, image generation can be performed according to the routing weights corresponding to the first generated image features, second generated image features, as well as domain knowledge and general knowledge, thereby obtaining the target image. That is, based on the first generated image features and second generated image features obtained through feature interaction, image generation is performed in combination with the routing weights to ensure that the target image can not only accurately reflect the knowledge of a specific domain but also conform to the overall image description.

[0084] Based on the above embodiments, feature interaction is performed based on the reference image features and reference generated image features to obtain the first generated image features; feature interaction is performed based on the general text features and text generated image features to obtain the second generated image features, including:

[0085] Cross-attention interaction is performed based on the reference image features and reference generated image features to obtain the first generated image features;

[0086] Cross-attention interaction is performed based on the general text features and text generated image features, self-attention interaction is performed based on the text generated image features, and feature fusion is performed based on the features obtained from the self-attention interaction and cross-attention interaction to obtain the second generated image features.

[0087] Specifically, the feature interaction for the reference image features and the reference generated image features described above adopts the method of interactive attention, that is, cross-attention interaction is performed on the reference image features and the reference generated image features to achieve interaction between two different sequences. Through the similarity between the query, key, and value of the sequences, the attention of the current sequence to other sequences is dynamically adjusted, so as to realize the comprehensive utilization of knowledge. Finally, new features after interaction can be obtained, that is, the first generated image features.

[0088] Correspondingly, when performing feature interaction on the general text features and the text-generated image features, there are two interaction methods, namely self-attention interaction and cross-attention interaction. That is, cross-attention interaction can be performed on the general text features and the text-generated image features, and self-attention interaction can be performed on the text-generated image features. And feature fusion can be performed on the features obtained by cross-attention interaction and the features obtained by self-attention interaction to obtain the second generated image features. Here, different modal information can be noticed through cross-attention interaction, and self-information can be noticed more carefully through self-attention interaction, so that the finally generated image is more accurate and richer in details.

[0089] In the embodiment of the present invention, the self-attention and cross-attention mechanisms are introduced and applied to text and image features, which can enable the image generation process to better capture global context information and realize the comprehensive utilization of multi-modal information, so that the finally generated target image matches the image description.

[0090] Based on the above embodiments, image generation is performed based on the first generated image features, the second generated image features, and the routing weights to obtain a target image, including:

[0091] Based on the routing weights, weighted fusion is performed on the first generated image features and the second generated image features to obtain generated image features;

[0092] Based on the generated image features, image generation is performed to obtain a target image.

[0093] Specifically, the process of generating a target image according to the first generated image features, the second generated image features, and the routing weights described above is actually to first perform weighted fusion at the feature level to obtain features for image generation, that is, generated image features, and then perform image generation based on this to obtain the final target image.

[0094] Specifically, here, the first generated image feature and the second generated image feature are weighted and fused according to the routing weights, so as to obtain the fused feature, that is, the generated image feature. That is, weighting is performed according to the routing weight corresponding to the first generated image feature and the domain knowledge, weighting is performed according to the routing weight corresponding to the second generated image feature and the general knowledge, and the two are compared and fused / added to obtain the generated image feature.

[0095] Immediately afterwards, the target image can be generated based on the generated image feature, and the target image is obtained.

[0096] It should be noted that the sum of the routing weight corresponding to the domain knowledge and the routing weight corresponding to the general knowledge is 1.

[0097] Based on the above embodiments, the process of weighted fusion of the first generated image feature and the second generated image feature can be represented by the following formula:

[0098]

[0099]

[0100] In the formula, and are respectively the routing weight corresponding to the domain knowledge and the routing weight corresponding to the general knowledge; and are the first generated image feature and the second generated image feature, is the generated image feature.

[0101] Based on the above embodiments, determining the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and general knowledge respectively, includes:

[0102] Performing text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text;

[0103] Based on the correlation between the domain knowledge and the image description text, as well as the matching degree between the domain knowledge and the general knowledge, determining the routing weights corresponding to the domain knowledge and general knowledge respectively.

[0104] Specifically, the process of the domain knowledge and general knowledge in the above image description text, as well as the routing weights corresponding to the domain knowledge and general knowledge respectively, may specifically include:

[0105] First, the image description text can be parsed. Specifically, a large language model can be used to parse the input image description text to accurately identify the general knowledge and domain knowledge therein. That is, by leveraging the powerful learning and understanding ability of the large language model, the knowledge in a specific domain and the knowledge in the general domain are identified, and the information obtained through natural language processing means is regularized into the domain knowledge corresponding to the specific domain and the general knowledge corresponding to the general domain.

[0106] Next, the influence degree of this domain knowledge and general knowledge on the image generation process can be evaluated. The basis for evaluation here can be the importance of the domain knowledge, the matching degree with other information, the correlation degree, etc. Based on this evaluation, the routing weights corresponding to the domain knowledge and general knowledge can be obtained. Here, specifically, according to the correlation between the domain knowledge and the image description text, as well as the matching degree between the domain knowledge and the general knowledge, the importance / influence of the domain knowledge in the image generation process is measured, and thus the routing weight corresponding to the domain knowledge can be determined. And according to the normalization principle, the routing weight of the corresponding general knowledge can be directly determined.

[0107] Here, according to the correlation of the domain knowledge and the matching degree with other information, the influence of this domain knowledge in the image generation process is dynamically calculated to determine the routing weight. This weight determines the prominence of the domain knowledge on the generated target image in the subsequent image generation process, so as to ensure that the finally generated target image can accurately reflect the knowledge of the specific domain and maintain the overall consistency with the original image description.

[0108] In the embodiment of the present invention, a routing mechanism based on a large language model is adopted. The image description text is parsed by the large language model to separate the domain knowledge and the general knowledge, and the corresponding routing weights are evaluated to participate in the subsequent image generation process, so as to ensure that the influence weights of the general knowledge and the domain knowledge can be dynamically allocated in different image generation tasks.

[0109] Based on the above embodiment, the domain knowledge base contains multiple domain knowledge pairs, and each domain knowledge pair contains a domain knowledge text and the corresponding domain knowledge image;

[0110] Step 120 includes:

[0111] Perform text retrieval in the domain knowledge base to obtain the target knowledge text; the target knowledge text is the first domain knowledge text when arranged in descending order according to the text similarity and / or semantic similarity with the domain knowledge;

[0112] Determine the target domain knowledge pair where the target knowledge text is located in the domain knowledge base, and use the domain knowledge image in the target domain knowledge pair as the reference image.

[0113] Specifically, the process of retrieving knowledge from the domain knowledge base according to the domain knowledge to obtain the reference image may specifically include:

[0114] First, retrieve in the domain knowledge base according to the domain knowledge to find text information that has a certain similarity with the domain knowledge at the text level and / or semantic level, that is, the domain knowledge text, and the retrieved domain knowledge texts can be sorted according to the similarity degree, so as to obtain a sequence of matching domain knowledge texts; that is, the retrieved domain knowledge texts can be sorted in descending or ascending order according to the text similarity and / or semantic similarity to obtain a sequence of domain knowledge texts, and the first or last domain knowledge text in this sequence can be selected as the target knowledge text. It can also be understood that the domain knowledge text most similar to the domain knowledge is selected as the target knowledge text.

[0115] Subsequently, according to the matching relationship between the domain knowledge text in the domain knowledge base and the knowledge image (domain knowledge image), the domain knowledge image corresponding to the target knowledge text can be determined, that is, first determine the domain knowledge pair corresponding to the target knowledge text in the domain knowledge base, that is, the target domain knowledge pair, then determine the domain knowledge image in this target domain knowledge pair, and use this domain knowledge image as the reference image corresponding to the domain knowledge.

[0116] Figure 2 It is the overall flowchart of the image generation method provided by the present invention, as Figure 2 shown, the method includes:

[0117] First, obtain the image description text.

[0118] Next, perform text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text, and determine the routing weights corresponding to the domain knowledge and general knowledge respectively based on the relevance between the domain knowledge and the image description text and the matching degree between the domain knowledge and the general knowledge; the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0119] Subsequently, perform text retrieval in the domain knowledge base to obtain the target knowledge text; the domain knowledge base contains multiple domain knowledge pairs, and each domain knowledge pair contains a domain knowledge text and the corresponding domain knowledge image; the target knowledge text is the first domain knowledge text when arranged in descending order according to the text similarity and / or semantic similarity with the domain knowledge; determine the target domain knowledge pair where the target knowledge text is located in the domain knowledge base, and use the domain knowledge image in the target domain knowledge pair as the reference image.

[0120] Here, the domain knowledge base is dynamically updated.

[0121] After that, feature extraction is performed on the reference image, and reference generated image features are determined based on the reference image features obtained by feature extraction; feature extraction is performed on the general knowledge, and text generated image features are determined based on the general text features obtained by feature extraction; based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, image generation is performed to obtain the target image.

[0122] Among them, image generation is performed based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the target image, including: performing feature interaction based on the reference image features and the reference generated image features to obtain the first generated image features; performing feature interaction based on the general text features and the text generated image features to obtain the second generated image features; based on the routing weights, performing weighted fusion on the first generated image features and the second generated image features to obtain the generated image features; performing image generation based on the generated image features to obtain the target image.

[0123] Further, performing feature interaction based on the reference image features and the reference generated image features to obtain the first generated image features; performing feature interaction based on the general text features and the text generated image features to obtain the second generated image features, including:

[0124] Performing cross-attention interaction based on the reference image features and the reference generated image features to obtain the first generated image features; performing cross-attention interaction based on the general text features and the text generated image features, and performing self-attention interaction based on the text generated image features, and performing feature fusion based on the features obtained by the self-attention interaction and the cross-attention interaction to obtain the second generated image features.

[0125] The method provided by the embodiments of the present invention performs knowledge retrieval from the domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific domain, which can provide a reference for the generation of knowledge in a specific domain. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image can accurately reflect the knowledge in a specific domain, ensure timeliness and text consistency, and can flexibly respond to different knowledge requirements, not only ensuring the accuracy of the generated image, but also enhancing diversity and detail richness.

[0126] The image generation device provided by the present invention will be described below. The image generation device described below can be correspondingly referred to the image generation method described above.

[0127] Figure 3 is a schematic structural diagram of the image generation device provided by the present invention. As Figure 3 shown, the device includes:

[0128] A text parsing unit 310, configured to obtain an image description text, and determine domain knowledge and general knowledge in the image description text, and routing weights corresponding to the domain knowledge and the general knowledge respectively;

[0129] A knowledge retrieval unit 320, configured to perform knowledge retrieval from a domain knowledge base based on the domain knowledge to obtain a reference image; the domain knowledge base is dynamically updated;

[0130] An image generation unit 330, configured to generate an image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain a generated target image;

[0131] Among them, the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0132] The image generation device provided by the present invention performs knowledge retrieval from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific domain, which can provide a reference for the generation of knowledge in a specific domain. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image can accurately reflect the knowledge in a specific domain, ensure timeliness and text consistency, and can flexibly respond to different knowledge requirements, not only ensuring the accuracy of the generated image, but also enhancing diversity and detail richness.

[0133] Based on the above embodiment, the image generation unit 330 is configured to:

[0134] Extract features from the reference image, and determine reference generated image features based on the reference image features obtained by feature extraction;

[0135] Extract features from the general knowledge, and determine text generated image features based on the general text features obtained by feature extraction;

[0136] Generate an image based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the target image.

[0137] Based on the above embodiment, the image generation unit 330 is configured to:

[0138] Perform feature interaction based on the reference image features and the reference generated image features to obtain the first generated image features;

[0139] Perform feature interaction based on the general text features and the text generated image features to obtain the second generated image features;

[0140] Based on the first generated image features and the second generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, perform image generation to obtain the target image.

[0141] Based on the above embodiments, the image generation unit 330 is configured to:

[0142] Perform cross-attention interaction based on the reference image features and the reference generated image features to obtain the first generated image features;

[0143] Perform cross-attention interaction based on the general text features and the text generated image features, perform self-attention interaction based on the text generated image features, and perform feature fusion based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain the second generated image features.

[0144] Based on the above embodiments, the image generation unit 330 is configured to:

[0145] Based on the routing weights, perform weighted fusion on the first generated image features and the second generated image features to obtain generated image features;

[0146] Perform image generation based on the generated image features to obtain the target image.

[0147] Based on the above embodiments, the text parsing unit 310 is configured to:

[0148] Perform text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text;

[0149] Based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, determine the routing weights corresponding to the domain knowledge and the general knowledge respectively.

[0150] Based on the above embodiments, the domain knowledge base contains multiple domain knowledge pairs, and each domain knowledge pair contains a domain knowledge text and the corresponding domain knowledge image;

[0151] The knowledge retrieval unit 320 is configured to:

[0152] Perform text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when arranged in descending order according to the text similarity and / or semantic similarity with the domain knowledge.

[0153] Determine, from the domain knowledge base, a target domain knowledge pair where the target knowledge text is located, and use the domain knowledge image in the target domain knowledge pair as the reference image.

[0154] Figure 4 An example of a schematic diagram of the physical structure of an electronic device is shown as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute an image generation method, which includes: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and the general knowledge respectively; performing knowledge retrieval from the domain knowledge base based on the domain knowledge to obtain a reference image; the domain knowledge base is dynamically updated; generating a target image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, where the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0155] In addition, when the logical instructions in the above-mentioned memory 430 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or this part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0156] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image generation method provided by each of the above methods. The method includes: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and the general knowledge respectively; based on the domain knowledge, performing knowledge retrieval from a domain knowledge base to obtain a reference image; the domain knowledge base is dynamically updated; based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, performing image generation to obtain a generated target image; wherein, the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0157] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the image generation method provided by each of the above methods. The method includes: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and the general knowledge respectively; based on the domain knowledge, performing knowledge retrieval from a domain knowledge base to obtain a reference image; the domain knowledge base is dynamically updated; based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, performing image generation to obtain a generated target image; wherein, the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0159] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that, Including: Obtain an image description text, and determine the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and the general knowledge respectively; Based on the domain knowledge, perform knowledge retrieval from a domain knowledge base to obtain a reference image; The domain knowledge base is dynamically updated; Based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, perform image generation to obtain a generated target image; Among them, the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image; The determining the domain knowledge and general knowledge in the image description text, as well as the routing weights corresponding to the domain knowledge and the general knowledge respectively, includes: Perform text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text; Based on the relevance between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, determine the routing weights corresponding to the domain knowledge and the general knowledge respectively.

2. The image generation method according to claim 1, wherein The performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain a generated target image includes: Extract features from the reference image, and determine reference generated image features based on the reference image features obtained by feature extraction; Extract features from the general knowledge, and determine text generated image features based on the general text features obtained by feature extraction; Based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, perform image generation to obtain the target image.

3. The image generation method according to claim 2, wherein The performing image generation based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the target image includes: Perform feature interaction based on the reference image features and the reference generated image features to obtain first generated image features; Perform feature interaction based on the general text features and the text generated image features to obtain second generated image features; Based on the first generated image features and the second generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, perform image generation to obtain the target image.

4. The image generation method according to claim 3, wherein The performing feature interaction based on the reference image features and the reference generated image features to obtain first generated image features; The performing feature interaction based on the general text features and the text generated image features to obtain second generated image features includes: Perform cross-attention interaction based on the reference image features and the reference generated image features to obtain first generated image features; Perform cross-attention interaction based on the general text features and the text-generated image features, perform self-attention interaction based on the text-generated image features, and perform feature fusion based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain second generated image features.

5. The image generation method according to claim 3, characterized in that, The generating the target image based on the first generated image feature, the second generated image feature, and the routing weights corresponding to the domain knowledge and the general knowledge respectively includes: Based on the routing weights, perform weighted fusion on the first generated image feature and the second generated image feature to obtain a generated image feature; Based on the generated image feature, perform image generation to obtain the target image.

6. The image generation method according to any one of claims 1 to 5, characterized in that The domain knowledge base contains multiple domain knowledge pairs, and each domain knowledge pair contains a domain knowledge text and a corresponding domain knowledge image; The retrieving reference image from the domain knowledge base based on the domain knowledge includes: Perform text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when arranged in descending order of text similarity and / or semantic similarity with the domain knowledge; From the domain knowledge base, determine the target domain knowledge pair where the target knowledge text is located, and use the domain knowledge image in the target domain knowledge pair as the reference image.

7. An image generation device, characterized in that, Includes: A text parsing unit for obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; A knowledge retrieval unit for retrieving reference image from the domain knowledge base based on the domain knowledge; The domain knowledge base is dynamically updated; An image generation unit for performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain the generated target image; Wherein, the routing weights are used to reflect the presentation degree of the corresponding knowledge on the target image; The text parsing unit is used for: Perform text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text; Based on the relevance between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, determine the routing weights corresponding to the domain knowledge and the general knowledge respectively.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image generation method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image generation method according to any one of claims 1 to 6.

Images