Image generation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an image description text, determining domain knowledge and general knowledge in the image description text, and determining routing weights corresponding to the domain knowledge and the general knowledge; based on the domain knowledge, performing knowledge retrieval from the domain knowledge base to obtain a reference image; dynamically updating the domain knowledge base; performing image generation based on the reference image, the general knowledge and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain a generated target image; wherein the routing weight is used for reflecting the presentation degree of the corresponding knowledge on the target image, knowledge retrieval in a specific field is carried out by utilizing the advantages of dynamic knowledge retrieval, reference can be provided for knowledge generation in the specific field, and meanwhile, image generation is carried out by virtue of a dynamic routing strategy, so that the accuracy of the generated image is ensured; and the diversity and the detail richness are also enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image generating method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, text-to-image generation technology has become an important tool in many fields such as image content creation, advertising design, education, and scientific research. The core advantage of this type of generation model is that it can automatically generate corresponding images based on the text description entered by the user, thereby improving the efficiency and creative space of image creation.

[0003] However, current text-image generation technology relies on large-scale general training data to ensure the diversity and versatility of generation. This training method limits the model's generation capabilities when processing knowledge in specific fields. Specifically, since most of the training data comes from image-text pairs in general scenarios, the model finds it difficult to accurately capture and convey professional concepts and details when generating images in specific fields, resulting in insufficient accuracy and richness of the generated results.

[0004] In addition, with the acceleration of knowledge updating, knowledge concepts in some rapidly changing or emerging fields are often not fully covered by the model, which further aggravates the limitations of the model in generating images in these fields, affecting the accuracy and diversity of the generated results. Summary of the invention

[0005] The present invention provides an image generation method, device, electronic device and storage medium, which are used to solve the defects of limitations in image generation in specific fields in the prior art, insufficient accuracy and richness of generated images, achieve fast and accurate image generation, and ensure the diversity and richness of details of generated images.

[0006] The present invention provides an image generation method, comprising: Acquire an image description text, and determine domain knowledge and general knowledge in the image description text, and routing weights corresponding to the domain knowledge and the general knowledge respectively; Based on the domain knowledge, knowledge retrieval is performed from a domain knowledge base to obtain a reference image; the domain knowledge base is dynamically updated; Performing image generation based on the reference image, the general knowledge, and routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain a generated target image; The routing weight is used to reflect the degree of presentation of the corresponding knowledge on the target image.

[0007] According to an image generation method provided by the present invention, the image generation is performed based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain a generated target image, including: Extracting features from the reference image, and determining reference generated image features based on reference image features obtained through feature extraction; Extracting features from the general knowledge, and determining text-generated image features based on general text features obtained through feature extraction; Image generation is performed based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image.

[0008] According to an image generation method provided by the present invention, the image generation is performed based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image, including: Perform feature interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; Perform feature interaction based on the general text feature and the text-generated image feature to obtain a second generated image feature; Image generation is performed based on the first generated image feature and the second generated image feature, and routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image.

[0009] According to an image generation method provided by the present invention, the step of performing feature interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; and performing feature interaction based on the general text feature and the text generated image feature to obtain a second generated image feature comprises: Performing cross-attention interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; A cross-attention interaction is performed based on the common text feature and the text-generated image feature, a self-attention interaction is performed based on the text-generated image feature, and feature fusion is performed based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain a second generated image feature.

[0010] According to an image generation method provided by the present invention, the image generation is performed based on the first generated image feature and the second generated image feature, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image, including: Based on the routing weight, weighted fusion is performed on the first generated image feature and the second generated image feature to obtain a generated image feature; Image generation is performed based on the generated image features to obtain the target image.

[0011] According to an image generation method provided by the present invention, the determining of the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, includes: Performing text parsing on the image description text to obtain domain knowledge and general knowledge in the image description text; Based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, the routing weights corresponding to the domain knowledge and the general knowledge are determined.

[0012] According to an image generation method provided by the present invention, the domain knowledge base includes a plurality of domain knowledge pairs, each of which includes a domain knowledge text and a corresponding domain knowledge image; The step of retrieving knowledge from a domain knowledge base based on the domain knowledge to obtain a reference image includes: Performing text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when the target knowledge text is arranged in descending order of text similarity and / or semantic similarity with the domain knowledge; A target domain knowledge pair where the target knowledge text is located is determined from the domain knowledge base, and a domain knowledge image in the target domain knowledge pair is used as the reference image.

[0013] The present invention also provides an image generating device, comprising: A text parsing unit, used to obtain an image description text, and determine the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; A knowledge retrieval unit, used to retrieve knowledge from a domain knowledge base based on the domain knowledge to obtain a reference image; the domain knowledge base is dynamically updated; An image generating unit, configured to generate an image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge, to obtain a generated target image; The routing weight is used to reflect the degree of presentation of the corresponding knowledge on the target image.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described image generation methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image generating method as described in any one of the above is implemented.

[0016] The image generation method, device, electronic device and storage medium provided by the present invention perform knowledge retrieval from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific field, which can provide a reference for knowledge generation in the specific field. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image accurately reflects the knowledge in the specific field, ensures timeliness and text consistency, and can flexibly respond to different knowledge needs, which not only ensures the accuracy of the generated image, but also enhances diversity and detail richness. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of the image generation method provided by the present invention; Figure 2 It is an overall flow chart of the image generation method provided by the present invention; Figure 3 It is a structural schematic diagram of the image generating device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] At present, the text-image generation technology mainly relies on large-scale general training data to ensure the diversity and versatility of generation. Common generation schemes include text-image generation methods based on a single model, such as DALL-E and Stable Diffusion. Such schemes convert the input text description into the corresponding image through a single generation model, which performs well in general scenarios. However, they are limited by the coverage of training data in knowledge generation in professional fields. That is, since the training data is mostly graphic data in general scenarios with limited coverage, the generation ability of the model is limited when processing knowledge in specific fields, especially in complex and highly professional fields such as medicine and mechanical engineering. This knowledge limitation stems from the lack of targeted knowledge retrieval and dynamic update mechanisms in existing models, especially when it comes to knowledge in some rapidly changing or emerging fields, the accuracy and richness of model generation are particularly insufficient. Therefore, combining knowledge retrieval with generation enhancement to better support multi-domain and high-precision knowledge generation has become a major research hotspot in the current text-image generation field.

[0021] In order to meet this challenge, in recent years, it has been proposed to introduce knowledge enhancement methods such as knowledge graphs and external knowledge bases into the generative model, hoping that through knowledge expansion and supplementation, the generative model can perform better in the generation of specific knowledge. This type of enhancement scheme has improved the professionalism of the model to a certain extent and improved the generation effect in specific fields. However, due to the untimely update of the knowledge base or the single knowledge embedding method, the generation results are still inaccurate and incomplete. In addition, if there is too much reliance on real images in the knowledge base, it will seriously affect the imagination of the model in image generation, resulting in the final generated image being affected by the retrieval content and the richness being reduced.

[0022] In this regard, the present invention provides an image generation method, which aims to utilize the advantages of dynamic knowledge retrieval to solve the current difficulties in knowledge generation in specific fields. At the same time, with the help of dynamic routing strategies, it is possible to flexibly respond to different knowledge requirements in the image generation process, which not only ensures the accuracy of the generated images, but also enhances the diversity and richness of details. Figure 1 It is a flow chart of the image generation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining the image description text, and determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and general knowledge respectively; wherein the routing weights are used to reflect the presentation degree of the corresponding knowledge on the target image; Step 120, based on the domain knowledge, knowledge retrieval is performed from the domain knowledge base to obtain a reference image; the domain knowledge base is dynamically updated; Step 130 , performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge, to obtain a generated target image.

[0023] Specifically, considering the limitations of current text-image generation schemes in knowledge coverage and image generation, that is, the model shows generation missing / bias when processing knowledge concepts outside the training data, especially in the generation of knowledge in a specific field, it is difficult to accurately convey professional concepts and details, and when it comes to knowledge in rapidly changing or emerging fields, the generation quality and diversity are low, and the generation results are difficult to fully and accurately present the diverse information contained in the text. In an embodiment of the present invention, it is proposed that image generation can be performed based on dynamic knowledge retrieval, so that knowledge retrieval can be performed through a domain knowledge base that is maintained and dynamically updated in real time, and accurate and real-time reference knowledge in a specific field can be obtained. On this basis, weight distribution is performed with the help of a dynamic routing mechanism, so that the final generated image can meet the needs, which can not only accurately reflect the knowledge in a specific field, but also satisfy the overall image description.

[0024] It is understandable that in the actual application process, before performing image generation and text-image generation tasks, the first thing that needs to be determined is the description of the image to be generated, that is, the image description text. The image to be generated described by the image description text here can be an image in a general field, such as a street view, a character image, etc., or an image in a specific field, such as a human body structure diagram in the medical field, an equipment structure diagram in the mechanical engineering field, etc., and the embodiment of the present invention does not make specific limitations. Considering that the current text-image generation technology has achieved good results in the general field, but has limitations in specific fields, in the embodiment of the present invention, the text-to-image generation is mainly carried out for specific fields, and the image description text can be a description of the image in a specific field, such as "generate an image containing the general structure and internal structure of the human heart, which is required to be vivid and lifelike with rich colors."

[0025] After determining the image description text, in an embodiment of the present invention, the image description text can be parsed to identify the general domain knowledge and the specific domain knowledge contained therein; that is, the image description text can be accurately parsed to distinguish the general knowledge and domain knowledge therein. Here, specifically, with the help of the powerful learning and understanding capabilities of a large language model, the knowledge in the specific domain and the knowledge in the general domain can be identified, and the identified information can be organized into domain knowledge corresponding to the specific domain and general knowledge corresponding to the general domain through natural language processing.

[0026] Furthermore, after determining the domain knowledge and general knowledge in the image description text, in the embodiment of the present invention, a weight evaluation is also required to determine the impact of the domain knowledge on image generation, so as to obtain the routing weights corresponding to the domain knowledge and general knowledge. Specifically, due to the differences in information contained in different image description texts, as well as the differences in the proportion and importance of domain knowledge and general knowledge therein, in order to better and more accurately generate images for image description texts, in the embodiment of the present invention, after determining the domain knowledge and general knowledge, it is also necessary to evaluate their guiding role and importance in this image generation task, thereby obtaining the routing weights corresponding to the domain knowledge and general knowledge.

[0027] Here, the weight evaluation process can also be implemented with the help of a large language model. Specifically, the impact of domain knowledge and general knowledge on the image generation task can be evaluated based on the relevance and importance of domain knowledge, as well as the matching degree with other information in the image description text, so as to obtain the corresponding routing weights. This weight determines the prominence of domain knowledge in the generated image during subsequent image generation, that is, the routing weight can better reflect the importance / influence of the corresponding knowledge in the image generation process, as well as the degree of presentation in the final generated image. Image generation in this way can make the generated image both accurately reflect the domain knowledge and maintain overall consistency with the original image description text.

[0028] After parsing the image description text to obtain domain knowledge and general knowledge, further knowledge retrieval can be performed for domain knowledge in the embodiment of the present invention to ensure the timeliness and completeness of knowledge generation in this specific domain. That is, considering that the current model knowledge coverage is limited and there are limitations in knowledge generation in a specific domain, in the embodiment of the present invention, it is proposed that after determining the domain knowledge, a retrieval enhancement method can be introduced to perform knowledge retrieval for the domain knowledge, so as to retrieve from the domain knowledge base corresponding to the specific domain, so as to obtain the image corresponding to the domain knowledge, and the image can be used as a reference for subsequent image generation tasks, so as to obtain the final generated image.

[0029] Specifically, based on the domain knowledge, knowledge retrieval can be performed in the domain knowledge base under the specific domain to retrieve images associated with and matching the domain knowledge as reference images corresponding to the domain knowledge for subsequent image generation.

[0030] The domain knowledge base in a specific domain contains a large number of organized and structured knowledge images and their corresponding text information. Therefore, when performing knowledge retrieval, the domain knowledge can be used as a guide to retrieve the matching text information, thereby obtaining the corresponding knowledge image, and determining the reference image corresponding to the domain knowledge based on the knowledge image.

[0031] It is also worth noting that the domain knowledge base here is maintained and dynamically updated in real time, and its content can be dynamically updated according to domain needs and user feedback to maintain the timeliness and integrity of the data in the library. Here, the content in the dynamically updated domain knowledge base can be continuously accumulated and updated over time. This dynamic update mechanism can keep the content in the library at the forefront of technology and keep pace with the latest knowledge, so that the final generated image has good accuracy, timeliness and rich details. Here, there are many ways to dynamically update the domain knowledge base, such as the knowledge update and maintenance method based on the RAG (Retrieval-Augmented Generation) model, which can achieve dynamic updates of large-scale knowledge bases while ensuring the content quality in the domain knowledge base.

[0032] In an embodiment of the present invention, a domain knowledge base that is maintained in real time is constructed, and knowledge retrieval in a specific domain is performed based on it, which can provide a reference for knowledge generation in the specific domain, ensure that the generated image can accurately reflect the knowledge or concepts in the specific domain, and ensure timeliness and knowledge synchronization.

[0033] After that, the image can be generated based on the retrieved reference image, general knowledge, and the routing weights corresponding to the domain knowledge and general knowledge, so as to obtain the desired generated image, that is, the target image corresponding to the image description text. Here, specifically, the image can be generated based on the knowledge content of the specific field represented by the reference image, combined with the general knowledge and the routing weights corresponding to the two, so as to provide the precise visual features of the knowledge content of the specific field required for image generation through the reference image, provide relevant information of the image description text through the general knowledge, and combine the importance / influence of the two in the image generation process represented by the routing weights, as well as the degree of presentation on the generated image, to generate the final target image.

[0034] Specifically, since the routing weight can characterize the influence of domain knowledge and general text on the target image, when generating images based on the routing weight, if the routing weight of domain knowledge is higher, the generated target image will be more biased towards domain knowledge, and when the weight is biased towards general knowledge, the generated target image will be more consistent with the description of general knowledge. In this way, the dynamic adjustment of the target image between domain knowledge and general knowledge is achieved, so that the generated target image can not only accurately reflect the domain knowledge, but also conform to the overall image description.

[0035] The image generation method provided by the present invention performs knowledge retrieval from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific field, which can provide a reference for knowledge generation in the specific field. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image can accurately reflect the knowledge in the specific field, ensure timeliness and text consistency, and can flexibly respond to different knowledge needs, which not only ensures the accuracy of the generated image, but also enhances the diversity and richness of details.

[0036] Based on the above embodiment, step 130 includes: Extracting features of the reference image, and determining features of the reference generated image based on the features of the reference image obtained by the feature extraction; Extracting features of general knowledge, and determining features of text-generated images based on general text features obtained through feature extraction; Image generation is performed based on reference image features and reference generated image features, general text features and text generated image features, and routing weights corresponding to domain knowledge and general knowledge, to obtain a target image.

[0037] Specifically, the process of generating an image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge to obtain the generated target image may specifically include the following steps: In detail, the image generation process in the embodiment of the present invention is implemented using a dual expert model architecture, specifically, two expert models are introduced, namely, an image reference generation model and a text reference generation model, which process knowledge images in specific fields and general texts separately, so as to accurately generate knowledge in specific fields through the image reference generation model, while using the text reference generation model to ensure the overall quality and diversity of the generated images.

[0038] Specifically, here, the general knowledge may be processed first to extract knowledge features that are helpful for image generation, thereby obtaining text features of the general knowledge, that is, general text features. Here, the general knowledge may be extracted through a text reference generation model to obtain general text features; then, feature construction may be performed based on the general text features to construct a feature representation of the image to be generated, and the general text features may be used as the initialized image representation, and feature construction may be performed step by step from then on, and new image features may be generated in each step based on the current image representation and the general text features, and finally the feature representation of the image to be generated may be obtained, that is, the text generated image features.

[0039] At the same time, the reference image can be processed to extract knowledge features in a specific field that are helpful for image generation, thereby obtaining image features of the reference image, i.e., reference image features. Here, the reference image features can be obtained by extracting features from the reference image through an image reference generation model. Similarly, feature construction can be performed based on the reference image features to construct a feature representation of the image to be generated, i.e., reference generated image features.

[0040] It should be noted that in order to ensure the accuracy of image generation, the two expert models in the embodiment of the present invention can be pre-trained by corresponding sample data, that is, two initial models can be constructed respectively on the basis of the current text-generated image expert model and image-generated image expert model, and then trained using text data, image data, etc. to obtain two expert models with better image generation capabilities.

[0041] After that, the image can be generated based on the text-generated image features and the reference-generated image features to obtain the target image. That is, based on the reference-generated image features and the text-generated image features, the image can be generated by combining the routing weights corresponding to the domain knowledge and the general knowledge, so as to ensure that the generated image accurately reflects the knowledge in the specific domain while also conforming to the overall image description, and obtain the final target image.

[0042] In the embodiment of the present invention, a dual expert model is used to process general knowledge and reference images respectively, and on this basis, routing weights are combined to generate images, which can achieve a dynamic balance between domain knowledge and general knowledge, thereby ensuring the accuracy and richness of the generated images.

[0043] Based on the above embodiment, image generation is performed based on reference image features and reference generated image features, general text features and text generated image features, and routing weights corresponding to domain knowledge and general knowledge, respectively, to obtain a target image, including: Perform feature interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; Perform feature interaction based on the general text feature and the text-generated image feature to obtain a second generated image feature; Image generation is performed based on the first generated image feature and the second generated image feature, as well as routing weights corresponding to the domain knowledge and the general knowledge, to obtain a target image.

[0044] Specifically, the process of generating an image based on reference image features and reference generated image features, general text features and text generated image features, and routing weights corresponding to domain knowledge and general knowledge to obtain a target image may specifically include: First, the reference image features and the reference generated image features can be interacted to capture the correlation between the two, thereby obtaining new features, which are referred to as first generated image features. Specifically, an attention mechanism can be used to perform cross-sequence interaction and feature fusion on the reference generated image features and the reference image features, thereby obtaining the first generated image features.

[0045] At the same time, the general text features and the text-generated image features can be interacted, and the correlation between the two can be captured to obtain new features, namely, the second generated image features. Here, specifically, an attention mechanism can be introduced to perform feature interaction between the text-generated image features and the general text features. The feature interaction here can include cross-modal interaction, image modality self-attention interaction, etc. Finally, the second generated image features can be obtained through feature fusion.

[0046] After that, the image can be generated based on the first generated image features and the second generated image features, as well as the routing weights corresponding to the domain knowledge and the general knowledge, so as to obtain the target image. That is, the image is generated based on the first generated image features and the second generated image features obtained by feature interaction, combined with the routing weights, to ensure that the target image can not only accurately reflect the knowledge of a specific field, but also conform to the overall image description.

[0047] Based on the above embodiment, feature interaction is performed based on the reference image feature and the reference generated image feature to obtain the first generated image feature; and feature interaction is performed based on the general text feature and the text generated image feature to obtain the second generated image feature, including: Perform cross-attention interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; A cross-attention interaction is performed based on the general text features and the text-generated image features, and a self-attention interaction is performed based on the text-generated image features. Feature fusion is performed based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain a second generated image feature.

[0048] Specifically, the feature interaction between the reference image features and the reference generated image features adopts the interactive attention method, that is, cross-attention interaction is performed on the reference image features and the reference generated image features to achieve interaction between two different sequences, and dynamically adjust the current sequence's attention to other sequences through the similarity between the sequence query (Query), key (Key) and value (Value), thereby realizing the comprehensive utilization of knowledge, and finally obtaining new features after interaction, namely, the first generated image features.

[0049] Correspondingly, when performing feature interaction between general text features and text-generated image features, there are two interaction modes, namely, self-attention interaction and cross-attention interaction, that is, cross-attention interaction can be performed between general text features and text-generated image features, and self-attention interaction can be performed on text-generated image features, and the features obtained by cross-attention interaction and the features obtained by self-attention interaction can be feature fused to obtain the second generated image features. Here, through cross-attention interaction, information of different modalities can be paid attention to, and through self-attention interaction, more detailed attention can be paid to one's own information, so that the final generated image is more accurate and richer in details.

[0050] In the embodiment of the present invention, the self-attention and cross-attention mechanisms are introduced to act on text and image features, so that the image generation process can better capture global context information and realize the comprehensive utilization of multimodal information, so that the final generated target image matches the image description.

[0051] Based on the above embodiment, image generation is performed based on the first generated image feature and the second generated image feature, as well as the routing weight, to obtain a target image, including: Based on the routing weight, weighted fusion is performed on the first generated image feature and the second generated image feature to obtain a generated image feature; Image generation is performed based on the generated image features to obtain the target image.

[0052] Specifically, the above process of generating an image according to the first generated image feature and the second generated image feature and the routing weight to obtain the target image is actually to first perform weighted fusion at the feature level to obtain the features used for image generation, i.e., generated image features, and then generate an image based on this to obtain the final target image.

[0053] Specifically, here, the first generated image feature and the second generated image feature are weighted and fused according to the routing weight, so as to obtain the fused feature, i.e., the generated image feature. That is, the first generated image feature and the routing weight corresponding to the domain knowledge are weighted, the second generated image feature and the routing weight corresponding to the general knowledge are weighted, and the two are compared and fused / added to obtain the generated image feature.

[0054] Next, the image features can be generated based on this to generate the final image and obtain the target image.

[0055] It should be noted that the sum of the routing weight corresponding to the domain knowledge and the routing weight corresponding to the general knowledge is 1.

[0056] Based on the above embodiment, the process of weighted fusion of the first generated image feature and the second generated image feature can be expressed by the following formula: In the formula, and They are the routing weights corresponding to domain knowledge and the routing weights corresponding to general knowledge; and Then are the first generated image features and the second generated image features, To generate image features.

[0057] Based on the above embodiment, determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and general knowledge respectively, includes: Perform text parsing on the image description text to obtain the domain knowledge and general knowledge in the image description text; Based on the correlation between domain knowledge and image description text, and the matching degree between domain knowledge and general knowledge, the routing weights corresponding to the domain knowledge and general knowledge are determined respectively.

[0058] Specifically, the process of the domain knowledge and general knowledge in the above image description text, and the routing weights corresponding to the domain knowledge and general knowledge, respectively, may specifically include: First, the image description text can be parsed. Specifically, the input image description text can be parsed through a large language model to accurately identify the general knowledge and domain knowledge therein. That is, with the help of the powerful learning and understanding ability of the large language model, the knowledge in the specific field and the knowledge in the general field can be identified, and the identified information can be organized into domain knowledge corresponding to the specific field and general knowledge corresponding to the general field through natural language processing.

[0059] Next, the degree of influence of this domain knowledge and general knowledge on the image generation process can be evaluated. The basis for the evaluation here can be the importance of the domain knowledge, the matching degree and correlation degree with other information, etc. Based on this evaluation, the routing weights corresponding to the domain knowledge and general knowledge can be obtained. Here, specifically, the importance / influence of the domain knowledge in the image generation process can be measured based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and general knowledge. Based on this, the routing weight corresponding to the domain knowledge can be determined, and the routing weight of the corresponding general knowledge can be directly determined according to the normalization principle.

[0060] Here, based on the relevance of the domain knowledge and its matching degree with other information, the influence of the domain knowledge in the image generation process is dynamically calculated to determine the routing weight. This weight determines the prominence of the domain knowledge in the generated target image in the subsequent image generation process, thereby ensuring that the final generated target image can accurately reflect the knowledge in a specific field while maintaining overall consistency with the initial image description.

[0061] In an embodiment of the present invention, a routing mechanism based on a large language model is adopted. The image description text is parsed by a large language model to separate domain knowledge and general knowledge, and the corresponding routing weights are evaluated to participate in the subsequent image generation process, thereby ensuring that the influence weights of general knowledge and domain knowledge can be dynamically allocated in different image generation tasks.

[0062] Based on the above embodiment, the domain knowledge base includes multiple domain knowledge pairs, each of which includes a domain knowledge text and a corresponding domain knowledge image; Step 120 includes: Performing text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when the target knowledge text is arranged in descending order of text similarity and / or semantic similarity with the domain knowledge; From the domain knowledge base, a target domain knowledge pair where the target knowledge text is located is determined, and the domain knowledge image in the target domain knowledge pair is used as a reference image.

[0063] Specifically, the process of performing knowledge retrieval from a domain knowledge base based on domain knowledge to obtain a reference image may specifically include: First, based on the domain knowledge, a search is performed in the domain knowledge base to find text information that has a certain similarity with the domain knowledge at the text level and / or semantic level, that is, the domain knowledge text, and the found domain knowledge text can be sorted according to the degree of similarity to obtain a matching domain knowledge text sequence; that is, the retrieved domain knowledge texts can be sorted in order from high to low or from low to high in terms of text similarity and / or semantic similarity to obtain a domain knowledge text sequence, and the first or last domain knowledge text can be selected from this sequence as the target knowledge text. It can also be understood that the domain knowledge text that is most similar to the domain knowledge is selected as the target knowledge text.

[0064] Subsequently, the domain knowledge image corresponding to the target knowledge text can be determined based on the matching relationship between the domain knowledge text and the knowledge image (domain knowledge image) in the domain knowledge base. That is, the domain knowledge pair corresponding to the target knowledge text in the domain knowledge base, namely, the target domain knowledge pair, is first determined, and then the domain knowledge image in this target domain knowledge pair is determined, and this domain knowledge image is used as the reference image corresponding to the domain knowledge.

[0065] Figure 2 is the overall flow chart of the image generation method provided by the present invention, such as Figure 2 As shown, the method includes: First, get the image description text.

[0066] Next, the image description text is parsed to obtain the domain knowledge and general knowledge in the image description text. Based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, the routing weights corresponding to the domain knowledge and the general knowledge are determined respectively; the routing weights are used to reflect the presentation degree of the corresponding knowledge on the target image.

[0067] Subsequently, a text search is performed in the domain knowledge base to obtain a target knowledge text; the domain knowledge base contains multiple domain knowledge pairs, each of which contains a domain knowledge text and a corresponding domain knowledge image; the target knowledge text is the first domain knowledge text when the text similarity and / or semantic similarity with the domain knowledge is arranged from high to low; from the domain knowledge base, the target domain knowledge pair where the target knowledge text is located is determined, and the domain knowledge image in the target domain knowledge pair is used as a reference image.

[0068] Here, the domain knowledge base is updated dynamically.

[0069] Afterwards, feature extraction is performed on the reference image, and reference generated image features are determined based on the reference image features obtained through feature extraction; feature extraction is performed on general knowledge, and text generated image features are determined based on general text features obtained through feature extraction; image generation is performed based on reference image features and reference generated image features, general text features and text generated image features, as well as routing weights corresponding to domain knowledge and general knowledge, to obtain a target image.

[0070] Among them, image generation is performed based on reference image features and reference generated image features, general text features and text generated image features, and routing weights corresponding to domain knowledge and general knowledge respectively to obtain a target image, including: performing feature interaction based on reference image features and reference generated image features to obtain first generated image features; performing feature interaction based on general text features and text generated image features to obtain second generated image features; performing weighted fusion of the first generated image features and the second generated image features based on the routing weights to obtain generated image features; and performing image generation based on the generated image features to obtain a target image.

[0071] Further, performing feature interaction based on the reference image feature and the reference generated image feature to obtain the first generated image feature; performing feature interaction based on the general text feature and the text generated image feature to obtain the second generated image feature, including: A cross-attention interaction is performed based on the reference image features and the reference generated image features to obtain the first generated image features; a cross-attention interaction is performed based on the general text features and the text-generated image features, and a self-attention interaction is performed based on the text-generated image features, and feature fusion is performed based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain the second generated image features.

[0072] The method provided by the embodiment of the present invention performs knowledge retrieval from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific domain, which can provide a reference for knowledge generation in the specific domain. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image can accurately reflect the knowledge in the specific domain, ensure timeliness and text consistency, and can flexibly respond to different knowledge needs, which not only ensures the accuracy of the generated image, but also enhances the diversity and richness of details.

[0073] The image generating device provided by the present invention is described below. The image generating device described below and the image generating method described above can be referred to each other.

[0074] Figure 3 is a schematic diagram of the structure of the image generating device provided by the present invention, such as Figure 3As shown, the device comprises: A text parsing unit 310 is used to obtain an image description text, and determine the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; A knowledge retrieval unit 320 is used to retrieve knowledge from a domain knowledge base based on the domain knowledge to obtain a reference image; the domain knowledge base is dynamically updated; An image generating unit 330 is configured to generate an image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain a generated target image; The routing weight is used to reflect the degree of presentation of the corresponding knowledge on the target image.

[0075] The image generation device provided by the present invention performs knowledge retrieval from a domain knowledge base based on the domain knowledge in the image description text to obtain a reference image; the domain knowledge base is dynamically updated, and the advantage of dynamic knowledge retrieval is used to perform knowledge retrieval in a specific field, which can provide a reference for knowledge generation in the specific field. On this basis, image generation is performed based on a dynamic routing strategy, which can ensure that the generated target image can accurately reflect the knowledge in the specific field, ensure timeliness and text consistency, and can flexibly respond to different knowledge needs, which not only ensures the accuracy of the generated image, but also enhances the diversity and richness of details.

[0076] Based on the above embodiment, the image generating unit 330 is used for: Extracting features from the reference image, and determining reference generated image features based on reference image features obtained through feature extraction; Extracting features from the general knowledge, and determining text-generated image features based on general text features obtained through feature extraction; Image generation is performed based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image.

[0077] Based on the above embodiment, the image generating unit 330 is used for: Perform feature interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; Perform feature interaction based on the general text feature and the text-generated image feature to obtain a second generated image feature; Image generation is performed based on the first generated image feature and the second generated image feature, and routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image.

[0078] Based on the above embodiment, the image generating unit 330 is used for: Performing cross-attention interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; A cross-attention interaction is performed based on the common text feature and the text-generated image feature, a self-attention interaction is performed based on the text-generated image feature, and feature fusion is performed based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain a second generated image feature.

[0079] Based on the above embodiment, the image generating unit 330 is used for: Based on the routing weight, weighted fusion is performed on the first generated image feature and the second generated image feature to obtain a generated image feature; Image generation is performed based on the generated image features to obtain the target image.

[0080] Based on the above embodiment, the text parsing unit 310 is used for: Performing text parsing on the image description text to obtain domain knowledge and general knowledge in the image description text; Based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, the routing weights corresponding to the domain knowledge and the general knowledge are determined.

[0081] Based on the above embodiment, the domain knowledge base includes a plurality of domain knowledge pairs, each of which includes a domain knowledge text and a corresponding domain knowledge image; The knowledge retrieval unit 320 is used to: Performing text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when the target knowledge text is arranged in descending order of text similarity and / or semantic similarity with the domain knowledge; A target domain knowledge pair where the target knowledge text is located is determined from the domain knowledge base, and a domain knowledge image in the target domain knowledge pair is used as the reference image.

[0082] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the image generation method, which includes: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; based on the domain knowledge, performing knowledge retrieval from a domain knowledge base to obtain a reference image; dynamically updating the domain knowledge base; performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, to obtain a generated target image; wherein the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0083] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0084] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image generation method provided by the above-mentioned methods, and the method includes: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; based on the domain knowledge, performing knowledge retrieval from a domain knowledge base to obtain a reference image; dynamically updating the domain knowledge base; performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively, to obtain a generated target image; wherein the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0085] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image generation method provided by the above-mentioned methods, the method comprising: obtaining an image description text, and determining the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; based on the domain knowledge, performing knowledge retrieval from a domain knowledge base to obtain a reference image; dynamically updating the domain knowledge base; performing image generation based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge respectively to obtain a generated target image; wherein the routing weight is used to reflect the presentation degree of the corresponding knowledge on the target image.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that: include: Acquire an image description text, and determine domain knowledge and general knowledge in the image description text, and routing weights corresponding to the domain knowledge and the general knowledge respectively; Based on the domain knowledge, knowledge retrieval is performed from a domain knowledge base to obtain a reference image; The domain knowledge base is dynamically updated; Performing image generation based on the reference image, the general knowledge, and routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain a generated target image; The routing weight is used to reflect the degree of presentation of the corresponding knowledge on the target image.

2. The image generation method according to claim 1, characterized in that: The step of generating an image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge to obtain a generated target image includes: Extracting features from the reference image, and determining reference generated image features based on reference image features obtained through feature extraction; Extracting features from the general knowledge, and determining text-generated image features based on general text features obtained through feature extraction; Image generation is performed based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image.

3. The image generation method according to claim 2, characterized in that: The step of generating an image based on the reference image features and the reference generated image features, the general text features and the text generated image features, and the routing weights corresponding to the domain knowledge and the general knowledge to obtain the target image includes: Perform feature interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; Perform feature interaction based on the general text feature and the text-generated image feature to obtain a second generated image feature; Image generation is performed based on the first generated image feature and the second generated image feature, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, to obtain the target image.

4. The image generation method according to claim 3, characterized in that: performing feature interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; Performing feature interaction based on the general text feature and the text-generated image feature to obtain a second generated image feature includes: Performing cross-attention interaction based on the reference image feature and the reference generated image feature to obtain a first generated image feature; A cross-attention interaction is performed based on the common text feature and the text-generated image feature, and a self-attention interaction is performed based on the text-generated image feature. Feature fusion is performed based on the features obtained from the self-attention interaction and the cross-attention interaction to obtain a second generated image feature.

5. The image generation method according to claim 3, characterized in that: The step of generating an image based on the first generated image feature and the second generated image feature, and the routing weights corresponding to the domain knowledge and the general knowledge, to obtain the target image, includes: Based on the routing weight, weighted fusion is performed on the first generated image feature and the second generated image feature to obtain a generated image feature; Image generation is performed based on the generated image features to obtain the target image.

6. The image generation method according to any one of claims 1 to 5, characterized in that: The determining of the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge, respectively, includes: Performing text parsing on the image description text to obtain domain knowledge and general knowledge in the image description text; Based on the correlation between the domain knowledge and the image description text, and the matching degree between the domain knowledge and the general knowledge, the routing weights corresponding to the domain knowledge and the general knowledge are determined.

7. The image generation method according to any one of claims 1 to 5, characterized in that: The domain knowledge base includes a plurality of domain knowledge pairs, each of which includes a domain knowledge text and a corresponding domain knowledge image; The step of retrieving knowledge from a domain knowledge base based on the domain knowledge to obtain a reference image includes: Performing text retrieval in the domain knowledge base to obtain a target knowledge text; the target knowledge text is the first domain knowledge text when the target knowledge text is arranged in descending order of text similarity and / or semantic similarity with the domain knowledge; A target domain knowledge pair where the target knowledge text is located is determined from the domain knowledge base, and a domain knowledge image in the target domain knowledge pair is used as the reference image.

8. An image generating device, characterized in that: include: A text parsing unit, used to obtain an image description text, and determine the domain knowledge and general knowledge in the image description text, and the routing weights corresponding to the domain knowledge and the general knowledge respectively; A knowledge retrieval unit, configured to retrieve knowledge from a domain knowledge base based on the domain knowledge to obtain a reference image; The domain knowledge base is dynamically updated; An image generating unit, configured to generate an image based on the reference image, the general knowledge, and the routing weights corresponding to the domain knowledge and the general knowledge, to obtain a generated target image; The routing weight is used to reflect the degree of presentation of the corresponding knowledge on the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image generating method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 7 is implemented.

Images