Adaptive content eliminator and elimination method for image generation large model

Abstract

The invention discloses an adaptive content eliminator and elimination method for an image generation large model, and the method comprises the steps: constructing an elimination space and a reserved space according to a target content, the elimination space corresponding to semantic information needing to be removed, and the reserved space corresponding to semantic information needing to be reserved; converting a text prompt input by a user into semantic representation, and projecting the semantic representation into the elimination space and the reserved space token by token to obtain a reserved component of the reserved space; measuring a semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculating a correlation degree; according to the correlation degree result, the reserved space is reconstructed, and the direction of the reserved component is corrected. The method does not need to consume a large amount of computing resources to finely adjust the model, and does not introduce additional parameters. And when the single or multiple target contents are successfully removed, the influence on the generation of irrelevant contents can be ignored, so that the content elimination performance of the generative model is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically to an adaptive content eliminator and elimination method for a large image generation model. Background Art

[0002] In recent years, the rapid development of text-to-image (T2I) diffusion models has enabled users to easily generate high-quality images with simple text prompts. With their powerful generation capabilities, these models have been widely used in content creation, art design, and other fields. However, the training of such models usually relies on large-scale datasets crawled from the Internet, which are of varying quality and inevitably contain unsafe content. This results in the model accidentally presenting offensive content or content involving copyright disputes when generating images, which brings ethical, legal, and security challenges to practical applications. To address these issues, an intuitive solution is to retrain the generative model after strictly screening the dataset. However, this approach has significant limitations. First, it is often necessary to design specific detectors to screen data for different types of target content, which not only increases the complexity of development, but also cannot cope with unknown or newly emerging target content. Second, retraining large generative models from scratch is extremely costly, time-consuming, and resource-intensive. In addition, even if the model is retrained after screening, it may still introduce unexpected bad tendencies due to data bias. Another simple solution is to use text filters or safety checkers. For example, text filters can directly block text prompts containing target content, and security checkers can filter results containing unsafe content by detecting generated images. However, these methods have limited protection capabilities and are easily bypassed by cleverly designed malicious prompts, and cannot fundamentally solve the problem. Therefore, it is particularly important to develop a low-cost, efficient, and accurate technology to remove unwanted semantics in generated images (i.e., content removal).

[0003] Existing content removal technologies are mainly divided into two categories. The first is the fine-tuning-based method. This type of method fine-tunes the model parameters to specifically remap the semantics of the target content and the alternative content, or weaken the expression of the target content. Although this type of method is relatively good in the effect of content removal, its practical application faces two major bottlenecks: first, each target content needs to be trained separately, resulting in the consumption of time and resources; second, this fine-tuning process is difficult to take into account both the accurate removal of the target content and the protection of the irrelevant content generation process, which is easy to have a negative impact on the generation quality. This also directly leads to the poor performance of these methods in the scene of eliminating multiple target contents. The second type of method is the method that does not require additional training. The method that does not require training achieves content removal by directly intervening in the image generation process, such as the safe latent diffusion model (Safe Latent Diffusion, SLD) and negative text prompts. This type of method can significantly reduce the computational cost and does not require time to fine-tune. However, its shortcomings are also very prominent: the removal effect is usually difficult to accurately control, especially in terms of protecting the quality of irrelevant content generation. Summary of the invention

[0004] In this embodiment, an adaptive content eliminator, elimination method, electronic device and storage medium for a large image generation model are provided to accurately remove target content that is unsafe or involves copyright issues during the image generation process, while ensuring that there is no negative impact on the generation process of other irrelevant content.

[0005] In a first aspect, an embodiment of the present invention provides an adaptive content elimination method for image generation of a large model, the adaptive content elimination method for image generation of a large model comprising: Constructing an elimination space and a retention space according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The text prompts input by the user are converted into semantic representations, including word segmentation of the text prompts, the generated words or phrases are used as tokens, and they are projected token by token into the elimination space and the retention space to obtain the retention component of the retention space; Measuring the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculating the relevance; According to the correlation results, the retained space is reconstructed and the direction of the retained component is corrected.

[0006] In an optional embodiment, an elimination space and a retention space are constructed according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained, including: Define the target content and obtain the semantic representation of the target content; For a single target content, the semantic representation of the target content is used as the basis vector to span the elimination space; For multiple target contents, the semantic representation of each target content is processed by Schmidt orthogonalization method to obtain a set of standard orthogonal bases, and the standard orthogonal bases are used to span the elimination space; The space that is orthogonal to the elimination space is used as the retention space.

[0007] In an optional embodiment, the text prompt input by the user is converted into a semantic representation, including word segmentation of the text prompt, using the generated words or phrases as tokens, and projecting them token by token into the elimination space and the retention space to obtain the retention component of the retention space, including: Perform word segmentation on the text input by the user to generate independent words or phrases, use the generated words or phrases as tokens, encode the tokens using the pre-trained language model, and generate semantic representations; For a single target content, calculating the projection on the elimination space and the retention space to obtain the elimination component of the token in the elimination space and the retention component of the retention space; For multiple target contents, projection calculation is performed on each target content to obtain a total eliminated component and a retained component; The calculated retained component replaces the original semantic representation and participates in the subsequent generation process.

[0008] In an optional embodiment, the token includes a special token, and the special token is not projected.

[0009] In an optional embodiment, measuring the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content and calculating the relevance includes: Obtaining the semantic representation of the current text prompt and the semantic representation of the target content; The cosine similarity is used to calculate the correlation between the semantic representation of the text prompt and the semantic representation of the target content.

[0010] In an optional embodiment, reconstructing the reserved space and correcting the direction of the reserved component according to the correlation result includes: When the correlation is greater than the preset threshold, it means that the correlation between the semantic representation of the text prompt and the semantic representation of the target content is strong, and the orthogonal constraints of the retained space and the eliminated space are strengthened; When the correlation is less than the preset threshold, it means that the correlation between the semantic representation of the text prompt and the semantic representation of the target content is weak or irrelevant, then the orthogonal constraints of the retained space and the eliminated space are relaxed to mitigate the negative impact of the elimination of the target content on the generation of irrelevant content.

[0011] In an optional embodiment, reconstructing the reserved space and correcting the direction of the reserved component according to the correlation result further includes: For a single target content, the direction of the retained component is directly corrected according to the cosine similarity between the semantic representation of the target content and the semantic representation corresponding to the current text prompt; For multiple target contents, the projection of the currently input text prompt along the direction corresponding to each target content is scaled according to the similarity, and the direction of the retained component is corrected along each target content.

[0012] Compared with the prior art, the adaptive content elimination method for image generation large model of the present invention has the following beneficial effects: This method can effectively remove single or multiple target contents from the generated image without consuming a lot of computing resources to fine-tune the model and without introducing additional parameters. While successfully removing single or multiple target contents, the impact on the generation of irrelevant content is negligible, significantly improving the content removal performance of the generated model.

[0013] In a second aspect, an embodiment of the present invention provides an adaptive content remover for an image generation large model, comprising: A projection module, the projection module comprising a semantic space construction unit and a token-by-token projection unit, the semantic space construction unit is used to construct an elimination space and a retention space according to the target content, the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The token-by-token projection unit is used to convert the text prompt input by the user into a semantic representation, including word segmentation processing of the text prompt, using the generated words or phrases as tokens, and projecting them token by token into the elimination space and the retention space to obtain the retention component of the retention space; An adaptive direction correction module, the adaptive direction correction module comprising a relevance capture unit and a direction correction unit, the relevance capture unit is used to measure the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculate the relevance; The direction correction unit is used to reconstruct the reserved space and correct the direction of the reserved component according to the correlation result.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the adaptive content elimination method for image generation of a large model as described in the first aspect.

[0016] Compared with the prior art, the beneficial effects of the adaptive content eliminator, electronic device and storage medium for image generation large model of the present invention are the same as the adaptive content elimination method for image generation large model described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of an adaptive content elimination method for generating a large model for an image in an embodiment of the present invention; Figure 2 It is a principle diagram of applying the adaptive content elimination method for image generation large model in an embodiment of the present invention to the image generation large model; Figure 3 It is a structural block diagram of an adaptive content remover for image generation of a large model in an embodiment of the present invention; Figure 4 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0020] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] In an embodiment of the present invention, an adaptive content elimination method for image generation large model is provided. Figure 1 Flowchart of the adaptive content elimination method for image generation large model of the present invention, such as Figure 1 As shown, the process includes the following steps: S100, constructing a removal space and a retention space according to target content, wherein the removal space corresponds to semantic information that needs to be removed, and the retention space corresponds to semantic information that needs to be retained; First of all, it should be noted that common text image generation models usually contain a compression network for compressing images into a latent space, and a UNet that gradually denoises from randomly sampled Gaussian noise in the latent space. UNet is the core component of text image generation, in which the cross attention mechanism is used to interact between text and image representations, so that text information can guide the visual content of the generated image to be consistent with the text prompt.

[0022] In this embodiment, the first step is to construct semantic subspaces (elimination space and retention space). According to the semantic information of the target content, two subspaces are constructed in the semantic space of the generated model: an elimination space representing semantic information related to the target content to be removed and a retention space representing semantic information irrelevant to the target content. It is ensured that the two subspaces are completely independent and complementary and can cover the semantic range in the current space.

[0023] In the cross-attention mechanism, the Value space determines some content information in the generated image, so we choose the Value space as the semantic space for layer-by-layer projection. The overall semantic space is , the corresponding reserved space and eliminated space are recorded as: and .

[0024] Specifically, an elimination space and a retention space are constructed according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained, including: Define the target content and obtain the semantic representation of the target content; clarify what content needs to be removed from the generated image. For example, if the goal is to remove copyrighted works of art, the characteristics of these works will be defined as the target content.

[0025] For a single target content, the semantic representation of the target content is used as the basis vector to span the elimination space; this means that all semantic information related to the target content will be projected into this space.

[0026] For multiple target contents, the semantic representation of each target content is processed by the Schmidt orthogonalization method to obtain a set of standard orthogonal bases, which are used to span the elimination space; the purpose of this is to ensure that the semantic information between different target contents will not interfere with each other and facilitate subsequent projection calculations.

[0027] The space that is orthogonal to the eliminated space is used as the retained space. It contains all semantic information that does not belong to the target content. That is, it is the orthogonal complementary space of the eliminated space, ensuring that the entire original semantic space can be completely covered by these two subspaces and do not overlap.

[0028] Once the elimination space and the retention space are established, the next step is to project the semantic representations obtained by the input text prompt in different layers and time steps of the generative model into these two subspaces token by token, as follows: S200, converting the text prompt input by the user into a semantic representation, including word segmentation of the text prompt, using the generated words or phrases as tokens, and projecting them token by token into the elimination space and the retention space to obtain the retention component of the retention space; Specifically, the text prompts input by the user are converted into semantic representations, including word segmentation of the text prompts, and the generated words or phrases are used as tokens, and are projected token by token into the elimination space and the retention space to obtain the retention components of the retention space, including: The text input by the user is segmented to generate independent words or phrases (tokens), and the generated words or phrases (tokens) are used as tokens. The tokens are encoded using a pre-trained language model to generate semantic representations. First, the text prompts input by the user are segmented to break them down into independent words or phrases (i.e. tokens), and pre-trained language models (such as CLIP, BERT, etc.) are used to encode these tokens to generate high-dimensional vector representations corresponding to each token, that is, their semantic representations.

[0029] It should be noted that the tokens include special tokens, which are not projected. For example, for some language models, the first token may be a special symbol (such as [CLS]) to capture the information of the entire sentence, and the last few tokens may also be special fillers. Decide whether to project these special tokens based on needs. For example, in actual operations, prefix tokens do not contain discriminative semantic information, so they are not projected; at the same time, since the text of the target content is short, its corresponding text representation contains a large number of suffix tokens, and the information contained in the suffix tokens is often redundant, so the last valid token of the target content will be used to copy and overwrite the suffix token for subsequent calculations.

[0030] For a single target content, the projections on the elimination space and the retention space are calculated to obtain the elimination component of the token in the elimination space and the retention component of the retention space; that is, through the orthogonal projection formula in linear algebra, the purpose is to find the components of the token in the two subspaces (elimination component and retention component).

[0031] For multiple target contents, projection calculation is performed on each target content to obtain a total eliminated component and a retained component; The calculated retained component replaces the original semantic representation to participate in the subsequent generation process. While removing unnecessary target content, the integrity of other non-target content is maintained.

[0032] Specifically, combined Figure 2 As shown, the corresponding representation of the current text prompt in this semantic space , where l represents the number of tokens contained in the semantic representation. Represents the feature dimension of each token. By representing the semantics of the text hint Project to the reserved space and eliminate space By using the complementary and independent properties of the two, we can get: ,in The corresponding retained component in the semantic representation, is the corresponding eliminated component in the semantic representation, and In order to achieve more flexible and accurate elimination, Perform token-by-token projection, remember , , They are , , For the jth token of , we have: ; ; ; in =[ ] represents the semantic representation of the target content in the semantic space when a single target content is eliminated. In actual operation, the first token is a prefix token, which does not contain discriminative semantic information, so no projection is performed. In addition, since the text of the target content is short, its corresponding text representation contains a large number of suffix tokens, and the information contained in the suffix tokens is often redundant. To avoid redundancy, we use the last token corresponding to the text representation of the target content to copy and overwrite the suffix token to calculate the semantic representation, so as to perform subsequent projection, that is, use Alternative Participate in the calculation, that is, The calculated semantic representation is used as the basis of the Zhangcheng elimination space to ensure that the semantic information related to the target content is effectively eliminated at all positions. Finally, the retained components after token-by-token projection are obtained. =[ ]replace =[ ] participate in subsequent calculations. When multiple target contents are eliminated, the representation of each target content in the semantic space can be recorded as { }, then the corresponding elimination space is composed of the subspace formed by the tokens at the corresponding positions of the target content, and the retained space is its complementary independent space. That is: ; in: ; You can get: ; S300, measuring the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculating the relevance; Measure the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculate the relevance, including: Obtain the semantic representation of the current text prompt and the semantic representation of the target content; convert the text prompt input by the user into a representation in a high-dimensional vector space through a pre-trained language model (such as CLIP, BERT, etc.). Similarly, use the same or compatible language model to encode all target contents that need to be removed to obtain their respective semantic representations.

[0033] It should be noted that in order to ensure the consistency and comparability of the calculation results, these semantic representations are usually normalized so that the modulus of each vector is equal to 1. This can avoid the distance measurement deviation caused by the difference in vector length.

[0034] The cosine similarity is used to calculate the correlation between the semantic representation of the text prompt and the semantic representation of the target content.

[0035] S400: Reconstruct the reserved space and correct the direction of the reserved component according to the correlation result.

[0036] First, one or more thresholds are set to distinguish between high and low relevance situations. Based on the calculated relevance, the strength of the relationship between the current text prompt and the target content is determined.

[0037] Specifically, according to the correlation results, the reserved space is reconstructed and the direction of the reserved component is corrected, including: When the correlation is greater than the preset threshold, it means that the semantic representation of the text prompt and the semantic representation of the target content are highly correlated, and the orthogonal constraints of the retained space and the eliminated space are strengthened to ensure more accurate removal of the target content. This means that the independence and complementarity of the two subspaces must be maintained more strictly.

[0038] When the correlation is less than the preset threshold, it means that the correlation between the semantic representation of the text prompt and the semantic representation of the target content is weak or irrelevant, then the orthogonal constraints of the retained space and the eliminated space are relaxed to mitigate the negative impact of the elimination of the target content on the generation of irrelevant content.

[0039] According to the correlation results, the reserved space is reconstructed and the direction of the reserved component is corrected, which also includes: For a single target content, the direction of the retained component is directly corrected according to the cosine similarity between the semantic representation of the target content and the semantic representation corresponding to the current text prompt; For multiple target contents, the projection of the currently input text prompt along the direction corresponding to each target content is scaled according to the similarity, and the direction of the retained component is corrected along each target content.

[0040] According to the result of the correlation, the basis vectors used to span the reserved space are dynamically adjusted. For high correlation, more orthogonalization may be required to ensure that the new basis vectors better represent the semantic information of non-target content; for low correlation, some restrictions can be relaxed to allow some degree of overlap, but not to the extent that the overall effect is affected.

[0041] Specifically, when eliminating a single target content, the direction correction is directly performed according to the cosine similarity between the semantic representation of the target content and the semantic representation corresponding to the current text prompt, that is: ; ; in, Indicates the cosine similarity between the target content and the jth token of the current text prompt, is the hyperparameter of the direction correction unit, which directly corrects the direction of the retained component.

[0042] When multiple target contents are eliminated, similar to the case of erasing a single target content, the projection of the currently input text hint along the direction corresponding to each target content needs to be scaled according to the similarity, that is, the direction of the retained component is corrected along each target content, that is: ; ; Through the above steps, the retained space is reconstructed according to the correlation results and the direction of the retained component is corrected. This method not only ensures the accurate elimination of the target content, but also protects the generation quality of irrelevant content from interference to the maximum extent.

[0043] In order to illustrate the effectiveness of the present invention, the following experiments were conducted for verification.

[0044] This method is applied to or the eliminator is embedded in Stable Diffusion V1.4 to explore its performance when eliminating single or multiple IP roles. At the same time, irrelevant IPs are used to verify that the adaptive content eliminator has little impact on irrelevant content generation:

[0045] Table 1 Performance comparison when eliminating IP roles As can be seen from Table 1, for the target content that needs to be eliminated, whether it is single or multiple, the adaptive content elimination method for the large image generation model shows content elimination performance. At the same time, the generated images of irrelevant content are almost unaffected, and the FID is much lower than other methods, less than half of other methods, showing outstanding prior protection capabilities, and this protection capability for irrelevant content generation is not affected by the increase in eliminated content.

[0046] This method is applied to Stable Diffusion V1.4 to explore its performance in eliminating painting styles as follows:

[0047] Table 2 Performance comparison when eliminating painting styles As can be seen from Table 2, when eliminating a certain painting style, NP shows the best elimination result, but NP has a great impact on irrelevant content while eliminating the target content. Our adaptive content elimination method for large image generation models shows extremely strong accuracy in the process of content elimination, and solves the impact of content elimination on irrelevant content generation without losing elimination performance, achieving the best trade-off between target content elimination and irrelevant content protection.

[0048] This method is applied to Stable Diffusion V1.4 to explore its performance in removing unsafe content (nudity) as follows:

[0049] Table 3 Performance comparison when eliminating unsafe content As can be seen from Table 3, for unsafe content, our adaptive content removal method for large image generation models shows the best content removal performance, which can eliminate unsafe content to the greatest extent and prevent the generation of unsafe visual information.

[0050] The embodiment of the present invention also provides an adaptive content eliminator for image generation large model, which is used to implement the above method embodiment, and the description has been made no further. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0051] like Figure 3 As shown, Figure 3 : is a block diagram of an adaptive content eliminator for image generation of a large model in the present invention, the eliminator comprising: A projection module, the projection module comprising a semantic space construction unit and a token-by-token projection unit, the semantic space construction unit is used to construct an elimination space and a retention space according to the target content, the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The token-by-token projection unit is used to convert the text prompt input by the user into a semantic representation, including word segmentation processing of the text prompt, using the generated words or phrases as tokens, and projecting them token by token into the elimination space and the retention space to obtain the retention component of the retention space; An adaptive direction correction module, the adaptive direction correction module comprising a relevance capture unit and a direction correction unit, the relevance capture unit is used to measure the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculate the relevance; The direction correction unit is used to reconstruct the reserved space and correct the direction of the reserved component according to the correlation result.

[0052] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: Constructing an elimination space and a retention space according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The text prompts input by the user are converted into semantic representations, including word segmentation of the text prompts, the generated words or phrases are used as tokens, and they are projected token by token into the elimination space and the retention space to obtain the retention component of the retention space; Measuring the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculating the relevance; According to the correlation results, the retained space is reconstructed and the direction of the retained component is corrected.

[0053] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0054] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including: Constructing an elimination space and a retention space according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The text prompts input by the user are converted into semantic representations, including word segmentation of the text prompts, the generated words or phrases are used as tokens, and they are projected token by token into the elimination space and the retention space to obtain the retention component of the retention space; Measuring the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculating the relevance; According to the correlation results, the retained space is reconstructed and the direction of the retained component is corrected.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive content elimination method for image generation large model, characterized in that: The elimination method comprises: Constructing an elimination space and a retention space according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The text prompts input by the user are converted into semantic representations, including word segmentation of the text prompts, the generated words or phrases are used as tokens, and they are projected token by token into the elimination space and the retention space to obtain the retention component of the retention space; Measuring the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculating the relevance; According to the correlation results, the retained space is reconstructed and the direction of the retained component is corrected.

2. The adaptive content elimination method for image generation large model according to claim 1, characterized in that: Constructing an elimination space and a retention space according to the target content, wherein the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained, including: Define the target content and obtain the semantic representation of the target content; For a single target content, the semantic representation of the target content is used as the basis vector to span the elimination space; For multiple target contents, the semantic representation of each target content is processed by Schmidt orthogonalization method to obtain a set of standard orthogonal bases, and the standard orthogonal bases are used to span the elimination space; The space that is orthogonal to the elimination space is used as the retention space.

3. The adaptive content elimination method for image generation large model according to claim 1, characterized in that: The text prompts input by the user are converted into semantic representations, including word segmentation of the text prompts, and the generated words or phrases are used as tokens, and are projected token by token into the elimination space and the retention space to obtain the retention components of the retention space, including: Perform word segmentation on the text input by the user to generate independent words or phrases, use the generated words or phrases as tokens, encode the tokens using the pre-trained language model, and generate semantic representations; For a single target content, calculating the projection on the elimination space and the retention space to obtain the elimination component of the token in the elimination space and the retention component of the retention space; For multiple target contents, projection calculation is performed on each target content to obtain a total eliminated component and a retained component; The calculated retained component replaces the original semantic representation and participates in the subsequent generation process.

4. The adaptive content elimination method for image generation large model according to claim 3, characterized in that: The tokens include special tokens, and the special tokens are not projected.

5. The adaptive content elimination method for image generation large model according to claim 1, characterized in that: Measure the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculate the relevance, including: Obtaining the semantic representation of the current text prompt and the semantic representation of the target content; The cosine similarity is used to calculate the correlation between the semantic representation of the text prompt and the semantic representation of the target content.

6. The adaptive content elimination method for image generation large model according to claim 1, characterized in that: According to the correlation results, the reserved space is reconstructed and the direction of the reserved component is corrected, including: When the correlation is greater than the preset threshold, it means that the correlation between the semantic representation of the text prompt and the semantic representation of the target content is strong, and the orthogonal constraints of the retained space and the eliminated space are strengthened; When the correlation is less than the preset threshold, it means that the correlation between the semantic representation of the text prompt and the semantic representation of the target content is weak or irrelevant, then the orthogonal constraints of the retained space and the eliminated space are relaxed to mitigate the negative impact of the elimination of the target content on the generation of irrelevant content.

7. The adaptive content elimination method for image generation large model according to claim 6, characterized in that: According to the correlation results, the reserved space is reconstructed and the direction of the reserved component is corrected, which also includes: For a single target content, the direction of the retained component is directly corrected according to the cosine similarity between the semantic representation of the target content and the semantic representation corresponding to the current text prompt; For multiple target contents, the projection of the currently input text prompt along the direction corresponding to each target content is scaled according to the similarity, and the direction of the retained component is corrected along each target content.

8. An adaptive content remover for large image generation models, characterized in that: include: A projection module, the projection module comprising a semantic space construction unit and a token-by-token projection unit, the semantic space construction unit is used to construct an elimination space and a retention space according to the target content, the elimination space corresponds to the semantic information to be removed, and the retention space corresponds to the semantic information to be retained; The token-by-token projection unit is used to convert the text prompt input by the user into a semantic representation, including word segmentation processing of the text prompt, using the generated words or phrases as tokens, and projecting them token by token into the elimination space and the retention space to obtain the retention component of the retention space; An adaptive direction correction module, the adaptive direction correction module comprising a relevance capture unit and a direction correction unit, the relevance capture unit is used to measure the semantic distance between the semantic representation of the current text prompt and the semantic representation of the target content, and calculate the relevance; The direction correction unit is used to reconstruct the reserved space and correct the direction of the reserved component according to the correlation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the adaptive content elimination method for image generation large model according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the adaptive content elimination method for image generation large model as described in any one of claims 1 to 7 are implemented.

Images