Generative model content control method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a generative model content control method and device, equipment and a storage medium, relates to the technical field of visual models, and is suitable for financial science and technology and medical health scenes. According to the method, a semantic dictionary is defined according to a de-embedding matrix of a generative model, specialization scores of attention heads in the generative model to sub-dictionaries in the semantic dictionary are calculated by adopting a greedy algorithm of synchronous orthogonal matching pursuit, and then key heads are selected according to the specialization scores. And then obtaining directional editing information and an output signal of the key head, and performing weighting coefficient calculation on the output signal by using the directional editing information, so that the generative model outputs a directional editing result. According to the method, the functions of the specific components in the generative model can be quantified through sparse decomposition, the content generation result of the model is efficiently controlled by manipulating very few key components, and reliable and accurate control over the output content of the model is achieved under the condition that additional training is not needed.

Description

Technical Field

[0001] This application relates to the field of visual modeling technology, and in particular to a generative model content control method, apparatus, device, and storage medium. Background Technology

[0002] In fields such as finance and insurance, artificial intelligence (AI) models can be applied to process business data, adapting to scenarios such as content moderation, attribute enhancement, and model security. For example, large generative models based on the Transformer architecture, such as Large Language Models (LLMs) and Multilingual Large Language Models (MLLMs), can be used for natural language processing and visual understanding tasks.

[0003] The Transformer is a neural network architecture that uses a self-attention mechanism to allow the model to weigh the importance of different parts of the data while processing it, resulting in a more accurate model output. The self-attention mechanism consists of many parallel computational units, known as attention heads. Each attention head can learn a specific function or pattern, such as tracking grammatical relationships or recognizing specific types of words.

[0004] However, the internal workings of attention-head-based models remain largely black boxes when processing data; that is, their internal mechanisms are difficult to understand. While it's known that the model can complete the task, it's hard to know which internal components are responsible for which specific functions. For example, which attention heads generate color-related words. This lack of interpretability directly leads to difficulties in precisely controlling the model's output, making it difficult to suppress toxic content or enhance specific desired attributes. Summary of the Invention

[0005] In view of this, embodiments of this application provide a generative model content control method, apparatus, device, and storage medium to solve the problem of inability to accurately control model content generation.

[0006] According to a first aspect of this application, a generative model content control method is provided, the method comprising: A semantic dictionary is defined based on the de-embedding matrix of the generative model, and the semantic dictionary includes at least one sub-dictionary for representing target concepts of the same type; Calculate the specialization score of the attention head to the sub-dictionary in the generative model. The specialization score is the variance ratio calculated using a greedy algorithm of synchronous orthogonal matching pursuit. The specialization score is used to characterize the dictionary atomic indexes of the sub-dictionary that can interpret the output signal of the attention head. Key heads are selected based on the specialization score, wherein the key head is at least one attention head that is related to the target concept; Acquire directional editing information and the output signal of the key head; The output signal is weighted using the directional editing information so that the generative model outputs the directional editing result.

[0007] According to a second aspect of this application, a generative model content control device is provided, the device comprising: A dictionary definition module is used to define a semantic dictionary based on the de-embedding matrix of the generative model, wherein the semantic dictionary includes at least one sub-dictionary for representing target concepts of the same type; The specialization detection module is used to calculate the specialization score of the attention head to the sub-dictionary in the generative model. The specialization score is the variance ratio calculated using a greedy algorithm of synchronous orthogonal matching pursuit. The specialization score is used to characterize the dictionary atomic indexes that the sub-dictionary can interpret the output signal of the attention head. A head selection module is used to select key heads based on the specialization score, wherein the key head is at least one attention head related to the target concept; The directional editing module is used to acquire directional editing information and the output signal of the key head; The result output module is used to perform weighted coefficient calculation on the output signal using the directional editing information, so that the generative model outputs the directional editing result.

[0008] According to a third aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described generative model content control method.

[0009] According to a fourth aspect of this application, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described generative model content control method.

[0010] By employing the above technical solutions, embodiments of this application provide a method, apparatus, device, and storage medium for content control of a generative model. The method first defines a semantic dictionary based on the de-embedding matrix of the generative model, and then uses a greedy algorithm of synchronous orthogonal matching pursuit to calculate the specialization score of the attention head in the generative model relative to the sub-dictionary in the semantic dictionary. Next, key heads are selected based on the specialization scores. Then, directional editing information and the output signals of the key heads are acquired, and weighted coefficients are calculated on the output signals using the directional editing information to enable the generative model to output directional editing results. This method quantifies the function of specific components in the generative model through sparse decomposition and efficiently controls the content generation results of the model by manipulating a very small number of key components, achieving reliable and precise control of the model's output content without additional training.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic flowchart of the generative model content control method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the generative model content generation process provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of setting intervention coefficients according to intervention type, provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of the generative model content control device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the computer device structure provided in an embodiment of this application. Detailed Implementation

[0013] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0014] In this application embodiment, the generative model, also known as a content generation model, is a type of artificial intelligence (AI) model. The content generation model can generate various types of content, including text, images, audio, and video, based on input prompts. Content generation models can be applied to business data processing in scenarios such as fintech and healthcare to replace manual processes for generating and processing business data, including but not limited to content moderation, attribute enhancement, and model security.

[0015] Taking content generation models in scenarios such as fintech and healthcare as examples, in the fields of fintech and healthcare, large generative models based on the Transformer architecture, such as Large Language Model (LLM) and Multilingual Large Language Model (MLLM), can be used for natural language processing and visual understanding tasks.

[0016] The Transformer is a neural network architecture that uses a self-attention mechanism to allow the model to weigh the importance of different parts of the data while processing it, resulting in a more accurate model output. The self-attention mechanism consists of many parallel computational units, known as attention heads. Each attention head can learn a specific function or pattern, such as tracking grammatical relationships or recognizing specific types of words.

[0017] When processing data, the internal workings of attention-head-based models remain largely black boxes, meaning their internal mechanisms are difficult to understand. While it's known the model can complete the task, it's hard to know which internal components are responsible for which specific functions. For example, which attention heads generate color-related words. This lack of interpretability directly leads to difficulties in precisely controlling the model's output, making it hard to suppress toxic content or enhance specific desired attributes.

[0018] To enable control over the model's output, in some embodiments, the input data generated by the model can be optimized based on prompt engineering. By defining prompts, more accurate input instructions can be formed to guide the model to output result data in a targeted manner, thereby enabling targeted editing.

[0019] For example, a verbal instruction such as "Please don't say anything toxic" can be used to guide the model in content generation. However, this control method has low reliability; the content generation model may still forget the input instructions related to the prompt in some cases, making it impossible to achieve precise control over the generated content.

[0020] In some embodiments, the content generation model can be gradually trained through model fine-tuning to make its output more closely reflect the current business scenario. During model fine-tuning, a specific dataset can be used to perform additional training on the content generation model, enabling it to be tailored to the specific business scenario on top of a general model, thereby improving the accuracy of the output results.

[0021] Because controlling the model output through model fine-tuning requires a large amount of sample data that matches the current scenario for model fine-tuning training, this control method is costly, requiring a lot of computing resources and data. Furthermore, as model fine-tuning progresses, it may lead to catastrophic forgetting in the content generation model, causing the content generation model's ability to decline in other tasks.

[0022] In some embodiments, the output of the content generation model can also be controlled through heuristic analysis. That is, by using interpretable tools, such as Logit Lens, the intermediate states of the model can be detected, and the mechanism by which the model affects the input data can be analyzed based on the intermediate states. Then, directional adjustments can be made based on the mechanism, so that the content generation model can generate content according to the adjustment direction specified by the user.

[0023] However, interpretability tools rely on heuristics for targeted adjustments based on intermediate states, analyzing only a single sample. This makes it difficult to draw statistically significant and generalizable conclusions about the model's mechanism of action, thus reducing the model's adaptability.

[0024] In some embodiments, the functionality of each attention head in the content generation model can be determined by training external probes. This involves training a separate classifier ("probe") for each attention head in the content generation model to determine its functionality. However, training external probes is computationally intensive and difficult to scale to massive models with tens of thousands of attention heads. Therefore, artificial intelligence models used in business domains such as finance, insurance, and healthcare require a simple, low-cost, scalable strategy that requires no additional training to accurately understand and control the internal mechanisms of these models.

[0025] To address the issue of inaccurate control over model content generation, some embodiments of this application provide a method for controlling the content of a generative model. This method utilizes the generative model's built-in lexical mapping (de-embedding matrix) as a semantic dictionary, and identifies a few key attention heads most relevant to a specific concept using a matching pursuit algorithm. Thus, by intervening only in these few attention heads, reliable and precise control over the model's output content can be achieved without additional training.

[0026] The method can be applied to electronic devices with data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this application embodiment uses an electronic device as the execution subject of the method. It should be understood that the method can also be applied to other types of execution subjects, which are not illustrated in this application embodiment. Figure 1 As shown, the method includes: S101. Define a semantic dictionary based on the de-embedding matrix of the generative model.

[0027] like Figure 2 As shown, in order to achieve targeted editing control over the content generation process of the generative model, this application embodiment improves the content generation process of the generative model. The improved content generation process can include two stages: an offline preparation stage and a real-time application stage. The offline preparation stage is used for data analysis and functional relevance scoring, and therefore can also be called the analysis and scoring stage. The real-time application stage is used for intervention and control of the content generation process, and therefore is also called the intervention and control stage.

[0028] During the analysis and scoring phase, a semantic dictionary can be defined first based on the unembedding matrix of the generative model. This semantic dictionary includes at least one sub-dictionary for representing similar target concepts. The structure of the generative model itself can be utilized. At the back-end layer of the generative model within the Transformer architecture, an unembedding matrix can be used to map the complex vector representations within the model back to a readable vocabulary.

[0029] The vocabulary can include multiple text tokens, which are the basic units for the model to process text. A token can be a complete word, such as "apple," or a subword, such as "un-" or "-able." Each row in the deembedding matrix can be viewed as a semantic vector representation of a token within the model. Therefore, this complete deembedding matrix D can be considered as a general dictionary containing all semantic concepts.

[0030] S102. Calculate the specialization score of the attention head pair sub-dictionary in the generative model.

[0031] After defining the semantic dictionary based on the de-embedding matrix of the generative model, the functionality of the components in the generative model can be quantified through sparse decomposition. The specialization score of the attention head to the sub-dictionary in the generative model can be calculated to find out which attention head h is most specialized for which concept (such as color).

[0032] The specialization score is the variance ratio calculated using a greedy algorithm of Simultaneous Orthogonal Matching Pursuit (SOMP), and the specialization score is used to characterize the dictionary atomic indexes that the sub-dictionary can interpret the attention head output signal.

[0033] Sparse decomposition can model the functional quantization problem as a sparse signal recovery problem. The objective of the sparse signal recovery problem is to approximate the output of an attention head h as a linear combination of a few token semantic vectors in the total dictionary D. Therefore, a greedy algorithm of synchronous orthogonal matching pursuit can be used to find sparse combinations.

[0034] To evaluate the specialization of an attention head for a specific concept, the attention head h is compared with a color sub-dictionary consisting of tokens such as red and blue. D concept This sub-dictionary can be calculated. D concept The proportion of variance that can be explained by the output signal of the attention head h is used as the specialization score of the attention head to the target concept.

[0035] In some embodiments, when calculating the specialization score of the attention head pair to the sub-dictionary in a generative model, semantic vectors can be extracted from the semantic dictionary first, and the output matrix of the attention head at the current step can be obtained. The output matrix is ​​the matrix representation obtained by each attention head after processing the input data. To obtain the output matrix of the attention head at the current step, an input dataset can be obtained, which includes multiple sample data containing concept elements corresponding to the target concept. The input dataset is then input into the generative model, and the output matrix of each attention head is recorded when the generative model processes each sample data.

[0036] For example, in the image description control process of a Vision-Language Model (VLM), when controlling a VLM to enhance or suppress color descriptions in image description, the concept of "color" can be defined first during the analysis and scoring phase. To do this, all color-related tokens, such as red, purple, black, and green, can be selected from the model's vocabulary (token list) to construct a color sub-dictionary, denoted as D. {colors} .

[0037] Next, signal collection is performed, which involves preparing an input dataset containing thousands of images and feeding this dataset into the VLM. When the VLM processes each image, the output activation of each attention head in the model can be recorded, i.e., the output matrix H. For example, the Mistral-7B model has 4096 attention heads; by recording the output activation of these 4096 attention heads, 4096 output matrices can be obtained.

[0038] After obtaining the output matrix, the residual matrix between the current step's output matrix and the previous step's sparse reconstruction result can be calculated. For example, the output of an attention head h on n different samples can be represented as an output matrix, i.e.:

[0039] in, H Indicates the output matrix; n Indicates the number of data points; d This represents the internal dimension of the generative model.

[0040] At the same time, the de-embedding matrix of the model is used ( v (The size of the vocabulary) serves as the semantic dictionary. This is achieved by finding a column-sparse coefficient matrix. , making To achieve this goal, a synchronous orthogonal matching pursuit greedy algorithm can be used. For each current step (t step) in the algorithm, a dictionary is selected. D The middle and current residuals The most relevant atom, i.e., the semantic vector of the token.

[0041] When calculating the residual matrix, the sparse reconstruction result of the previous step (t-1) can be obtained from the current step (t step). H r t Then calculate the output matrix of the current step. H Compared with the previous sparse reconstruction results H rt The difference is used to obtain the residual matrix. R t ,Right now R t = H - H r t .

[0042] Then, based on the semantic vector and the residual matrix, the dictionary atom index selected in the current step is calculated, where the dictionary atom index is the set of independent variable points with the maximum L1 norm between a single semantic vector and the residual matrix of all samples.

[0043] For example, the formula for calculating the dictionary atomic index in the atomic selection step of the synchronous orthogonal matching pursuit greedy algorithm can be expressed as:

[0044] in, This indicates the dictionary atomic index selected at step t, where the dictionary atomic index corresponds to a token in the vocabulary; D [ j ] represents a dictionary D The first in j The row, i.e. the j-th row in the embedding matrix, represents the d-dimensional semantic vector of the j-th token; This indicates that the output matrix H of the current step is compared with the sparse reconstruction result H of the previous step. r t The residual matrix between; Denotes the L1 norm, used to aggregate the evaluation of a single token vector D[j] with the residuals R of all samples. t The overall correlation between them.

[0045] Therefore, by iteratively calculating based on the dictionary's atomic indices, multiple text tokens capable of explaining the attention head function can be obtained. Then, the variance ratio of these multiple text tokens is calculated based on the target concept to obtain a specialization score.

[0046] To obtain a specialization score, in some embodiments, when calculating the variance ratio of multiple text tokens based on the target concept, multiple related tokens associated with the target concept can be extracted from the semantic dictionary to construct a target concept sub-dictionary. Then, the percentage of the total variance of the output matrix that the text tokens corresponding to the target concept sub-dictionary can explain is calculated to obtain the specialization score. Finally, related concept tags are set for the attention head based on the specialization score.

[0047] For example, after calculating the dictionary atomic index based on the above embodiments, iterative calculations can be performed based on the dictionary atomic index, that is, by executing the above process k times, the k tokens that best explain the function of the attention head H can be found. Furthermore, to evaluate the specialization of a head for a specific concept, the dictionary D can be first restricted to a sub-dictionary containing only color-related tokens. .

[0048] Then run the above algorithm and calculate the percentage of the total variance of the output matrix H that this set of tokens can ultimately explain, as the color specialization score for that header. For an output matrix H containing n data points, the total variance of the output matrix H can be calculated first. The total variance can be obtained by calculating the trace of the covariance matrix of the output matrix H. During the calculation, the mean-centered matrix H can be calculated first. centered H centered =H H1 / n, where H1 is a vector of all 1s with dimension n×1. Then calculate H... centered The covariance matrix C H C H =1 / (m) 1) H centered T H centered Then calculate the covariance matrix C. H The trace, i.e., the sum of the diagonal elements, is Total Variance = Tr(C H ).

[0049] Then calculate the sub-dictionary D. concept To calculate the explained variance of the output matrix H, we can first calculate the variance of H in D. concept The projection on, i.e., H projected =HD concept D concept T Among them, D concept D concept T It is a projection matrix that projects H onto D. concept Within the spanned subspace, calculate H again. projected The total variance, i.e., H projected, centered =H projected H projected 1 / n; C Hprojected =H projected, centered T H projected, centered / (n 1);Explained Variance=Tr(C Hprojected ).

[0050] Next, calculate the variance explained ratio, which can be obtained by Variance Explained Ratio = Explained Variance / Total Variance.

[0051] S103. Select key leaders based on professional scores.

[0052] After calculating the specialization scores of attention heads paired with sub-dictionaries in a generative model, key heads can be selected based on these scores. A key head is at least one attention head that is relevant to the target concept. Specialization scores decompose the complex functionality of a single attention head into contributions to a set of sparse, interpretable semantic concepts. For example, when processing "a girl wearing red clothes," a grammatical head might focus on the relationship between "wearing" and "girl"; a color head might be strongly activated by the token "red." Therefore, specialization scores can automatically and quantitatively identify this color head.

[0053] In some embodiments, to determine key heads, when selecting key heads based on specialization scores, the attention heads can first be sorted according to their specialization scores to obtain a sorted result sequence. This sorted result sequence includes the recognition information of multiple attention heads arranged in descending order of their specialization scores. Then, a preset number of attention heads (which can be an integer greater than or equal to 1) is obtained. The first preset number of attention heads are then selected from the sorted result sequence to obtain the key heads.

[0054] For example, a generative model includes 4096 attention heads. Following the method described in the above embodiments, each of these 4096 heads can be individually processed using a color sub-dictionary D. {colors} Run the sparse decomposition algorithm. After computation, each attention head will receive a color specialization score, such as L18.H27 having a color specialization score of 0.52, L25.H14 having a color specialization score of 0.01, and so on. Then, sort all attention heads in descending order of color specialization score to obtain the sorted result sequence.

[0055] A preset number k is set according to the precision requirements of the business scenario. That is, the preset number k is set to an integer greater than or equal to 1, such as k=16 or k=32. Then, the top-k heads most relevant to the target concept can be selected from the sorted result sequence. That is, the top-k attention heads with the highest scores are selected, and these k attention heads are marked as color key heads.

[0056] S104. Obtain the output signals of the directional editing information and key headers.

[0057] After selecting key headers based on professional scores, the process can proceed to the intervention and control phase of model-generated content, enabling real-time intervention in the content generation process. During real-time intervention, directional editing information and the output signals of key headers can be acquired for signal rescaling.

[0058] When performing signal rescaling, targeted editing information can be acquired first. This information may include specified toxic content, specific themes, emotional tone, etc. The targeted editing information can be actively input by the user. In some embodiments, the targeted editing information can be input data in the form of natural language text. When acquiring targeted editing information, the user-input natural language text can be received first, then a text processing model can be used to extract targeted editing fields from the natural language text, and finally, targeted editing information can be generated based on these fields. The targeted editing information may include the target concept and its targeted editing parameters.

[0059] For example, when a user inputs the generative model's suggested natural language text "a girl wearing a red dress," the text processing model can be invoked first to segment the natural language text, resulting in the keyword set "a / red / dress / girl." Then, keywords related to "color" are extracted from this keyword set to obtain the targeted editing field. Next, targeted editing parameters are set based on this field, specifically the parameters for the color red, to determine the intervention coefficient. For example, the default color intervention coefficient is 5.

[0060] Targeted editing information can also be obtained through other means. Specifically, in some embodiments, current scenario information can be obtained during the intervention and control phase. Then, based on this current scenario information, the key focus areas for the current business scenario are determined, and targeted editing information is set accordingly.

[0061] For example, during the intervention and control phase, if the user does not input any targeted editing information, the electronic device can obtain scene recognition information of the current business scenario, i.e., the application area of ​​the generative model, such as the customer service scenario in insurance business. In this scenario, the business data shows a high degree of attention to the color of the generated content, so the focus can be determined to be on color. In this case, the default color intervention coefficient can be set to 5.

[0062] During signal rescaling, the computational process of the generative model can also be monitored. That is, when the generative model is normally receiving user input (such as text or images) and performing inference, monitoring tools such as TensorBoard or LlamaBoard can be used to monitor the computational process. When the computational process of the generative model reaches the selected Top-k key headers in the first stage, the output signals of the k key headers are intercepted and used for weighted coefficient calculation.

[0063] S105. Use the directional editing information to perform weighted coefficient calculation on the output signal so that the generative model outputs the directional editing result.

[0064] After obtaining the directional editing information and the output signal of the key header, the output signal of the key header can be directionally edited based on the directional editing information. In order to obtain an output result that conforms to the directional editing information, directional editing can be performed according to the intervention coefficient corresponding to the directional editing information. That is, the output result of the generative model is controlled by calculating the product of the intervention coefficient and the output signal, so that the generative model will generate a precisely edited output result, which is called the directional editing result.

[0065] In some embodiments, to output the directed editing result, when calculating the weighted coefficients of the output signal using the directed editing information, the intervention coefficient can be calculated first based on the directed editing information, and then the product of the intervention coefficient and the output signal can be calculated to obtain the intervention signal. The intervention signal is then fed back into the computational flow of the generative model so that it propagates through subsequent layers of the generative model. Finally, the directed editing result is obtained by acquiring the output data of the generative model.

[0066] For example, a user uploads a new image to VLM containing a woman wearing a purple hat and a black jacket, and is prompted by natural language text to "describe this image." The intervention coefficient is determined by obtaining targeted editing information. α With a value of 5, the VLM's computational flow can be monitored when processing the input image and prompts. When 16 color keyheads are computed, their output signals are captured, resulting in 16 output matrices. These 16 keyhead output signal values ​​are then multiplied by 5 to form an amplified color signal. This amplified color signal is then fed back into the computational flow, propagating through subsequent layers of the generative model. The final output is the descriptive text "A woman is wearing a black jacket and a purple hat," thereby increasing the probability of color-related tokens being generated in the generative model.

[0067] By applying the technical solutions of the above embodiments, the generative model content control method described in the above embodiments can solve the problem of inaccurate understanding and control of generated content caused by the black-box nature of generative models. It achieves reliable suppression or enhancement of specific semantic attributes in the model output without high-cost fine-tuning or reliance on unreliable prompts. To this end, the method can be based on an analysis-then-intervention framework, quantifying the function of specific components (attention heads) in the model through sparse decomposition, and efficiently controlling the macroscopic behavior of the model by manipulating a very small number (sparse) key components.

[0068] The method is training-free and can be applied to any pre-trained Transformer model without any gradient updates or fine-tuning, significantly reducing application costs. It also achieves efficient and precise control, allowing editing of specific concepts by intervening in a tiny fraction (less than 1% of the total) of the most critical attention heads. This control is highly effective with minimal side effects on other model capabilities. By shifting interpretability from heuristic single-sample guessing to quantitative analysis based on sparse decomposition and multi-sample statistics, it provides a more robust and scalable way to understand the model's internal functions, enabling principled interpretability. Furthermore, the method is task-independent, applicable to both pure text models (e.g., question answering, toxicity mitigation) and multimodal models (e.g., image classification, image captioning), demonstrating strong versatility.

[0069] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a generative model content control method. The difference between this method and the above embodiments lies in that, when using directional editing information to perform weighted coefficient calculation on the output signal, different directional interventions can be performed according to different directional editing information. That is, when calculating intervention coefficients based on directional editing information, different intervention coefficients can be set according to the intervention type. For example... Figure 3 As shown, the method includes: S201. Extract type-related fields from the targeted editing information; S202. Determine the intervention type for targeted editing information based on the type-related fields; S203. If the intervention type is an enhanced intervention, set the intervention coefficient to the enhanced coefficient. S204. If the intervention type is a suppressive intervention, set the intervention coefficient to the suppressive coefficient.

[0070] When using targeted editing information to calculate weighted coefficients for the output signal, intervention coefficients can be calculated based on the targeted editing information. These intervention coefficients can include enhancement or suppression coefficients targeting the target concept. Enhancement coefficients are used to amplify the role of the target concept in the content generation process, while suppression coefficients are used to suppress the role of the target concept in the content generation process.

[0071] For example, when implementing the inhibitory concept of "detoxification," the output signal of this key head can be multiplied by a negative coefficient, such as the intervention coefficient. α =-1, which is equivalent to reversing the contribution of the key head in the computation process, thereby suppressing the target concept in the final output. However, when performing enhanced concepts such as "adding color description," the output signal of the color-related key head can be multiplied by a positive coefficient, such as the intervention coefficient. α =5, amplifying the contribution of the color concept, thus encouraging the concept in the final output.

[0072] In order to set the intervention coefficient, when calculating the intervention coefficient, the type-related fields can be extracted from the targeted editing information first, and the intervention type of the targeted editing information can be determined according to the type-related fields, wherein the intervention type is either enhancement intervention or suppression intervention.

[0073] Then, different intervention coefficients are set according to different intervention types. If the intervention type is an enhancing intervention, the intervention coefficient is set to the enhancing coefficient, that is, the intervention type is set to a negative coefficient; if the intervention type is an inhibiting intervention, the intervention coefficient is set to the inhibiting coefficient, that is, the intervention type is set to a positive coefficient.

[0074] For example, by reading fields related to the concept of inhibition in the targeted editing information, it can be determined that the intervention type of the targeted editing information is "detoxification," that is, the intervention coefficient corresponding to the intervention type is determined to be the inhibition coefficient. α =-1. Therefore, when the VLM starts processing the input image and prompts, it can monitor the model's computation process. When it detects the model calculating the output signals of the 16 color keypoints, it will intercept the output signals of the 16 color keypoints. Then, it multiplies these 16 color keypoint output signal values ​​by -1, effectively inverting the signals. This is equivalent to penalizing color-related concepts within the model. The final output is the descriptive text "a woman wearing sunglasses and a hat," thus suppressing the colors even if they are recognized internally by the model during final generation.

[0075] It should be noted that the process of the generative model content control method described in the above embodiments is also applicable to other tasks, such as suppressing "region" related words in question-answering tasks, or suppressing "toxic" related words in dialogues.

[0076] By applying the technical solutions of the above embodiments, the generative model content control method described in the above embodiments can be used to solve the black-box and uncontrollable problems of large generative models based on a sparse decomposition analysis and control framework. This method does not rely on costly model fine-tuning, but instead locates a very small number of key components (attention heads) within the model responsible for specific functions in a mathematically principled manner. By precisely editing the signals of these key heads during inference and performing different directional interventions based on different directional editing information, reliable control over the model's output content is achieved. This method has the advantages of being training-free, efficient, scalable, and highly versatile, providing an important pathway for developing safer, more reliable, and more controllable artificial intelligence systems.

[0077] In some embodiments, as a specific implementation of the generative model content control method described in the above embodiments, some embodiments of this application also provide a generative model content control device, such as... Figure 4 As shown, the device includes: A dictionary definition module is used to define a semantic dictionary based on the de-embedding matrix of the generative model, wherein the semantic dictionary includes at least one sub-dictionary for representing target concepts of the same type; The specialization detection module is used to calculate the specialization score of the attention head to the sub-dictionary in the generative model. The specialization score is the variance ratio calculated using a greedy algorithm of synchronous orthogonal matching pursuit. The specialization score is used to characterize the dictionary atomic indexes that the sub-dictionary can interpret the output signal of the attention head. A head selection module is used to select key heads based on the specialization score, wherein the key head is at least one attention head related to the target concept; The directional editing module is used to acquire directional editing information and the output signal of the key head; The result output module is used to perform weighted coefficient calculation on the output signal using the directional editing information, so that the generative model outputs the directional editing result.

[0078] It should be noted that other corresponding descriptions of the functional units involved in the generative model content control device provided in the embodiments of this application can be found in the corresponding descriptions in the generative model content control method provided in the above embodiments, and will not be repeated here.

[0079] By applying the technical solutions of the above embodiments, the generative model content control device described in the above embodiments can define a semantic dictionary based on the de-embedding matrix of the generative model through a dictionary definition module. The specialization detection module then uses a greedy algorithm of synchronous orthogonal matching pursuit to calculate the specialization score of the attention head in the generative model relative to the sub-dictionary in the semantic dictionary, enabling the head selection module to select key heads based on the specialization score. Then, the directional editing module acquires the directional editing information and the output signal of the key heads, and the result output module uses the directional editing information to perform weighted coefficient calculation on the output signal, so that the generative model outputs the directional editing result. The device can quantify the function of specific components in the generative model through sparse decomposition and efficiently control the content generation result of the model by manipulating a very small number of key components, achieving reliable and accurate control of the model's output content without additional training.

[0080] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 5 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0081] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0082] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0083] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0085] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0086] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0087] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0088] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0089] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for content control in generative models, characterized in that, The method includes: A semantic dictionary is defined based on the de-embedding matrix of the generative model, and the semantic dictionary includes at least one sub-dictionary for representing target concepts of the same type; Calculate the specialization score of the attention head to the sub-dictionary in the generative model. The specialization score is the variance ratio calculated using a greedy algorithm of synchronous orthogonal matching pursuit. The specialization score is used to characterize the dictionary atomic indexes of the sub-dictionary that can interpret the output signal of the attention head. Key heads are selected based on the specialization score, wherein the key head is at least one attention head that is related to the target concept; Acquire directional editing information and the output signal of the key head; The output signal is weighted using the directional editing information so that the generative model outputs the directional editing result.

2. The method according to claim 1, characterized in that, Calculating the specialization score of the attention head to the sub-dictionary in the generative model includes: Extract semantic vectors from the semantic dictionary; Obtain the output matrix of the attention head in the current step, and calculate the residual matrix between the output matrix and the sparse reconstruction result of the previous step; The dictionary atom index selected in the current step is calculated based on the semantic vector and the residual matrix. The dictionary atom index is the set of independent variable points that maximize the L1 norm between a single semantic vector and the residual matrix of all samples. Iterative calculations are performed based on the dictionary atomic indexes to obtain multiple text tokens capable of interpreting the attention head function; The variance ratio of multiple text tokens is calculated based on the target concept to obtain the specialization score.

3. The method according to claim 2, characterized in that, The variance ratio of multiple text tokens is calculated based on the target concept to obtain the specialization score, including: Extract multiple related tokens associated with the target concept from the semantic dictionary to construct a target concept sub-dictionary; The specialization score is obtained by calculating the percentage of the total variance of the output matrix that can be explained by the text token corresponding to the target concept sub-dictionary. The attention head is assigned relevant concept tags based on the specialization score.

4. The method according to claim 3, characterized in that, Obtaining the output matrix of the attention head in the current step also includes: Obtain an input dataset, which includes multiple sample data containing concept elements corresponding to the target concept; Input the input dataset into the generative model; Record the output matrix of each attention head when the generative model processes each sample data.

5. The method according to claim 1, characterized in that, Key leaders are selected based on the aforementioned specialization scores, including: The attention heads are sorted according to the specialization score to obtain a sorting result sequence, the sorting result sequence including the recognition information of multiple attention heads arranged in descending order of the specialization score; Obtain a preset quantity, wherein the preset quantity is an integer greater than or equal to 1; Select the preset number of attention heads from the sorted result sequence to obtain the key head.

6. The method according to claim 1, characterized in that, The output signal is weighted using the directional editing information to calculate the weighting coefficients, so that the generative model outputs the directional editing result, including: An intervention coefficient is calculated based on the targeted editing information, the intervention coefficient including an enhancement coefficient or an inhibition coefficient for the target concept; Calculate the product of the intervention coefficient and the output signal to obtain the intervention signal; The intervention signal is put back into the computational flow of the generative model so that the intervention signal propagates in subsequent layers of the generative model; Obtain the output data of the generative model to obtain the directional editing result.

7. The method according to claim 6, characterized in that, The intervention coefficient is calculated based on the targeted editing information, including: Extract type-related fields from the targeted editing information; The intervention type of the targeted editing information is determined based on the type-related fields, and the intervention type is either an enhancement intervention or a suppression intervention. If the intervention type is an enhancement intervention, the intervention coefficient is set to the enhancement coefficient, and the enhancement coefficient is a positive coefficient. If the intervention type is a suppression intervention, the intervention coefficient is set to the suppression coefficient, and the suppression coefficient is a negative coefficient.

8. A generative model content control device, characterized in that, The device includes: A dictionary definition module is used to define a semantic dictionary based on the de-embedding matrix of the generative model, wherein the semantic dictionary includes at least one sub-dictionary for representing target concepts of the same type; The specialization detection module is used to calculate the specialization score of the attention head to the sub-dictionary in the generative model. The specialization score is the variance ratio calculated using a greedy algorithm of synchronous orthogonal matching pursuit. The specialization score is used to characterize the dictionary atomic indexes that the sub-dictionary can interpret the output signal of the attention head. A head selection module is used to select key heads based on the specialization score, wherein the key head is at least one attention head related to the target concept; The directional editing module is used to acquire directional editing information and the output signal of the key head; The result output module is used to perform weighted coefficient calculation on the output signal using the directional editing information, so that the generative model outputs the directional editing result.

9. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images