An AI large model content generation security detection method and system

Abstract

The application discloses an AI large model content generation security detection method, and belongs to the technical field of data processing. The technical solution points of the application include the following steps: obtaining a user prompt word, and obtaining a prompt word entity and a corresponding weight; obtaining a conduction path of the prompt word entity and a risk label according to the prompt word entity and a preset directed graph; obtaining a path weight according to the conduction path and the weight corresponding to the prompt word entity; if the path weight is less than or equal to a preset value, an AI large model generates content, and it is judged whether the generated content is intercepted; the application mines a prompt word violation conduction path based on a preset directed graph, avoids generating violation content, and accurately locates a risk segment through an attention heat map after content generation, and then combines a Bayesian network to output a reasonable interception strategy, so that the spread of violation content is avoided; finally, the interception standard is optimized in a timely manner based on user feedback on the generated content, so that the accuracy of interception is ensured and user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, more particularly to an AI large model content generation security detection method and system. BACKGROUND

[0002] In the context of the continuous expansion of the Internet content ecosystem, content security detection, as a core link for platform compliance operation and user experience guarantee, is facing increasingly complex challenges. A large amount of diverse texts, images and other content are constantly emerging, and the disguising methods of illegal content are increasingly concealed, which puts strict requirements on the accuracy, foresight and dynamic adaptability of detection technology. However, the existing technology cannot accurately capture the text and image collaborative illegal risk, resulting in frequent misjudgment and misjudgment, increasing the risk of illegal content spread and governance cost, so the existing technology has defects. SUMMARY

[0003] In view of the defects of the prior art, the purpose of the present application is to provide an AI large model content generation security detection method and system, which can effectively intercept illegal content and reduce the risk of illegal content spread through double auditing before and after content generation.

[0004] To achieve the above purpose, the present application provides the following technical scheme:

[0005] The present application provides an AI large model content generation security detection method, comprising:

[0006] Obtaining a user prompt word and obtaining a prompt word entity and a corresponding weight;

[0007] According to the prompt word entity and the preset directed graph, the conduction path of the prompt word entity and the risk label is obtained;

[0008] According to the conduction path, it is judged whether the AI large model generates content;

[0009] If the AI large model generates content, it is judged whether to intercept the content according to the conduction path and the Bayesian network.

[0010] As a further improvement of the present application, according to the conduction path, it is judged whether the AI large model generates content, comprising:

[0011] According to the conduction path and the weight corresponding to the prompt word entity, the path weight is obtained;

[0012] If the path weight is greater than a preset value, the AI large model does not generate content;

[0013] If the path weight is less than or equal to a preset value, the AI large model generates content, and it is judged whether to intercept the content.

[0014] As a further improvement of the application, the judging whether to intercept the content comprises:

[0015] obtaining a risk entity according to the path weight and the content;

[0016] obtaining a risk probability according to the risk entity and an attention heat map;

[0017] obtaining a joint risk probability according to the risk probability and a Bayesian network;

[0018] judging whether to intercept the content according to the joint risk probability.

[0019] As a further improvement of the application, it further comprises:

[0020] obtaining an updated threshold value according to a current complaint rate and a preset state transition matrix;

[0021] replacing the preset value with the updated threshold value.

[0022] As a further improvement of the application, the obtaining the conduction path of the prompt word entity and the risk label according to the prompt word entity and a preset directed graph comprises:

[0023] determining the risk label according to the prompt word entity;

[0024] determining an associated node according to a similarity between the prompt word entity and a node in the preset directed graph;

[0025] determining a preset path corresponding to the associated node and the risk label to obtain the conduction path.

[0026] As a further improvement of the application, the obtaining the risk probability according to the risk entity and an attention heat map comprises:

[0027] obtaining a risk weight corresponding to the risk entity according to an attention mechanism model;

[0028] obtaining an attention heat map according to the risk entity and the risk weight;

[0029] obtaining the risk probability according to the attention heat map and an activation function.

[0030] As a further improvement of the application, the obtaining the joint risk probability according to the risk probability and a Bayesian network comprises:

[0031] obtaining an overall risk probability according to the risk probability and the Bayesian network;

[0032] According to the risk probability and the overall risk probability, the joint risk probability is obtained.

[0033] As a further improvement of the application, the update threshold is obtained according to the current complaint rate and a preset state transition matrix, including:

[0034] According to the current complaint rate and a preset state transition matrix, a transition probability is obtained.

[0035] According to the transition probability and an adjustment coefficient, an update threshold is obtained.

[0036] As a further improvement of the application, the update threshold is obtained according to the current complaint rate and a preset state transition matrix, including:

[0037] According to the joint risk probability, a risk category is determined, and a complaint rate condition corresponding to the risk category is determined.

[0038] If the current complaint rate meets the complaint rate condition, a transition probability is obtained according to the preset state transition matrix.

[0039] If the current complaint rate does not meet the complaint rate condition, the preset state transition matrix is updated, and a transition probability is obtained according to the updated preset state transition matrix.

[0040] The application provides an AI large model content generation security detection system, which is applied to the AI large model content generation security detection method.

[0041] An input and preprocessing module is configured to obtain a user prompt word and obtain a prompt word entity and a corresponding weight.

[0042] A pre-generation interception module is configured to obtain a conduction path of the prompt word entity and a risk label according to the prompt word entity and the corresponding weight and a preset directed graph, and determine whether the AI large model generates content according to the conduction path.

[0043] A detection engine module is configured to determine whether to intercept the content.

[0044] A feedback and iteration module is configured to obtain a current complaint rate, and replace a preset value with an update threshold according to the current complaint rate and a preset state transition matrix.

[0045] The application firstly extracts the entity and weight of the prompt word input by the user, mines the prompt word violation transmission path based on the preset directed graph, avoids generating violation content, obtains the risk probability through the attention heat map after content generation, strengthens the accuracy of risk identification, combines the Bayesian network to output a reasonable interception strategy, avoids the spread of violation content, and finally optimizes the interception standard in time based on the feedback of the user to the generated content, ensures the accuracy of interception and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A step schematic diagram of the AI large model content generation safety detection method of the application;

[0047] Figure 2 A structure schematic diagram of the preset directed graph;

[0048] Figure 3 A content schematic diagram of the preset directed graph;

[0049] Figure 4 A step schematic diagram of the content generated by interception;

[0050] Figure 5 A step schematic diagram of the transition probability. DETAILED DESCRIPTION

[0051] The technical scheme of the application will be described in detail below with the help of the drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical scheme of the application, rather than limitations of the technical scheme of the application.

[0052] In the following text, the term "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0053] As Figure 1 shown, the application embodiment provides an AI large model content generation safety detection method, which comprises:

[0054] Obtaining the user prompt word and obtaining the prompt word entity and the corresponding weight;

[0055] According to the prompt word entity and the preset directed graph, the transmission path of the prompt word entity and the risk label is obtained;

[0056] According to the transmission path, it is judged whether the AI large model generates content;

[0057] If the AI large model generates content, it is judged whether to intercept the content according to the transmission path.

[0058] For example, when a user needs to generate content using an AI large model, a piece of text is input into the AI large model to generate corresponding content that meets the text. The user input text is a user prompt word, and the prompt word entity is the key information in the user prompt word that has actual meaning and can represent the core demand or content direction. For example, when the user inputs "In a three-day trip to Thailand, I want to find a net red check-in strategy for an island thrill project", "Thailand (location), island (location), thrill project (activity type), and net red check-in (content style)" are prompt word entities that can directly determine the final service direction. The means of determining the prompt word entity is a technical means that can be implemented by those skilled in the art, and this embodiment will not be repeated here. For example, a NER (Named Entity Recognition) model can be used to first split the prompt word into single words or short word groups, and then the model uses the semantic rules obtained through training to exclude meaningless virtual words, and finally obtains each prompt word entity and the label corresponding to each prompt word entity, such as labels including time, object, and emotion.

[0059] Then, the weight corresponding to each prompt word entity is calculated, which represents the importance of the prompt word entity. The weight can be obtained by TF-IDF algorithm. Specifically, for each prompt word entity, the TF value corresponding to the prompt word entity is the frequency of the prompt word entity appearing in the prompt word, and the frequency is the ratio of the number of times the prompt word entity appears in the prompt word to the total number of prompt word entities in the prompt word. The IDF value corresponding to the prompt word entity is wherein represents the total number of documents stored in the database of the AI large model, represents the number of documents including the prompt word entity among all stored documents.

[0060] After obtaining the prompt word entity, the transmission path of the prompt word entity and the risk label is obtained according to the preset directed graph, and whether the AI large model generates content is judged according to the transmission path. If not, the user is returned information to input the modified prompt word, and if yes, it is judged whether the generated content is safe, i.e., whether it is illegal content, after the content is generated. If not, the generated content is returned to the user, and if yes, the user is returned information to input the modified prompt word, and the interception standard is optimized according to the user's feedback to the returned information.

[0061] The application avoids generating illegal content by first extracting the entity and weight of the prompt word input by the user, and mining the illegal content transmission path of the prompt word based on the preset directed graph, and obtaining the risk probability through the attention heat map after content generation, strengthening the accuracy of risk identification, combining the Bayesian network to output a reasonable interception strategy to avoid the spread of illegal content, and finally optimizing the interception standard in time based on the feedback of the user to the generated content, ensuring the accuracy of interception while improving the user experience.

[0062] Further, the embodiment of the application provides a step of obtaining a transmission path of a prompt word entity and a risk label according to the prompt word entity and a preset directed graph, comprising:

[0063] determining a risk label according to the prompt word entity;

[0064] determining an associated node according to the similarity between the prompt word entity and the nodes in the preset directed graph;

[0065] determining a preset path corresponding to the associated node and the risk label to obtain the transmission path.

[0066] The preset directed graph is established according to historical data in a preset time period, and the length of the preset time period is not limited in the embodiment. Since the prior art usually detects whether the generated content is illegal after the content is generated, the historical data is the generated content after the content is generated, and the historical data includes multiple pieces of illegal content and multiple pieces of safe content generated in the preset time period and the illegal types of the illegal content. The preset directed graph specifically includes entity layer nodes, intermediate layer nodes and risk nodes. Each entity layer node corresponds to an entity, each risk node corresponds to a risk label, and the intermediate layer nodes are used to connect the entity layer nodes and the risk nodes. Each intermediate layer node can correspond to multiple entity layer nodes, for example Figure 2 The intermediate node B corresponds to the entity A and the entity C. As shown in Figure 2 A simple directed graph is shown. The line segment connecting two nodes is called an edge, and the value on the edge represents the weight between the two nodes. The higher the weight, the higher the semantic correlation between the two nodes. For example, assume Figure 2 The entity A in the middle represents a stimulus item, and the node A represents a dangerous action. When the weight between the stimulus item and the dangerous action is 0.8, it means that there is an 80% probability that the dangerous action will be involved when the stimulus item is mentioned. Figure 2 The values are examples and are only used to illustrate the structure and meaning of the preset directed graph. The embodiment is not limited in this regard, and Figure 2 The weight of each edge is not specified.

[0067] Specifically, in establishing the preset directed graph, the entity corresponding to each entity layer node can be determined by the NER model, that is, the entity in the violation content can be extracted by the NER model, and the frequency of occurrence of each entity is counted to obtain entities with a frequency greater than a preset frequency, and an entity layer node is generated for each entity with a frequency greater than the preset frequency. In this embodiment, the preset frequency is not limited, for example, it can be set to 50 times. The risk node can be set according to platform rules, regulatory requirements or the type of violation content, such as setting the risk label as dangerous risk and vulgar risk, and then generating a risk node for each risk label. The intermediate layer node is used to represent the specific reason why the violation content has risks. In establishing the preset directed graph, the intermediate layer node can be determined by manually auditing the violation content, such as determining that the intermediate layer node corresponding to the vulgar risk label can be exposure scene or traffic direction. When determining the weight between an entity layer node and an intermediate layer node, for example, when determining the weight between entity A and node A, the number of contents including entity A in the historical data is counted first, and the number of contents including node A in these contents is counted. According to the ratio of the two quantities, the weight between entity A and node A is obtained. Similarly, if the weight between an intermediate layer node and a risk node needs to be determined, for example, the weight between node A and A risk label needs to be determined, the number of contents including node A in the historical data is counted first, and the number of contents whose violation type corresponding to the violation content is A risk label in these contents is counted. According to the ratio of the two quantities, the weight between node A and A risk label is obtained. In the above, when determining whether to include entity A, node A or A risk label, entity A, node A or A risk label needs to be matched with a plurality of entities corresponding to each content. For example, when determining whether a piece of content in the historical data includes entity A, the similarity between each entity in the piece of content and entity A needs to be calculated. If there is an entity in the piece of content with a similarity greater than a first preset similarity, it is considered that the piece of content includes entity A. The first preset similarity is not limited in this embodiment.

[0068] Moreover, the preset directed graph is not completely fixed after being established, and needs to be updated from time to time. For example, every preset time length, the conduction path corresponding to the prompt word considered to be in violation within the time length is obtained, if there is an entity layer node that does not appear in the conduction path of the time length, the entity layer node is deleted. Moreover, according to the prompt word considered to be in violation within the time length, the weight corresponding to the edge in the directed graph is calculated, and the weight is updated. The way of calculating the weight is the same as described above, and this embodiment will not be repeated here. For example, if the weight between entity A and node A calculated according to the prompt word considered to be in violation within the time length is 0.9, and the weight between entity A and node A in the preset directed graph established according to the historical data is 0.8, then the ratio of the number of prompt words considered to be in violation within the time length to the number of contents generated in the historical data is two weights. For example, if the ratio of the number is 1:9, then the updated weight is 0.9x0.1+0.8x0.9=0.81. This embodiment does not limit the preset time length, for example, it can be set to 30 days.

[0069] Then, the prompt words not considered to be in violation within the time length are obtained, a plurality of samples are randomly selected therefrom and manually audited to check whether there is a new risk label, such as a false information risk and the corresponding new entity and node. If there is, it is added to the preset directed graph, and the weight is calculated according to the plurality of samples. The way of calculating the weight is the same as described above, and this embodiment will not be repeated here.

[0070] After obtaining the prompt word entity corresponding to the user prompt word and the corresponding weight, the conduction path of the prompt word entity and the risk label is then determined. Specifically, the entity layer node corresponding to the prompt word entity needs to be determined first, and then the conduction path is determined according to the path between the entity layer node, the intermediate layer node and the risk node. If each prompt word entity is matched with each entity layer node, the data volume is large, and therefore a preset index table can be set. Specifically, a plurality of entity layer nodes can be subjected to cluster analysis, a category label is set for each category of entity layer node, and each category label and the corresponding entity layer node are stored in the index table. For example, the category label is high altitude, and the entity layer nodes corresponding to the category label include unprotected high altitude parachute jumping, etc. Cluster analysis and setting a category label for each category of entity layer node are technical means that can be realized by a person skilled in the art, and will not be described herein. At this time, when the entity layer node corresponding to the prompt word entity is determined, the similarity between each prompt word entity and each category label in the index table is first calculated, the category label with the highest similarity is obtained, then the similarity between the prompt word entity and each entity layer node in the category label is calculated, the entity layer node with the highest similarity is obtained, and then it is judged whether the similarity is greater than a set value. If it is greater, the entity layer node is taken as the entity layer node corresponding to the prompt word entity, and the entity layer node is recorded as an associated node. The specific value of the set value is not limited in the present embodiment.

[0071] Then, the preset path of each associated node and the risk label is determined. For example, as shown in Figure 3 , Figure 3 , Figure 2 For a specific example, when one of the associated nodes is a stimulating item, the corresponding intermediate layer nodes are dangerous action and exposure scene, the risk node corresponding to the dangerous action is danger risk, and the risk corresponding to the exposure scene is vulgar risk. Therefore, the preset path corresponding to the stimulating item is stimulating item→dangerous action→danger risk, and stimulating item→exposure scene→vulgar risk. Then, the set of each preset path corresponding to each associated node is taken as the conduction path.

[0072] The present embodiment provides a standardized semantic framework for prompt word risk analysis according to the preset directed graph, filters the associated nodes through the semantic matching degree, avoids the confusion in risk judgment caused by the difference in user expression, enables different prompt word entities to anchor to a unified risk analysis path, and sets a preset index table to narrow the entity matching range. While avoiding the high computational power consumption of full matching, the preset path of the associated node is accurately positioned, and through the analysis of the preset path, the generation of illegal content is reduced from the source, and a basis is provided for subsequent analysis based on the attention heat map.

[0073] Further, the embodiment of the application provides a step of judging whether the AI large model generates content according to the conduction path, comprising:

[0074] According to the weight corresponding to the conduction path and the prompt word entity, the path weight is obtained;

[0075] If the path weight is greater than a preset value, the AI large model does not generate content;

[0076] If the path weight is less than or equal to the preset value, the AI large model generates content, and whether the content is intercepted is judged.

[0077] Specifically, after obtaining the conduction path, firstly, the weight corresponding to each preset path in the conduction path is calculated, and the numerical value of the weight is equal to the product of the weight of each edge included in the preset path. For example, for the preset path: stimulus item->dangerous action->dangerous risk, according to Figure 3 It can be obtained that the weight between the stimulus item and the dangerous action is 0.8, and the weight between the dangerous action and the dangerous risk is 0.6, so the weight corresponding to the preset path is 0.48. Then, the weight corresponding to the entity layer node in the preset path is obtained, for example, the entity layer node in the preset path is the stimulus item, then the prompt word entity corresponding to the stimulus item and the weight corresponding to the stimulus item are obtained. Assuming that the prompt word entity is also the stimulus item, the TF-IDF value corresponding to the stimulus item is obtained as the weight corresponding to the entity layer node, and then the weight corresponding to the preset path is multiplied by the weight corresponding to the entity layer node to obtain the path weight corresponding to the preset path.

[0078] Then, the path weight corresponding to each preset path is compared with the preset value, if there is at least one path weight greater than the preset value, the AI large model does not generate content, and returns information to the user, wherein the information includes the risk label corresponding to the preset path with the path weight greater than the preset value, so that the user modifies the prompt word and re-enters, if each path weight is less than or equal to the preset value, the AI large model generates content, and executes the subsequent step.

[0079] The embodiment is based on the calculation rule of probability, and quantifies the risk value of content violation into a specific numerical value through multiplication between weights, so that the pre-generation risk prediction is more objective and interpretable. Specifically, the weight corresponding to each edge can also be understood as the probability of successful transmission of risk on the edge. In a preset path, only the entity layer node and the intermediate layer node are successfully transmitted, and then the transmission of the intermediate layer node and the risk node is performed. The complete transmission of the preset path is the simultaneous transmission of multiple edges. According to the calculation rule of joint probability of independent events in probability theory, the probability of simultaneous occurrence of multiple events needs to be multiplied. The weight corresponding to the prompt word entity is multiplied by the weight corresponding to the preset path, which can further reflect the importance of the prompt word entity for the violation risk. Through the steps provided in the embodiment, the prompt word violation transmission path can be mined based on the preset directed graph before the AI model generates the content, so as to avoid generating violation content and effectively block the violation content on the input side.

[0080] Further, as shown in Figure 4 The embodiment provides a step of judging whether to intercept the content, including:

[0081] obtaining a risk entity according to the path weight and the content;

[0082] obtaining a risk probability according to the risk entity and the attention heat map;

[0083] obtaining a joint risk probability according to the risk probability and the Bayesian network;

[0084] judging whether to intercept the content according to the joint risk probability.

[0085] Wherein, before the above steps are performed, first, the content required by the user is generated according to the prompt word input by the user. Then, the path weight corresponding to each preset path in the transmission path is obtained, each path weight is compared with a threshold value, the preset path corresponding to the path weight greater than the threshold value is obtained, and the prompt word entity corresponding to the entity layer node based on the preset path is obtained. The prompt word entity is taken as a risk fragment. The numerical value of the threshold value is not limited in the embodiment, but the threshold value should be less than the preset value used to judge whether the AI large model generates the content.

[0086] Then the generated content corresponding to the entity is obtained, and the generated content at least includes one of text content and image content. For each entity corresponding to the weight of the text content, the TF-IDF algorithm can be used to calculate it. This embodiment will not be repeated. For image content, first, the generated image is split into multiple continuous regions according to the image segmentation algorithm, for example, an island scene image can be split into a person region, a background region and an article region. Then a pre-trained visual model is used to perform semantic mapping of the image and the text on each segmented region, and output the entity label of the region, for example, the person region is identified as a child entity, the background region is identified as an island scenery entity, and the article region is identified as a safety equipment entity. Then filter the redundant entities (such as pure background blocks), and finally obtain multiple entities corresponding to the image content. Then calculate the proportion of the number of pixels corresponding to each entity in the image in the total number of pixels in the image to obtain the weight corresponding to each entity. The image segmentation algorithm can use Mask R-CNN, and the visual model can use CLIP. This embodiment does not limit this.

[0087] Then the similarity between each generated content corresponding to the entity and each risk segment is calculated, and the entity with a similarity greater than a second preset similarity is taken as a risk entity, that is, the entity that needs to be detected is determined. Then, based on the risk entity, the attention heat map and the Bayesian network, it is judged whether the generated content needs to be intercepted. The specific value of the second similarity is not limited in this embodiment.

[0088] After generating the content required by the user, the risk segment in the prompt word is first determined according to the path weight in this embodiment. Although the risk segment is not identified as illegal content in the detection before generation, the path weight is still high and needs to be further verified. Based on this, the risk segment is combined with the attention heat map, and the Bayesian network is introduced to obtain the joint risk probability of the multi-modal data, improve the accuracy of illegal content interception, and reduce the missed and misjudged.

[0089] Further, the embodiment provides a step of obtaining a risk probability according to a risk entity and an attention heat map, comprising:

[0090] obtaining a risk weight corresponding to the risk entity according to the attention mechanism model;

[0091] obtaining an attention heat map according to the risk entity and the risk weight;

[0092] obtaining a risk probability according to the attention heat map and the activation function.

[0093] Specifically, for risk entities corresponding to text content, each risk entity can be transformed into a semantic vector using a word embedding model (such as BERT). The vector value reflects the semantic features of the entity. For risk entities corresponding to image content, the pixel features corresponding to each risk entity can be transformed into vectors using a visual embedding model (CLIP, ResNet). The vector value reflects the visual features of the region. Then, the corresponding target risk vector is determined based on the risk label corresponding to the preset path with a path weight greater than a threshold. The similarity between the vector corresponding to each risk entity and each target risk vector is then calculated as the original score of each risk entity under each risk label. Subsequently, based on an attention mechanism model, each original score is normalized to obtain the attention weight of each risk entity under each risk label, thereby obtaining the risk weight corresponding to each risk entity. Each risk label corresponds to a preset target risk vector, which is obtained through training. The attention mechanism and similarity calculation are technical means that can be implemented by those skilled in the art, and will not be elaborated here.

[0094] For example, for the first The risk entity, in the first Risk weight under each risk label for:

[0095]

[0096] in, , The total number of risky entities, , This represents the total number of risk tags corresponding to preset paths whose path weights exceed a threshold. Indicates the first Attention weights corresponding to each risk entity Indicates the first The path weight corresponding to each risk entity can be specifically selected from the preset paths corresponding to path weights greater than a threshold. The preset paths for each risk label are used, and the average of the path weights corresponding to these preset paths is taken as the value of the first preset path. The path weights corresponding to each risk entity The balance coefficient is not limited in value in this embodiment. Those skilled in the art can set it according to actual conditions. For example, if less data is used when generating the preset directed graph, the system tends to rely more on the judgment of the generated content. It can be set to a relatively large value, such as 0.7-0.9. If the historical data used during training is abundant and the preset directed graph is relatively accurate, then... It can be set to a small value, such as 0.3-0.5.

[0097] Then, according to the risk weight corresponding to each risk entity under each risk label, a corresponding attention heat map is generated, and the text content and the image content correspond to different attention heat maps, that is, the number of attention heat maps is two, which correspond to the text content and the image content respectively, the number of rows of the attention heat map is the number of risk entities corresponding to the text content or the image content, and the number of columns is that is, the number of columns is the total number of preset paths corresponding to the risk label corresponding to the path weight greater than the threshold, the attention heat map can express the data of the risk weight through color, for example, the higher the value is, the more the color is inclined to red, and the lower the value is, the more the color is inclined to blue, and the attention heat map can further accurately locate the illegal content in the risk entity. Then, each risk weight corresponding to the red area in the attention heat map is obtained, and the corresponding risk probability is obtained according to the activation function.

[0098] The embodiment according to the attention heat map further accurately locates the illegal content in the risk entity, and converts the risk weight presented by the heat map into a quantifiable probability index, breaking through the subjective limitations of traditional manual review, and serving as a basis for subsequent interception of generated content.

[0099] Further, the embodiment provides a step of obtaining a joint risk probability according to a risk probability and a Bayesian network, comprising:

[0100] obtaining an overall risk probability according to the risk probability and the Bayesian network;

[0101] obtaining the joint risk probability according to the risk probability and the overall risk probability.

[0102] In the Bayesian network, each overall risk probability corresponding to each risk label, and a text risk probability and an image risk probability corresponding to each overall risk probability are stored. For example, if the overall risk probability corresponding to the vulgar risk label is 0.8, the text risk probability and the image risk probability corresponding to the overall risk probability are both 0.6, which means that when the probability of the current text vulgar and the probability of the image vulgar are both 0.6, the overall vulgar risk probability of the generated content is 0.8, and the Bayesian network can be trained according to historical data.

[0103] Specifically, after obtaining the risk probability, for each attention heat map, the mean value of each column is calculated, and then the text risk probability and the image risk probability corresponding to each risk label are obtained. Then, the overall risk probability corresponding to the text risk probability and the image risk probability in the Bayesian network is found, and finally the overall risk probability, the text risk probability and the image risk probability are multiplied to obtain the joint risk probability corresponding to each risk label, which represents the probability of the generated content existing the risk.

[0104] Then each joint risk probability is compared with a preset value, if each joint risk probability is less than or equal to the preset value, the generated content is sent to the user, if there is at least one joint risk probability greater than the preset value, it means that the generated content is illegal content, the generated content needs to be intercepted, and the user is returned to the existing risk type according to the risk label, please input the modified prompt word, the preset value can be the same as or different from the preset value when judging whether to generate content, which is not limited in this embodiment.

[0105] In this embodiment, the attention heat map focuses on the illegal content in the text and image, and based on the risk probability obtained by the attention heat map, the single-modal risk probability is aggregated by mean calculation, and then the overall risk probability stored by the Bayesian network is called, and finally the single-modal risk and the overall risk are multiplied to obtain the joint risk probability, which solves the problems of single-modal misjudgment and fuzzy risk judgment, and accurately intercepts the generated content on the output side, reducing the probability of outputting illegal content.

[0106] Further, the AI large model content generation security detection method provided in the embodiment of the application further comprises:

[0107] Obtain the current complaint rate, and obtain the update threshold value according to the current complaint rate and the preset state transition matrix;

[0108] Replace the preset value according to the update threshold value.

[0109] Further, the embodiment provides a step of obtaining an update threshold value according to a current complaint rate and a preset state transition matrix, comprising:

[0110] Obtain the transition probability according to the current complaint rate and the preset state transition matrix;

[0111] Obtain the update threshold value according to the transition probability and the adjustment coefficient.

[0112] The preset state transition matrix is determined according to the complaint data within a preset time. For example, the categories of the generated content can be defined, such as low risk, medium risk and high risk. Low risk means that the generated content is relatively healthy and has low risk of violation. Medium risk means that the generated content may contain illegal content and has a high risk of violation, which needs to be paid attention to. High risk means that the generated content has a clear tendency to violate and needs to be intercepted. A preset value is set for each category, which is used as the preset value in the above-mentioned pre-generation interception and post-generation interception. For example, the low-risk preset value is 0.7, the medium-risk preset value is 0.6, and the high-risk preset value is 0.5. The specific value of the preset value is not limited in this embodiment, but the higher the risk, the lower the corresponding preset value, that is, the greater the possibility of intercepting the generated content.

[0113] After obtaining the complaint data in the preset time, for example, the complaint data includes the category corresponding to the generated content of each day in the preset time. Specifically, the sum of the joint risk probability corresponding to each generated content of each day can be calculated, and the category corresponding to the generated content of each day is determined according to the interval in which the sum of the joint risk probability is located. For example , the day belongs to low risk, , the day belongs to medium risk, , the day belongs to high risk, wherein represents the sum of the joint risk probability, and is located between 0 and 1, and , and is a threshold value for classifying categories.

[0114] After obtaining the category corresponding to each day in the preset time, the number of transitions between each category is counted. For example, if a certain day is low risk and the next day is medium risk, the number of transitions from low risk to medium risk is recorded as 1. After obtaining the number of transitions between all categories, the preset transition probability corresponding to each category can be obtained. For example, low risk is recorded as , medium risk is recorded as , and the preset transition probability between low risk and medium risk is:

[0115]

[0116] wherein represents the number of transitions from low risk to medium risk in the preset time, represents the sum of the number of transitions from low risk to medium risk and the number of transitions from low risk to high risk in the preset time, and the transition probability between low risk and medium risk and the transition probability between medium risk and low risk have different meanings, that is . After obtaining the preset transition probability corresponding to each category, a preset state transition matrix can be obtained. The element in the first row and the first column of the matrix is equal to , the element in the first row and the second column is equal to , the element in the second row and the first column is equal to , and so on.

[0117] But the historical rules may not apply to the current mutation, for example, if the complaint rate of the day (i.e. the current complaint rate) exceeds the preset complaint rate condition, the values in the preset state transition matrix need to be updated. Each risk category corresponds to a preset complaint rate condition. The determination steps of the complaint rate condition are as follows: obtaining the complaint rate of each day within the preset time, the complaint rate refers to the ratio of the number of complaints to the number of generated content by users on the same day, then obtaining the corresponding category of each day within the preset time, classifying each complaint rate within the preset time according to the category, and then obtaining the mean and variance of the complaint rate corresponding to each category. According to the principle of 3 , the normal complaint rate interval corresponding to each category can be obtained, which is the complaint rate condition corresponding to the category.

[0118] According to the sum of the joint risk probability corresponding to each generated content of the day, the category of the day can be determined, then the complaint rate of the day is obtained, and it is judged whether it meets the complaint rate condition corresponding to the category, that is, whether the complaint rate of the day is located in the interval corresponding to the complaint rate condition. If yes, the preset state transition matrix does not need to be updated, and the probability of transition to each category is determined according to the category currently located and the preset state transition matrix. The value with the largest probability is taken as the transition probability, and the category corresponding to the value is taken as the transition category. The preset value corresponding to the transition category is taken as the update threshold.

[0119] If it does not meet the complaint rate condition corresponding to the category, it means that the preset state transition matrix obtained according to the historical rules cannot reasonably meet the current abnormal complaint rate situation, so the preset state transition matrix needs to be updated. First, the midpoint of the corresponding complaint rate condition is determined, then the difference between the midpoint and the complaint rate of the day is calculated, and the ratio of the difference to the midpoint is taken as the complaint rate deviation , then the complaint rate of the day is updated according to the complaint rate deviation . Specifically, the complaint rate condition corresponding to the category is divided into two cases: higher and lower. For the case that the complaint rate is higher than the complaint rate condition, the threshold needs to be relaxed to reduce the interception situation, so the value corresponding to the category in the preset state transition matrix is obtained, for example, for the high-risk category, the corresponding value is , the updated value , wherein is a parameter for controlling the influence range of the complaint rate deviation on the transition probability, the value of which is not limited in this embodiment, and those skilled in the art can determine it according to the demand for quick response to user complaints. Then the preset transition probability between the category and other categories is obtained, for example, for the high-risk category, the and , is the transition probability between high-risk and medium-risk, For the transition probability between high risk and low risk, since the threshold is relaxed, the probability of transitioning to a lower risk will increase, so the updated transition probability between high risk and medium risk The sum of the transition probabilities corresponding to the categories should be 1, the updated transition probability between high risk and low risk . Conversely, for the case where the complaint rate is lower than the complaint rate condition, the threshold needs to be increased, and the updated value The updated transition probability between high risk and medium risk Therefore, when updating the preset state transition matrix according to the current complaint rate, only one row is updated.

[0120] Then, according to the updated preset state transition matrix, the probability of transitioning to each category is determined, the value with the largest probability is taken as the transition probability, the category corresponding to the transition probability is determined, and the preset value corresponding to the category is obtained, to obtain the updated threshold is:

[0121]

[0122] wherein represents the preset value before updating, is an adjustment coefficient, the value of which is not limited in the embodiment, and those skilled in the art can determine it according to the demand for rapid response to user complaints, for example, a larger adjustment coefficient can be set if rapid response is required, is the number of the current category, for example, low risk = 1, medium risk = 2, and high risk = 3. Finally, the updated threshold replaces the preset value, and the subsequent interception step is performed.

[0123] As shown in Figure 5 According to the above analysis, it can be summarized that the step of obtaining the transition probability according to the current complaint rate and the preset state transition matrix provided by the embodiment includes:

[0124] Determine the risk category according to the joint risk probability, and determine the complaint rate condition corresponding to the risk category;

[0125] If the current complaint rate meets the complaint rate condition, obtain the transition probability according to the preset state transition matrix;

[0126] If the current complaint rate does not meet the complaint rate condition, update the preset state transition matrix, and obtain the transition probability according to the updated preset state transition matrix.

[0127] The embodiment is based on the complaint rate of the day, when the complaint rate is higher than the complaint rate condition, the preset value corresponding to the current risk is relaxed, when the complaint rate is less than the complaint rate condition, the preset value corresponding to the current risk is reduced, so that the updated preset state transition matrix can adapt to the sudden complaint rate abnormal situation, and then the preset value is adjusted through the updated preset state transition matrix, so that when subsequent violation interception is performed according to the adjusted preset value, the false interception and missed interception are reduced, and then the complaint rate is reduced and the user service experience is improved.

[0128] Further, the application embodiment provides an AI large model content generation security detection system, comprising:

[0129] An input and preprocessing module is configured to obtain a user prompt word and obtain a prompt word entity and a corresponding weight;

[0130] A pre-interception module is configured to obtain a prompt word entity and a corresponding weight, and a preset directed graph, to obtain a conduction path of the prompt word entity and a risk label, and to determine whether the AI large model generates content according to the conduction path;

[0131] A detection engine module is configured to determine whether to intercept content;

[0132] A feedback and iteration module is configured to obtain a current complaint rate, and replace a preset value with an updated threshold value according to the current complaint rate and a preset state transition matrix.

[0133] The AI large model content generation security detection method and system provided by the application embodiment first extracts the entity and weight of the prompt word input by the user, and mines the prompt word violation conduction path based on the preset directed graph, avoids generating violation content, and obtains the risk probability through the attention heat map after content generation, strengthens the accuracy of risk identification, combines the Bayesian network to output a reasonable interception strategy, avoids the spread of violation content, and finally optimizes the interception standard in time based on the feedback of the user to the generated content, ensures the accuracy of interception, and improves the user experience.

[0134] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for security detection of AI large model content generation, characterized in that, include: Obtain user suggestions and their corresponding entities and weights; Based on the prompt word entity and the preset directed graph, the transmission path between the prompt word entity and the risk label is obtained; wherein, the preset directed graph specifically includes entity layer nodes, intermediate layer nodes and risk nodes, each entity layer node corresponds to an entity, each risk node corresponds to a risk label, and the intermediate layer nodes are used to connect entity layer nodes and risk nodes, and each intermediate layer node can correspond to multiple entity layer nodes. Based on the transmission path, determine whether the AI ​​large model generates content; If the AI ​​model generates content, determine whether to intercept the content based on the transmission path and the Bayesian network. The step of determining whether the AI ​​large model generates content based on the transmission path includes: The path weight is obtained based on the weights corresponding to the transmission path and the prompt word entity. Specifically, it includes calculating the weight corresponding to each preset path in the transmission path. The value of the weight is equal to the product of the weights of each edge included in the preset path. The weight corresponding to the preset path is multiplied by the weight corresponding to the prompt word entity. If the path weight is greater than a preset value, the AI ​​large model will not generate content; If the path weight is less than or equal to a preset value, the AI ​​big model generates content and determines whether to intercept the content. The determination of whether to intercept the content includes: The risk entity is obtained based on the path weight and the content, specifically including: generating content required by the user based on the prompt words input by the user; obtaining the path weight corresponding to each preset path in the transmission path; comparing each path weight with a threshold to obtain the preset path corresponding to the path weight greater than the threshold; obtaining the corresponding prompt word entity based on the entity layer node corresponding to the preset path; taking the prompt word entity as a risk fragment; and obtaining the entity similarity corresponding to the generated content that is greater than a second preset similarity as a risk entity. Based on the risk entities and attention heatmap, the risk probability is obtained; Based on the aforementioned risk probabilities and the Bayesian network, the joint risk probability is obtained; Based on the combined risk probability, determine whether to block the content; The step of obtaining the risk probability based on the risk entity and the attention heatmap includes: The risk weights corresponding to the risk entities are obtained based on the attention mechanism model. An attention heatmap is obtained based on the risk entity and the risk weight; The risk probability is obtained based on the attention heatmap and activation function. The step of obtaining the joint risk probability based on the risk probability and the Bayesian network includes: Based on the risk probability and the Bayesian network, the overall risk probability is obtained; The joint risk probability is obtained based on the risk probability and the overall risk probability.

2. The AI ​​large model content generation security detection method according to claim 1, characterized in that, Also includes: Obtain the current appeal rate, and based on the current appeal rate and the preset state transition matrix, obtain the updated threshold; The preset value is replaced according to the update threshold.

3. The AI ​​large model content generation security detection method according to claim 2, characterized in that, The step of obtaining the transmission path between the prompt word entity and the risk label based on the prompt word entity and the preset directed graph includes: The risk label is determined based on the prompt word entity; Based on the similarity between the prompt word entity and the nodes in the preset directed graph, the associated nodes are determined; The preset path corresponding to the associated node and the risk label is determined to obtain the transmission path.

4. The AI ​​large model content generation security detection method according to claim 2, characterized in that, The step of obtaining the update threshold based on the current appeal rate and the preset state transition matrix includes: The transition probability is obtained based on the current appeal rate and the preset state transition matrix; The updated threshold is obtained based on the transition probability and the adjustment coefficient.

5. The AI ​​large model content generation security detection method according to claim 4, characterized in that, The step of obtaining the transition probability based on the current appeal rate and the preset state transition matrix includes: The risk category is determined based on the joint risk probability, and the appeal rate condition corresponding to the risk category is determined accordingly; If the current appeal rate meets the appeal rate condition, the transition probability is obtained according to the preset state transition matrix; If the current appeal rate does not meet the appeal rate condition, update the preset state transition matrix and obtain the transition probability based on the updated preset state transition matrix.

6. A security detection system for AI large-scale model content generation, applied to the security detection method for AI large-scale model content generation as described in any one of claims 1-5, characterized in that, include: The input and preprocessing module is used to obtain user prompts and their entities and corresponding weights. The pre-generation interception module is used to obtain the transmission path between the prompt word entity and the risk label based on the prompt word entity and its corresponding weight, and a preset directed graph, and to determine whether the AI ​​big model generates content based on the transmission path. The detection engine module is used to determine whether the content should be blocked; The feedback and iteration module is used to obtain the current appeal rate and replace the preset value with the update threshold based on the current appeal rate and the preset state transition matrix.

Images