Artificial intelligence generated content review method, system and device, medium and product
By performing intent risk analysis and rule checks during the model reasoning process of AI-generated content, the problem of hidden prompt word attacks is solved, achieving more efficient compliance and security review.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies struggle to identify and prevent hidden prompts in AI-generated content, leading to compliance and security issues, particularly in their inability to effectively address hidden risks at the intent level.
By performing intent risk analysis during the reasoning process of the artificial intelligence model, segmenting user input content, constructing intent transmission chains, calculating user input intent risks, correcting intermediate and final results when necessary, performing rule checks, and outputting optimized reasoning results or error prompts.
It improves the accuracy and timeliness of reviewing AI-generated content, prevents security risks caused by prompt word attacks, and ensures the integrity and accuracy of compliant output.
Smart Images
Figure CN121835688A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of network security, and in particular to an artificial intelligence generated content review method, system, device, medium and product. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, artificial intelligence (AI) is increasingly widely used in content generation, providing efficient and convenient content creation support for users. However, the compliance and security of AI-generated content have also become prominent. In order to ensure the compliance and safety of AI output, various compliance checking methods have been developed in the industry.
[0003] The current main compliance checking method of AI-generated content mainly uses text review system, artificial intelligence model and manual review. Among them, the text review system automatically identifies and removes content containing sensitive information or violating policies by setting keywords and semantic analysis; the artificial intelligence model mainly compares the model output with the compliance standard to ensure that the generated results meet the specification requirements; manual review combines the preliminary screening of automated tools to conduct detailed review of high-risk or complex content.
[0004] However, these methods usually detect and intercept the output text after the artificial intelligence model completes content generation. This mode has a certain effect on regular violations, but it is difficult to deal with the increasingly prominent "prompt word attack", such as jailbreaking attack, etc. Such attacks carefully construct input prompts, contain double meanings, ambiguous expressions or hidden intent-oriented content, use the sensitivity of the model to the input to induce the model to bypass its own screening mechanism, and generate inappropriate, incorrect or misleading violation content. In summary, the existing review method cannot identify the hidden risks at the intent level, and the prompt word attack often does not contain obvious sensitive information, but induces the model to violate the rules through context association, intent transmission, etc. The existing method cannot effectively extract such intent-oriented compliance relevance, resulting in a blind spot in the review, therefore, the existing review method cannot effectively prevent the compliance risks brought by the prompt word attack, and the AI-generated content may have problems such as inaccuracy, security risks, information misleading, etc. SUMMARY
[0005] The purpose of the present application is to provide an artificial intelligence generated content review method, system, device, medium and product, which improves the accuracy and security of artificial intelligence generated content.
[0006] To achieve the above purpose, the present application provides an artificial intelligence generated content review method, comprising: While the artificial intelligence model is reasoning, the user input content is analyzed for intent risk, and the user input intent risk is calculated. According to the user input intent risk, an intent risk analysis and rule check are performed on the inference result of the artificial intelligence model to generate an intent risk update result; According to the intent risk update result, an optimized inference result or an error prompt is output.
[0007] Optionally, the intent risk analysis and rule check are performed on the inference intermediate result or the inference result of the artificial intelligence model according to the user input intent risk to generate the intent risk update result, including: When the user input intent risk is not higher than the risk threshold and the artificial intelligence model inference is not ended, the inference intermediate result of the artificial intelligence model is corrected in the artificial intelligence model inference process, and the intent risk analysis and rule check are performed on the inference result after the artificial intelligence model inference is ended to obtain the intent risk update result; When the user input intent risk is higher than the risk threshold, the intent risk analysis and rule check are performed on the inference result after the artificial intelligence model inference is ended to obtain the intent risk update result.
[0008] Optionally, the inference intermediate result of the artificial intelligence model is corrected in the artificial intelligence model inference process, including: The current iteration round number, the intermediate result and the current loss of the artificial intelligence model are read; The intermediate result is corrected according to the user input intent risk and the current loss; Based on the corrected intermediate result, the next round iteration of the artificial intelligence model inference is performed according to the current iteration round number.
[0009] Optionally, after the next round iteration of the artificial intelligence model inference is performed based on the corrected intermediate result according to the current iteration round number, including: Before each iteration, the iteration ripple wave calculated based on the iteration round number and the loss value is added to the iteration result output by the current iteration, and the result after the iteration ripple wave is added is taken as input to perform the next round iteration.
[0010] Optionally, the intent risk analysis and rule check are performed on the inference result after the artificial intelligence model inference is ended to obtain the intent risk update result, including: The inference result is segmented into multiple result intent segments, and the user input intent risk of each result intent segment is calculated; Based on the user input intent risk of each result intent segment, a high-risk content segment is identified; A first rule check risk result is obtained by performing a rule check on the high-risk content segment; The first rule is used to check the risk result to update the user input intent risk of each result intent segment; The updated user input intent risk of each result intent segment and the high-risk content segment are input into an artificial intelligence model, and an intent risk update result is obtained by reasoning of the artificial intelligence model.
[0011] Optionally, the method for inputting the updated user input intent risk of each result intent segment and the high-risk content segment into an artificial intelligence model, and obtaining an intent risk update result by reasoning of the artificial intelligence model, comprises: In each iteration, the current intermediate result is corrected based on the first iteration loss and the current iteration number; After each iteration is completed, an iteration ripple wave is added to the iteration result output by the current iteration, and the result after adding the iteration ripple wave is input for the next iteration.
[0012] Optionally, the method for outputting an optimized reasoning result or an error prompt according to the intent risk update result comprises: The intent risk update result is input into the artificial intelligence model, and an optimized reasoning result is obtained by integrating and optimizing the artificial intelligence model; The optimized reasoning result is subjected to rule checking to obtain a second rule checking risk result; If the second rule checking risk result meets a preset rule checking risk threshold, the optimized reasoning result is output, otherwise an error prompt is output.
[0013] Optionally, the method for analyzing the intent risk of user input content and calculating the user input intent risk comprises: The user input content is subjected to intent segmentation by a segmented intent recognition model to obtain a plurality of intent segments; The intent transmission chain is constructed based on the intent segments; The user input intent risk is calculated based on the intent transmission chain.
[0014] Optionally, the method for segmenting the user input content by a segmented intent recognition model to obtain a plurality of intent segments comprises The prompt word, context and user question of the current user conversation are read according to the user input content; The prompt word, context and user question of the current user conversation are input into the segmented intent recognition model, and the segmented intent recognition model is used to segment the intent to obtain a plurality of intent segments.
[0015] Optionally, the method for constructing the intent transmission chain based on the intent segments comprises: Analyze the correlation between the segmented intents, and construct a segmented intent correlation conduction matrix. Calculate the main stem flow path of the path in the segmented intent correlation conduction matrix, and construct an intent conduction chain.
[0016] Optionally, the user input intent risk is calculated based on the intent conduction chain, including: The user input intent risk of each intent segment in the intent conduction chain is calculated by an intent extraction model. The user input intent risk of each intent segment and the corresponding position in the intent conduction chain are weighted and calculated to obtain the user input intent risk.
[0017] Optionally, after the output of the optimized inference result, it further includes: The optimized inference result is manually marked to obtain a manually marked inference result. Based on the original inference result output by the artificial intelligence model, the optimized inference result, and the manually marked inference result, a training data set is constructed. The segmented intent recognition model is optimized and trained by using the training data set to obtain an optimized segmented intent recognition model.
[0018] Optionally, the rule check includes at least one of keyword event detection, semantic context analysis, and behavior comparison based on a graph model.
[0019] Optionally, the segmented intent recognition model includes a multi-modal based deep learning neural network and a self-attention mechanism based deep learning neural network.
[0020] To achieve the above purpose, the application further provides an artificial intelligence generated content review system, comprising: An intent risk analysis module for analyzing the intent risk of user input content while the artificial intelligence model is reasoning, and calculating the user input intent risk. An intent risk analysis update module for analyzing and checking the intent risk of the inference result of the artificial intelligence model according to the user input intent risk, and generating an intent risk update result. A risk filtering and intercepting module for outputting an optimized inference result or an error prompt according to the intent risk update result.
[0021] Optionally, the intent risk analysis update module includes: a first intent risk analysis updating unit, configured to, when the user input intent risk is not higher than the risk threshold and the artificial intelligence model reasoning is not ended, correct an intermediate result of reasoning of the artificial intelligence model in the process of artificial intelligence model reasoning, and perform intent risk analysis and rule checking on a result of reasoning after the artificial intelligence model reasoning is ended, to obtain an intent risk updating result; a second intent risk analysis updating unit, configured to, when the user input intent risk is higher than the risk threshold, perform intent risk analysis and rule checking on the result of reasoning after the artificial intelligence model reasoning is ended, to obtain the intent risk updating result.
[0022] Optionally, the intent risk analysis module comprises: an intent segmentation unit, configured to perform intent segmentation on the user input content by using a segmented intent recognition model, to obtain a plurality of intent segments; an intent conduction chain construction unit, configured to construct an intent conduction chain based on the intent segments; an intent risk calculation unit, configured to calculate the user input intent risk based on the intent conduction chain.
[0023] Optionally, the risk filtering and intercepting module is configured to: use the intent risk updating result as an input of the artificial intelligence model, integrate and optimize the result of reasoning by using the artificial intelligence model, to obtain an optimized result of reasoning; perform rule checking on the optimized result of reasoning, to obtain a second rule checking risk result; if the second rule checking risk result meets a preset rule checking risk threshold, output the optimized result of reasoning, otherwise output an error prompt.
[0024] Optionally, the method further comprises a segmented intent recognition model optimization training module, configured to: perform manual marking on the optimized result of reasoning, to obtain a manually marked result of reasoning; construct a training data set based on the original result of reasoning output by the artificial intelligence model, the optimized result of reasoning and the manually marked result of reasoning; use the training data set to perform optimization training on the segmented intent recognition model, to obtain an optimized segmented intent recognition model.
[0025] To achieve the above object, the present application further provides an artificial intelligence generated content review device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the artificial intelligence generated content review method according to any one of the above when executing the computer program.
[0026] To achieve the above object, the present application further provides a computer readable storage medium storing a computer program, wherein the computer program controls a device where the computer readable storage medium is located to execute the method for reviewing the artificial intelligence generated content according to any one of the above when running.
[0027] To achieve the above object, the present application further provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the method for reviewing the artificial intelligence generated content according to any one of the above.
[0028] Compared with the prior art, the method, system, device, medium and product for reviewing the artificial intelligence generated content provided by the present application can dynamically capture the hidden induced risks in the model input and timely correct the violation tendency in the reasoning process, thereby effectively improving the accuracy and timeliness of the review of the artificial intelligence generated content and guaranteeing the integrity and accuracy of the compliant output and preventing the security risks caused by the prompt word attack. BRIEF DESCRIPTION OF DRAWINGS
[0029] In order to more clearly illustrate the technical solutions of the present application, the drawings to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings described in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0030] Figure 1 is a flowchart of a method for reviewing artificial intelligence generated content provided by an embodiment of the present application; Figure 2 is another flowchart of a method for reviewing artificial intelligence generated content provided by an embodiment of the present application; Figure 3 is a structural block diagram of a system for reviewing artificial intelligence generated content provided by an embodiment of the present application; Figure 4 is a structural block diagram of a device for reviewing artificial intelligence generated content provided by an embodiment of the present application. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0032] Referring to Figure 1 , Figure 1is a flowchart of a method for reviewing AI-generated content according to an embodiment of the present application. The method for reviewing AI-generated content includes steps S1-S3: Step S1, while the AI model is reasoning, the user input content is analyzed for intent risk, and the user input intent risk is calculated. Step S2, according to the user input intent risk, the reasoning result of the AI model is analyzed for intent risk and rule checking, and an intent risk update result is generated. Step S3, according to the intent risk update result, the optimized reasoning result or error prompt is output.
[0033] By way of example, the embodiments of the present application analyze the user's input information and context information for intent risk while the AI model generates content, obtain the user input intent risk, and perform secondary intent risk analysis and rule checking on the reasoning result of the AI model according to the risk, to ensure the compliance of each dimension of the output content. If the secondary intent risk analysis and rule checking are not problematic, the reasoning result is optimized and output, i.e., the optimized generated content is output; otherwise, an error prompt is output to inform the user that the currently generated content has compliance risks and cannot be output.
[0034] It is worth noting that the ultimate goal of the existing prompt word attack is to make the AI model misjudge the existing interception filtering behavior and bypass the shielding and screening capability of the model itself. However, in order to induce the AI model to make a mistake, the context must contain some intent guidance. Therefore, by extracting the compliance relevance of the intent guidance and performing pre-adjustment of the output based on the compliance relevance during the reasoning process of the model, the embodiments of the present application can avoid the illegal content induced by the model.
[0035] In summary, the method for reviewing AI-generated content according to the embodiments of the present application can dynamically capture the hidden induction risk in the model input and timely correct the violation tendency in the model reasoning process, thereby effectively improving the accuracy and timeliness of AI-generated content review, while ensuring the integrity and accuracy of the compliance output and preventing security risks caused by prompt word attacks.
[0036] In an alternative embodiment, the step S1 includes steps S101-S103: Step S101, the user input content is segmented by a segmented intent recognition model to obtain multiple intent segments. Step S102, based on the intent segments, an intent conduction chain is constructed. Step S103, based on the intent conduction chain, the user input intent risk is calculated.
[0037] The embodiment of the application can split the user input content by a segmented intention recognition model to obtain multiple intention segments, construct an intention conduction chain based on the segments, and finally calculate the user input intention risk, which can effectively improve the comprehensiveness and accuracy of intention risk analysis, and can identify the blind area of traditional static review on the hidden risk of the intention level.
[0038] In an optional embodiment, the step S101 comprises: reading the prompt word, the context and the user question of the current conversation of the user according to the user input content; inputting the prompt word, the context and the user question of the current conversation of the user into a segmented intention recognition model, performing intention segmentation by the segmented intention recognition model, and obtaining multiple intention segments.
[0039] In an optional embodiment, the segmented intention recognition model comprises a multi-modal based deep learning neural network and a self-attention mechanism based deep learning neural network.
[0040] Exemplarily, the prompt word, the context and the user question of the current conversation of the user are input into the segmented intention recognition model as input parameters. Meanwhile, the current context is read. and the user question are input into the segmented intention recognition model as input parameters. , the context and the user question are input into the segmented intention recognition model as input parameters. , the segmented strategy of the user question , the prompt word is obtained. , is the total number of segments, and the segmented strategy ， is the segmentation basis (when represents the user question , represents the prompt word ), represents the cutting position.
[0041] The segmented intention recognition model contains two parts: an input unified conversion part and a segmented intention recognition part The input unified conversion part is a multi-modal based deep learning neural network, which converts the original data of different formats into a unified vector conversion, and converts the original input of the user into a unified vector input to facilitate subsequent unified model processing; the segmented intention recognition part For deep learning neural networks based on Transformers, this part unifies the transformation of the input. The converted unified vector input is used for intent recognition to obtain user prompts. User issues In the context The relevant intention segmentation strategy position .
[0042] Furthermore, the segmented intent recognition model traverses the intent segmentation strategy locations. Based on the k-th segment According to its corresponding segmentation criteria Read the segmented original text And according to the cutting position To obtain the intended segmentation Among them, the starting position of the cut , The function represents reading content. From the position to The content, represented as a set of multiple intent segments, is as follows: .
[0043] This invention segmentes intent by reading prompts, context, and user questions from the user's current dialogue and inputting them into a segmented intent recognition model. This fully integrates key information from the entire dialogue chain, avoiding the omission of hidden inducement logic related to the context due to isolated analysis of the input text. This makes intent segmentation more relevant to actual dialogue scenarios and more targeted, further improving the ability to identify hidden prompt attacks.
[0044] In one optional embodiment, step S102 includes: Analyze the correlation between intent segments and construct a segment intent correlation transmission matrix; Calculate the main flow path of the path in the segmented intent-related transmission matrix, and construct the intent transmission chain.
[0045] For example, the intention segmentation is obtained. ( After that, construct the segmented intent-related transmission matrix. : ; Among them, the piecewise correlation matrix Piecewise correlation in row i and column j The calculation method is as follows: ; in, Intended segmentation ( The m-th intent word vector obtained by performing simple intent identification on the word vectors corresponding to the content words in the content.
[0046] Then calculate the segmentation intent-related transmission matrix. The main flow path in the middle path, and then the segmented intent-related transmission matrix Constructing an intent transmission chain .
[0047] First, add the first segment to the intent delivery chain. The starting position. Starting from the first row, traverse the segmentation intent-related transmission matrix. Extract the row elements from each row that do not belong to the intention transmission chain. The segment corresponding to the column index of the largest non-1 among all column elements whose row number is different from the current row number is added as the next item to the intent propagation chain. At the end of the chain, starting from the next item, find its corresponding row and proceed to the next iteration until all segments are added to the intent propagation chain. After iteration, an intent transmission chain containing all segments is obtained. .
[0048] This invention constructs an intent-related transmission matrix by first analyzing the correlation between intent segments, and then calculating the main flow path of the path in the matrix to build an intent transmission chain. This can quantify the correlation strength between intent segments, avoid missing hidden coherent guiding contexts due to disordered segment integration, and make the intent transmission chain more closely match the true intent flow of user input. This provides a basis for subsequent accurate calculation of user input intent risks and identification of prompt word attack logic linked between segments, further improving the accuracy and targeting of intent risk analysis.
[0049] In one optional embodiment, step S103 includes: The user input intent risk of each intent segment in the intent transmission chain is calculated using the intent extraction model; The user input intent risk is obtained by weighting the risk of each intent segment and its corresponding position in the intent transmission chain.
[0050] For example, computing intent transmission chain The i-th node Corresponding intention segmentation User input intent risk : ; in, Indicated by segmentation Input, using the intent extraction model Risks arising from inference of user input intent.
[0051] Then based on the intent transmission chain The intention transmission chain is obtained through integrated calculation. User input intent risk : ; in, Intent transmission chain The i-th node Corresponding intent segmentation In the user input intent segmentation set The position index in the middle, and ， .
[0052] This invention first uses an intent extraction model to calculate the individual intent risk of each segment corresponding to each node in the intent transmission chain, and then performs a weighted calculation based on the position of each segment in the transmission chain. This quantifies the risk intensity of a single intent segment and considers the influence weight of segments at different positions on the overall intent. This allows the final calculation of user input intent risk to be more in line with the real intent transmission logic and to more accurately capture the hidden inducement risks connected between segments.
[0053] In one optional embodiment, step S2 includes: When the user input intent risk is not higher than the risk threshold and the AI model inference has not ended, the intermediate results of the AI model inference are corrected during the AI model inference process, and intent risk analysis and rule checks are performed on the inference results after the AI model inference ends to obtain the intent risk update result. When the risk of the user's input intent exceeds the risk threshold, after the artificial intelligence model has finished reasoning, the reasoning result is subjected to intent risk analysis and rule checking to obtain an updated intent risk result.
[0054] This invention, through differentiated processing based on risk level, not only enables dynamic intervention during the reasoning process to prevent risk amplification, but also allows for review coverage of different risk scenarios, improving the accuracy and comprehensiveness of AI-generated content review and effectively preventing prompt word attacks.
[0055] In one optional embodiment, the step of correcting the intermediate results of the artificial intelligence model's inference process includes: Read the current iteration number, intermediate results, and current loss of the artificial intelligence model; The intermediate result is corrected based on the risk of the user's input intent and the current loss; Based on the corrected intermediate results, the next iteration of the artificial intelligence model inference is performed according to the current iteration round number.
[0056] Further, after the next iteration of the artificial intelligence model inference based on the corrected intermediate results and according to the current iteration round number, the process includes: Before each iteration, an iterative ripple wave calculated based on the iteration number and loss value is added to the output of the current iteration. The result after adding the iterative ripple wave is then used as input for the next iteration.
[0057] For example, in completing user input intent risk After the calculation, the first step is to read the artificial intelligence model, such as the large language model. In the reasoning process, before the large language model has finished reasoning, the user input intent risk is first assessed. Is it less than or equal to the iterative derivation correction coefficient? (A preset risk threshold) When this condition is met, the current large language model is completed. After one round of iterations, pause the inference iteration of the large language model and read the large language model. inference iteration state and from the iterative state Read the current iteration round number Intermediate results Compared with current losses The intermediate results are processed as follows. Make corrections: ; Among them, the function Representing a large language model The activation function used in the output layer. For large language models The first iteration loss, This indicates broadcast addition.
[0058] Intermediate results After the update is complete, proceed to the next iteration. After each subsequent iteration, the results must be updated. Add iterative ripple waves Then, it is fed into the next iteration (excluding the output of the final iteration), that is: ; Among them, iterative ripple waves The calculation method is as follows: ; in, This represents the round number of the current iteration.
[0059] It is worth noting that the embodiments of the present invention combine the risk of user input intent with the current loss to correct intermediate results, which can effectively block the amplification of hidden inducement risks during the reasoning process, while not affecting reasoning efficiency, and provide dynamic protection for outputting compliant and demand-appropriate results.
[0060] Furthermore, by adding an iterative ripple wave calculated based on the number of iteration rounds and the loss value to the forward iteration result of each iteration, it is possible to form a coherent inference logic chain between the input of subsequent iterations and the corrected intermediate results, avoiding inference gaps. It is also possible to dynamically adjust the iteration direction based on the number of iteration rounds and the loss value, thereby optimizing the inference path while maintaining the compliance of intermediate results and improving the accuracy of subsequent iterations. Moreover, it eliminates the need for repeated intention risk correction, thus balancing the compliance and efficiency of inference.
[0061] In one optional embodiment, the step of performing intent risk analysis and rule checks on the inference results after the artificial intelligence model inference is completed, to obtain intent risk update results, includes: The inference results are segmented into intent segments to obtain multiple result intent segments, and the user input intent risk of each result intent segment is calculated. Based on the user input intent risk of each result intent segment, identify high-risk content segments; The high-risk content is segmented and subjected to rule checks to obtain the first rule check risk result; The user input intent risk of each result intent segment is updated using the first rule to check the risk results. The updated user input intent risk of each result intent segment and the high-risk content segment are used as input to the artificial intelligence model, and the intent risk update result is obtained through inference by the artificial intelligence model.
[0062] In one alternative embodiment, the rule check includes at least one of graph model-based behavior comparison, keyword event detection, and semantic context parsing.
[0063] It should be noted that keyword event detection is based on rule matching between text and word vectors, statistical thresholds, and the total number of weighted sensitive or abnormal content identified. Semantic context parsing, on the other hand, is based on interception rules that analyze word meaning, syntax, and contextual relationships, combined with knowledge graphs and pre-trained models, to identify the total number of weighted sensitive or abnormal content identified. Behavior comparison based on graph models involves constructing a user behavior relationship comparison interception rule graph, using content path deviations (such as frequency, timing, and association anomalies), combined with dynamic thresholds, to calculate the total number of weighted sensitive or abnormal content identified.
[0064] In one optional embodiment, the step of using the updated user input intent risk of each result intent segment and the high-risk content segment as input to the artificial intelligence model, and obtaining the intent risk update result through the artificial intelligence model inference, includes: In each iteration, the current intermediate result is corrected based on the loss from the first iteration and the current iteration number; After each iteration, an iterative ripple wave is added to the output of the current iteration, and the result after adding the iterative ripple wave is used as input for the next iteration.
[0065] For example, when a user inputs intent risk Or, if the large language model inference has ended, obtain the final inference result of the large language model. Based on the final reasoning result As input, execute the segmented intent recognition model. The model yields the final inference result. Segmentation strategy , For the final result Total number of segments, segmentation strategy , For the basis of segmentation ( This is represented as the final reasoning result. ), Indicates the cutting position.
[0066] Then calculate the segmentation strategy. Segmented user input intent risk (Its calculation logic is related to the aforementioned risks associated with user input intent) The calculation logic is consistent with that of the previous one, and will not be repeated here. From this, user input intent risks are selected. Greater than the coefficient of overlap and competition in reasoning Segmentation strategy and obtain the segmentation strategy. The corresponding final reasoning result Segmentation .
[0067] Furthermore, the final reasoning result Segmentation Keyword event detection, semantic context parsing, and graph model-based behavior comparison are performed to obtain the risk results of rule-based checks. Risks related to user input intent After the update, it will be segmented. Risks related to user input intent Feed into large language model By performing secondary reasoning, we obtain the segmentation. The second update result : ; In this inference process, the results of each iteration are infused with iterative ripple waves using the same method. The results are then fed into the next iteration as input, along with the intermediate results from each iteration. The correction method is as follows: ; in, For large language models The first iteration loss, Indicates broadcast addition. This represents the round number of the current iteration.
[0068] It should be noted that in this embodiment of the invention, if the risk of the user's input intent is not higher than the risk threshold, the intermediate results of the AI model's inference are corrected during the inference process. However, regardless of whether the risk of the user's input intent is higher than the risk threshold, this embodiment of the invention will perform intent risk analysis and rule checks on the inference results after the AI model's inference is completed, and obtain an updated intent risk result. Therefore, the above processing steps are consistent regardless of whether the risk of the user's input intent is higher than the risk threshold after the AI model's inference is completed.
[0069] This invention, through a dual assessment of intent risk update and rule check risk of the inference results, enables a more comprehensive risk judgment. It also proactively corrects high-risk content through secondary inference, rather than simply blocking it. This improves the accuracy and compliance of the review while ensuring the integrity and usability of the output results. By dynamically correcting intermediate results based on the loss of the first iteration and the current iteration number in each iteration, the output deviation in the secondary inference process can be optimized. Furthermore, the ripple mechanism maintains the integrity of the inference, ultimately making the intent risk update results more accurate and compliant, thus improving the efficiency and quality of secondary inference.
[0070] In one optional embodiment, step S3 includes: The intention risk update result is used as the input of the artificial intelligence model, and the artificial intelligence model is used to integrate and optimize it to obtain the optimized inference result; The optimized reasoning result is subjected to rule checking to obtain the second rule checking risk result; If the risk check result of the second rule meets the preset risk check threshold, the optimized inference result is output; otherwise, an error message is output.
[0071] For example, to ensure the quality of the final output of the interception and replacement results, when a second update has been performed, the second update result is used. Replace the final inference result The intention update result was then obtained. As input, call the large language model Using custom optimization prompts, the results were refined and optimized to obtain the final integrated and optimized inference results. The aforementioned risk intent check and update logic will not be included in the model inference process.
[0072] Furthermore, before outputting the results, the integrated optimization results are... Keyword event detection, semantic context parsing, and graph model-based behavior comparison are performed to obtain optimized results and risk assessment results. ,when When the risk result of the second rule check does not meet the preset risk threshold, an error message is returned. When the risk check result of the second rule meets the preset risk check threshold, the model outputs the integrated and optimized inference result. .
[0073] The embodiments of the present invention enhance the practicality and relevance of the reasoning results through integration and optimization, and form a dual compliance verification through additional rule checking, effectively avoiding the omission of judgment in a single link, ensuring that the optimized reasoning results output are compliant not only at the level of intent risk, but also at the level of rules, further improving the reliability and security of the results obtained by users.
[0074] See Figure 2 , Figure 2 This is another flowchart illustrating a method for reviewing AI-generated content provided in an embodiment of the present invention. For example... Figure 2 As shown, in specific implementation, the user input intent is first segmented, a segmented intent relevance matrix is constructed, and an intent transmission chain is established. The user input intent risk is calculated and integrated. If the user input intent risk is not higher than the threshold and the large language model has not ended, the intermediate results are corrected and iterated. Then, the final inference result is used as input for segmented intent recognition. After segmented risk calculation, screening, rule checking, and risk updating, it is sent to the large language model for secondary sorting. The secondary update result is used to replace and integrate the optimization. Finally, the optimized inference result is subjected to rule checking. If the rule check risk result is greater than 0, an error message is returned; otherwise, the optimized sorting result is output.
[0075] In one alternative embodiment, after outputting the optimized inference result, the method further includes: The optimized reasoning results are manually labeled to obtain manually labeled reasoning results; A training dataset is constructed based on the original inference results output by the artificial intelligence model, the optimized inference results, and the manually labeled inference results; The segmented intent recognition model is optimized and trained using the training dataset to obtain the optimized segmented intent recognition model.
[0076] For example, the integrated and optimized reasoning results The effects were manually checked and verified based on the original inference results of the large model. Optimized reasoning results The training dataset is formed by combining the inference results with manually labeled data. Backpropagation is then used to refine the segmented intent recognition model. Perform a second training update.
[0077] It is worth noting that after the inference is completed, a second training is performed based on the review results of the result data, which ensures the automatic updating of the model and improves the comprehensiveness and accuracy of the review.
[0078] See Figure 3 , Figure 3 This is a structural block diagram of an AI-generated content review system provided in an embodiment of the present invention. The AI-generated content review system includes: The intent risk analysis module 21 is used to perform intent risk analysis on user input content and calculate user input intent risk while the artificial intelligence model is reasoning. The intent risk analysis and update module 22 is used to perform intent risk analysis and rule checks on the reasoning results of the artificial intelligence model based on the user input intent risk, and generate intent risk update results. The risk filtering and interception module 23 is used to output an optimized inference result or error message based on the intention risk update result.
[0079] In one optional embodiment, the AI-generated content review system further includes a segmented intent recognition model optimization training module, used for: The optimized reasoning results are manually labeled to obtain manually labeled reasoning results; A training dataset is constructed based on the original inference results output by the artificial intelligence model, the optimized inference results, and the manually labeled inference results; The segmented intent recognition model is optimized and trained using the training dataset to obtain the optimized segmented intent recognition model.
[0080] In one optional embodiment, the intent risk analysis module 21 includes: The intent segmentation unit is used to segment user input content into multiple intent segments using a segmented intent recognition model. An intent transmission chain construction unit is used to construct an intent transmission chain based on the intent segmentation. The intent risk calculation unit is used to calculate the risk of user input intent based on the intent transmission chain.
[0081] In one alternative embodiment, the intent segmentation unit is configured to: Read the prompts, context, and user questions from the current conversation based on the user's input. The prompts, context, and user questions of the current dialogue are input into the segmented intent recognition model. The intent is segmented by the segmented intent recognition model to obtain multiple intent segments.
[0082] In one alternative embodiment, the intent transmission chain construction unit is configured to: Analyze the correlation between intent segments and construct a segment intent correlation transmission matrix; Calculate the main flow path of the path in the segmented intent-related transmission matrix, and construct the intent transmission chain.
[0083] In one optional embodiment, the intent risk calculation unit is configured to: The user input intent risk of each intent segment in the intent transmission chain is calculated using the intent extraction model; The user input intent risk is obtained by weighting the risk of each intent segment and its corresponding position in the intent transmission chain.
[0084] In one optional embodiment, the intent risk analysis update module 22 includes: The first intent risk analysis and update unit is used to correct the intermediate results of the artificial intelligence model during the artificial intelligence model inference process when the user input intent risk is not higher than the risk threshold and the artificial intelligence model inference has not ended, and to perform intent risk analysis and rule checking on the inference results after the artificial intelligence model inference is completed, so as to obtain the intent risk update result. The second intent risk analysis and update unit is used to perform intent risk analysis and rule checks on the reasoning results after the artificial intelligence model inference is completed when the user input intent risk is higher than the risk threshold, so as to obtain the intent risk update result.
[0085] In one optional embodiment, the risk filtering and interception module 23 is configured to: The intention risk update result is used as the input of the artificial intelligence model, and the artificial intelligence model is used to integrate and optimize it to obtain the optimized inference result; The optimized reasoning result is subjected to rule checking to obtain the second rule checking risk result; If the risk check result of the second rule meets the preset risk check threshold, the optimized inference result is output; otherwise, an error message is output.
[0086] It should be noted that the AI-generated content review system provided in this embodiment of the invention is used to execute all the process steps of the AI-generated content review method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.
[0087] See Figure 4 , Figure 4 This is a structural block diagram of an AI-generated content review device provided in an embodiment of the present invention. The AI-generated content review device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the above-described embodiments of the AI-generated content review methods, for example...
[0088] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the AI-generated content review device.
[0089] The AI-generated content review device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of an AI-generated content review device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the AI-generated content review device may also include input / output devices, network access devices, buses, etc.
[0090] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the AI-generated content review device, connecting various parts of the entire AI-generated content review device via various interfaces and lines.
[0091] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the AI-generated content review device by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0092] If the modules / units integrated into the AI-generated content review device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0093] Furthermore, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for reviewing artificial intelligence-generated content as described in any of the above embodiments.
[0094] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for reviewing content generated by artificial intelligence, characterized in that, include: While the artificial intelligence model is reasoning, intent risk analysis is performed on the user input content to calculate the risk of user input intent. Based on the user input intent risk, the inference results of the artificial intelligence model are subjected to intent risk analysis and rule checks to generate intent risk update results; Based on the intention risk update result, output the optimized inference result or error message.
2. The method for reviewing AI-generated content as described in claim 1, characterized in that, The step of performing intent risk analysis and rule checks on the intermediate or final inference results of the artificial intelligence model based on the user input intent risk, and generating an intent risk update result, includes: When the user input intent risk is not higher than the risk threshold and the AI model inference has not ended, the intermediate results of the AI model inference are corrected during the AI model inference process, and intent risk analysis and rule checks are performed on the inference results after the AI model inference ends to obtain the intent risk update result. When the risk of the user's input intent exceeds the risk threshold, after the artificial intelligence model has finished reasoning, the reasoning result is subjected to intent risk analysis and rule checking to obtain an updated intent risk result.
3. The method for reviewing AI-generated content as described in claim 2, characterized in that, The process of correcting intermediate inference results of the artificial intelligence model during inference includes: Read the current iteration number, intermediate results, and current loss of the artificial intelligence model; The intermediate result is corrected based on the risk of the user's input intent and the current loss; Based on the corrected intermediate results, the next iteration of the artificial intelligence model inference is performed according to the current iteration round number.
4. The method for reviewing AI-generated content as described in claim 3, characterized in that, Following the next iteration of the AI model inference based on the corrected intermediate results and the current iteration round number, the process includes: Before each iteration, an iterative ripple wave calculated based on the iteration number and loss value is added to the output of the current iteration. The result after adding the iterative ripple wave is then used as input for the next iteration.
5. The method for reviewing AI-generated content as described in claim 2, characterized in that, After the artificial intelligence model completes its inference, the inference results are subjected to intent risk analysis and rule checks to obtain updated intent risk results, including: The inference results are segmented into intent segments to obtain multiple result intent segments, and the user input intent risk of each result intent segment is calculated. Based on the user input intent risk of each result intent segment, identify high-risk content segments; The high-risk content is segmented and subjected to rule checks to obtain the first rule check risk result; The user input intent risk of each result intent segment is updated using the first rule to check the risk results. The updated user input intent risk of each result intent segment and the high-risk content segment are used as input to the artificial intelligence model, and the intent risk update result is obtained through inference by the artificial intelligence model.
6. The method for reviewing AI-generated content as described in claim 5, characterized in that, The step of using the updated user input intent risk of each result intent segment and the high-risk content segment as input to the artificial intelligence model, and obtaining the intent risk update result through artificial intelligence model reasoning, includes: In each iteration, the current intermediate result is corrected based on the loss from the first iteration and the current iteration number; After each iteration, an iterative ripple wave is added to the output of the current iteration, and the result after adding the iterative ripple wave is used as input for the next iteration.
7. The method for reviewing AI-generated content as described in claim 1, characterized in that, The step of outputting optimized inference results or error messages based on the intention risk update results includes: The intention risk update result is used as the input of the artificial intelligence model, and the artificial intelligence model is used to integrate and optimize it to obtain the optimized inference result; The optimized reasoning result is subjected to rule checking to obtain the second rule checking risk result; If the risk check result of the second rule meets the preset risk check threshold, the optimized inference result is output; otherwise, an error message is output.
8. The method for reviewing AI-generated content as described in claim 1, characterized in that, The process of performing intent risk analysis on user input and calculating user input intent risk includes: The user input is segmented into multiple intent segments using a segmented intent recognition model. Based on the intent segmentation, an intent transmission chain is constructed; Based on the intent transmission chain, the risk of user input intent is calculated.
9. The method for reviewing AI-generated content as described in claim 8, characterized in that, The user input is segmented using a segmented intent recognition model to obtain multiple intent segments, including... Read the prompts, context, and user questions from the current conversation based on the user's input. The prompts, context, and user questions of the current dialogue are input into the segmented intent recognition model. The intent is segmented by the segmented intent recognition model to obtain multiple intent segments.
10. The method for reviewing AI-generated content as described in claim 8, characterized in that, The construction of the intent transmission chain based on the intent segmentation includes: Analyze the correlation between intent segments and construct a segment intent correlation transmission matrix; Calculate the main flow path of the path in the segmented intent-related transmission matrix, and construct the intent transmission chain.
11. The method for reviewing AI-generated content as described in claim 8, characterized in that, The calculation of user input intent risk based on the intent transmission chain includes: The user input intent risk of each intent segment in the intent transmission chain is calculated using the intent extraction model; The user input intent risk is obtained by weighting the risk of each intent segment and its corresponding position in the intent transmission chain.
12. The method for reviewing AI-generated content as described in claim 8, characterized in that, Following the output of the optimized inference result, the following is also included: The optimized reasoning results are manually labeled to obtain manually labeled reasoning results; A training dataset is constructed based on the original inference results output by the artificial intelligence model, the optimized inference results, and the manually labeled inference results; The segmented intent recognition model is optimized and trained using the training dataset to obtain the optimized segmented intent recognition model.
13. The method for reviewing AI-generated content as described in any one of claims 5 or 8, characterized in that, The rule check includes at least one of keyword event detection, semantic context parsing, and graph model-based behavior comparison.
14. The method for reviewing AI-generated content as described in any one of claims 7 or 9-10, characterized in that, The segmented intent recognition model includes a multimodal deep learning neural network and a self-attention mechanism-based deep learning neural network.
15. A system for reviewing content generated by artificial intelligence, characterized in that, include: The intent risk analysis module is used to perform intent risk analysis on user input content while the artificial intelligence model is reasoning, and to calculate the intent risk of user input. The intent risk analysis and update module is used to perform intent risk analysis and rule checks on the reasoning results of the artificial intelligence model based on the user-input intent risk, and generate intent risk update results. The risk filtering and interception module is used to output optimized inference results or error prompts based on the intention risk update results.
16. The AI-generated content review system as described in claim 15, characterized in that, The intent risk analysis update module includes: The first intent risk analysis and update unit is used to correct the intermediate results of the artificial intelligence model during the artificial intelligence model inference process when the user input intent risk is not higher than the risk threshold and the artificial intelligence model inference has not ended, and to perform intent risk analysis and rule checking on the inference results after the artificial intelligence model inference is completed, so as to obtain the intent risk update result. The second intent risk analysis and update unit is used to perform intent risk analysis and rule checks on the reasoning results after the artificial intelligence model inference is completed when the user input intent risk is higher than the risk threshold, so as to obtain the intent risk update result.
17. The AI-generated content review system as described in claim 15, characterized in that, The intent risk analysis module includes: The intent segmentation unit is used to segment user input content into multiple intent segments using a segmented intent recognition model. An intent transmission chain construction unit is used to construct an intent transmission chain based on the intent segmentation. The intent risk calculation unit is used to calculate the risk of user input intent based on the intent transmission chain.
18. The AI-generated content review system as described in claim 15, characterized in that, The risk filtering and interception module is used for: The intention risk update result is used as the input of the artificial intelligence model, and the artificial intelligence model is used to integrate and optimize it to obtain the optimized inference result; The optimized reasoning result is subjected to rule checking to obtain the second rule checking risk result; If the risk check result of the second rule meets the preset risk check threshold, the optimized inference result is output; otherwise, an error message is output.
19. The AI-generated content review system as described in claim 15, characterized in that, It also includes a segmented intent recognition model optimization training module, used for: The optimized reasoning results are manually labeled to obtain manually labeled reasoning results; A training dataset is constructed based on the original inference results output by the artificial intelligence model, the optimized inference results, and the manually labeled inference results; The segmented intent recognition model is optimized and trained using the training dataset to obtain the optimized segmented intent recognition model.
20. A device for reviewing content generated by artificial intelligence, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method for reviewing artificial intelligence-generated content as described in any one of claims 1 to 14.
21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the method for reviewing artificial intelligence-generated content as described in any one of claims 1 to 14.
22. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method for reviewing artificial intelligence-generated content as described in any one of claims 1 to 14.