Large-scale interactive information security filtering method and device

Abstract

This invention provides a method and apparatus for security filtering of interactive information in a large-scale model. The method includes: receiving query information input by a user; inputting the query information into a configurable multi-layer filtering module for filtering; if the multi-layer filtering module does not trigger interception, submitting the filtered query information to a large language model for inference; the large language model continuously outputting streaming text during the inference process and accumulating the continuously outputting streaming text into a buffer; when the accumulated text in the buffer meets a preset trigger condition, performing a sensitive content check on the accumulated text in the buffer; if sensitive content is detected in the accumulated text, terminating the inference process of the large language model; if no sensitive content is detected in the accumulated text, outputting a complete answer text after the inference process is completed. The method and apparatus of this invention can filter sensitive information in real time and accurately, avoiding the exposure of risky information.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a large model interaction information security filtering method and device. Background Art

[0002] With the wide application of large language models, the content security risks in their interaction processes have become increasingly prominent. The existing technologies mainly rely on traditional machine learning classifiers (such as support vector machines, random forests) for sensitive information identification, but this method has complex modeling, limited generalization ability, and poor recognition effect on mutated words outside the training set. The method based on natural language processing performs similarity matching through word segmentation and word vector calculation, but it is difficult to effectively handle circumvention behaviors with spelling mistakes, character insertions, or homophone substitutions. For example, when writing "sensitive word" as "min gan word", it will result in missed detections. In addition, simply by adding ideological constraints to the input prompt, the actual interception success rate is relatively low, and it is difficult to respond reliably especially when facing maliciously constructed queries. More critically, the existing technical solutions generally lack a real-time streaming monitoring mechanism for the model output process, and cannot interrupt the output of sensitive content in a timely manner during the text generation process, resulting in the complete exposure of risk information. These defects make it difficult for the existing technologies to achieve a balance among accuracy, efficiency, and real-time performance, and cannot meet the growing demand for secure interaction of large models.

[0003] Therefore, there is a lack of a large model interaction information security filtering method and device in the prior art that can dynamically adjust filtering strategies, integrate multiple detection mechanisms, and have real-time interruption capabilities. Summary of the Invention

[0004] Based on this, it is necessary to provide a large model interaction information security filtering method and device for the above technical problems.

[0005] In a first aspect, the present invention provides a large model interaction information security filtering method, which includes:

[0006] Receiving query information input by a user;

[0007] Inputting the query information into a configurable multi-layer filtering module for filtering processing, where the multi-layer filtering module at least includes a sensitive word filtering module, a named entity recognition module, and a sentiment analysis module, and the execution order, enabling status, and internal parameters of each module can be dynamically configured;

[0008] If the query information input into the multi-layer filtering module does not trigger interception, submitting the filtered query information to a large language model for inference;

[0009] The large language model continuously outputs streaming text during the inference process and accumulates the continuously outputting streaming text into a buffer. When the accumulated text in the buffer meets the preset trigger conditions, a sensitive content check is performed on the accumulated text in the buffer. If sensitive content is detected in the accumulated text, the inference process of the large language model is terminated. If no sensitive content is detected in the accumulated text, the complete answer text is output after the inference process is completed.

[0010] Optionally, after receiving the query information input by the user, the system also includes:

[0011] Input the query information into a lightweight risk prediction model to assess the risk of the user's input query information and generate a risk value;

[0012] Based on the risk value, a target policy is selected from multiple predefined filtering policies and output to the multi-layer filtering module. The multiple predefined filtering policies include fast channel, standard channel and strict channel and the channel threshold corresponding to each channel. The execution order, enabling status and internal parameters of each module in the multi-layer filtering module are also specified.

[0013] Adjust the execution order, enabling status, and / or internal parameters of each module in the multi-layer filtering module according to the target strategy.

[0014] Optionally, after no sensitive content is detected in the accumulated text, the following steps are also included:

[0015] Detect whether the accumulated text contains content of a preset type that needs to be modified;

[0016] If the accumulated text contains content of a preset type that needs to be modified, the current accumulated text is modified in a safe manner while preserving semantics, resulting in the modified accumulated text.

[0017] Optionally, after the multi-layer filtering module for query information input does not trigger interception, the following also applies:

[0018] Obtain the user's historical context determination information, which includes the types of sensitive words that have been triggered, the entities that have been mentioned, the sentiment tendency, and the risk level.

[0019] Based on historical context information, a comprehensive security assessment is performed on the current query information to obtain the current context information.

[0020] If the current context information is greater than or equal to the preset risk threshold, execute the preset intervention operation.

[0021] Optionally, the sensitive word filtering module is an Aho-Corasick automaton using Lewinstein fuzzy matching, and the step of inputting the query information into a configurable multi-layer filtering module for filtering includes:

[0022] Construct an N-gram index that runs parallel to the Aho-Corasick automaton using Lewinstein fuzzy matching, to record the mapping relationship between each N-gram fragment in the sensitive word library and its corresponding sensitive word;

[0023] By querying the N-gram index, the fragment similarity between the query information and each sensitive word can be quickly calculated.

[0024] Based on fragment similarity, a subset of candidate sensitive words is selected from the sensitive word database. The subset of candidate sensitive words includes all sensitive words that have potential fuzzy matching with the query information.

[0025] For a subset of candidate sensitive words, a fuzzy search is performed on the Aho-Corasick automaton using Lewinstein fuzzy matching.

[0026] Optionally, the named entity recognition module employs a recognition mechanism that integrates regular expressions, a dictionary, and a machine learning model. The step of inputting the query information into a configurable multi-layer filtering module for filtering includes:

[0027] Based on the preset regular expression, identify the first entity in the query information;

[0028] The system queries a pre-defined dictionary to identify the second entity in the query information.

[0029] The pre-defined BERT-based NER machine learning model is used to understand the context, identify the third entity in the query information, and verify the first and second entities. The verified first, second and third entities are then classified and integrated to obtain a set of associated entities.

[0030] The system uses preset keywords to replace entities in the query information that are the same as entities in the associated entity set, thus obtaining filtered query information.

[0031] Optionally, the sentiment analysis module is a Roberta-based pre-trained language model, and the step of inputting query information into a configurable multi-layer filtering module for filtering includes:

[0032] The sentiment analysis module is used to perform sentiment analysis on the query information and output the sentiment score and content type.

[0033] If the sentiment score is less than or equal to the first preset threshold or the content type is inappropriate, blocking will be triggered.

[0034] If the sentiment score is greater than the first preset threshold and less than or equal to the second preset threshold, the filtered query information will be submitted to the large language model.

[0035] If the sentiment score is greater than the second preset threshold, the filtered query information will be submitted to the large language model.

[0036] Optionally, before accumulating the continuously streamed output text into the buffer, the method further includes:

[0037] During the reasoning process of generating each streaming output text in the large language model, the real-time activation values ​​of the preset key neuron set in the internal neural network of the large language model are captured in real time; the key neuron set is a set of neurons that are determined by pre-analysis and have high responsiveness to predefined sensitive concept categories;

[0038] Based on the real-time activation values ​​of key neuron sets, calculate the instantaneous risk intensity of the current generation step for each sensitive concept category;

[0039] The instantaneous risk intensity of each sensitive concept category is weighted and accumulated to generate a comprehensive risk trajectory value;

[0040] When the comprehensive risk trajectory value is greater than or equal to the preset intervention threshold, a real-time intervention operation is triggered.

[0041] Alternatively, the lightweight risk prediction model can be a text convolutional neural network model or a small Transformer model.

[0042] Secondly, the present invention provides a large-scale interactive information security filtering device, comprising:

[0043] The acquisition module is used to acquire the query information input by the user;

[0044] The multi-layer filtering module, connected to the acquisition module, is used to filter query information. The multi-layer filtering module includes at least a sensitive word filtering module, a named entity recognition module, and a sentiment analysis module. The execution order, activation status, and internal parameters of each module can be dynamically configured. If the multi-layer filtering module does not trigger interception when query information is input, the filtered query text is submitted to the large language model for inference.

[0045] The large language model, connected to the multi-layer filtering module, continuously outputs streaming text during the inference process and accumulates the continuously outputting streaming text into a buffer. When the accumulated text in the buffer meets the preset trigger conditions, a sensitive content check is performed on the accumulated text in the buffer. If sensitive content is detected in the accumulated text, the inference process of the large language model is terminated. If no sensitive content is detected in the accumulated text, the complete answer text is output after the inference process is completed.

[0046] The large-scale interactive information security filtering method and device provided by this invention firstly achieves efficient and accurate identification of mutated sensitive words by employing the Aho-Corasick algorithm integrating Levinstein fuzzy matching and an N-gram pre-filtering mechanism, solving the problem of missed detection of spelling errors and deliberate avoidance behaviors in traditional methods. Secondly, it constructs a dynamic defense system covering the entire process of "input preprocessing - real-time monitoring during inference - pre-output truncation". Through dynamically configurable multi-layered filtering modules (sensitive word filtering module, named entity recognition module, and sentiment analysis module), the input is deeply cleaned. Based on this, continuous content checks are performed on the streaming output text of the large language model, enabling real-time detection and immediate termination of inference during the generation of sensitive content, thus blocking risk information before it is fully exposed to the user, achieving real-time intervention and greatly reducing security risks. Thirdly, it innovatively introduces a comprehensive risk assessment mechanism based on historical context. By analyzing the evolution path of sensitive word types, entity associations, and sentiment tendencies, the system can intelligently identify potential escalation of risky behaviors. Finally, the modular design and dynamically configurable parameters of the entire solution enable it to flexibly adapt to different application scenarios, ranging from rapid response to strict control, greatly improving the system's practicality and scalability. Furthermore, it achieves synergistic breakthroughs in accuracy, efficiency, and real-time performance, providing key technical support for building a secure and reliable large-scale model interactive environment. Attached Figure Description

[0047] Figure 1a A flowchart illustrating a large-model interactive information security filtering method provided in an embodiment of the present invention;

[0048] Figure 1b Another flowchart illustrating the large-model interactive information security filtering method provided in this embodiment of the invention;

[0049] Figure 2a A schematic diagram of a circuit module structure of the large-scale interactive information security filtering device provided in an embodiment of the present invention;

[0050] Figure 2b A schematic diagram of another circuit module structure of the large model interactive information security filtering device provided in this embodiment of the invention;

[0051] Figure 3 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] like Figure 1a As shown, this invention provides a method for secure filtering of large-scale model interaction information, the method comprising:

[0054] Step S10: Receive query information input by the user;

[0055] In one optional embodiment of the present invention, such as Figure 1b As shown, after step S10, the following steps are also included:

[0056] Step S11: Input the query information into the lightweight risk prediction model, perform risk assessment on the user-input query information, and generate a risk value;

[0057] The lightweight risk prediction model is either a pre-trained Text Convolutional Neural Network (TextCNN) model or a pre-trained small Transformer model. Those skilled in the art can choose according to actual needs, and no limitation is made here.

[0058] Step S12: Based on the risk value, select the target policy from multiple predefined filtering policies and output the target policy to the multi-layer filtering module. The multiple predefined filtering policies include fast channel, standard channel and strict channel and the channel threshold corresponding to each channel. The execution order, enabling status and internal parameters of each module in the multi-layer filtering module are also specified.

[0059] For the channel thresholds corresponding to each channel, those skilled in the art can flexibly set the channel thresholds corresponding to each channel according to actual needs, and there is no limitation here. Optionally, the channel threshold corresponding to the fast channel is [0, 0.3), the channel threshold corresponding to the standard channel is [0.3, 0.7), and the channel threshold corresponding to the strict channel is [0.7, 1]. It should be noted that if the lightweight risk prediction model adopts a text convolutional neural network (TextCNN) model or a small Transformer model, the risk value calculated after inputting the query information into the lightweight risk prediction model is in the range of [0, 1]. Therefore, the channel thresholds of each channel should be set in the range of [0, 1].

[0060] In one optional implementation, the execution order and activation status of each module in the multi-layer filtering module of the predefined filtering strategy are as follows: Fast channel (channel threshold [0, 0.3)): Enable the sensitive word filtering module; Standard channel (channel threshold [0.3, 0.7)): Enable the sensitive word filtering module and the named entity recognition module in sequence; Strict channel (channel threshold [0.7, 1]): Enable all modules (e.g., enable the sensitive word filtering module, the named entity recognition module, and the sentiment analysis module in sequence). For example, if the risk value is 0.75, the strict channel is selected, and the sensitive word filtering module, the named entity recognition module, and the sentiment analysis module are enabled in sequence.

[0061] In one specific embodiment of the present invention, the predefined filtering strategy can be as described in Table 1:

[0062] Table 1 Predefined filtering strategies

[0063]

[0064] It should be noted that the predefined filtering strategies can be flexibly selected by those skilled in the art according to actual needs, and no restrictions are imposed here.

[0065] Step S13: Adjust the execution order, enabling status and / or internal parameters of each module in the multi-layer filtering module according to the target strategy.

[0066] Step S20: Input the query information into the configurable multi-layer filtering module for filtering. The multi-layer filtering module includes at least a sensitive word filtering module, a named entity recognition module, and a sentiment analysis module. The execution order, activation status, and internal parameters of each module can be dynamically configured.

[0067] Optionally, the sensitive word filtering module is an Aho-Corasick automaton using Lewinstein fuzzy matching, and step S20 specifically includes:

[0068] Construct an N-gram index that runs parallel to the Aho-Corasick automaton using Lewinstein fuzzy matching, to record the mapping relationship between each N-gram fragment in the sensitive word library and its corresponding sensitive word;

[0069] By querying the N-gram index, the fragment similarity between the query information and each sensitive word can be quickly calculated.

[0070] Based on fragment similarity, a subset of candidate sensitive words is selected from the sensitive word database. The subset of candidate sensitive words includes all sensitive words that have potential fuzzy matching with the query information.

[0071] For a subset of candidate sensitive words, a fuzzy search is performed on the Aho-Corasick automaton using Lewinstein fuzzy matching.

[0072] To enable those skilled in the art to clearly understand the specific working process of the sensitive word filtering module, a detailed explanation is provided below using specific examples.

[0073] (1) Traverse the sensitive word database, generate a 2-gram fragment for each sensitive word, and record which words the fragment belongs to, thus obtaining an N-gram index. Establish a fragment index for each word in the sensitive word database to achieve fast preliminary screening.

[0074] The generated N-gram index data is as follows:

[0075] python

[0076] N_gram_Index = {

[0077] "gambling": ["赌博"],

[0078] "gambling with $": ["赌$博"],

[0079] "violence": ["暴力"],

[0080] "drugs": ["毒品"]

[0081] }

[0082] (2) Divide the query text Query_Text_2grams into 2-gram segments, and then quickly compare them with the 2-gram segments in the N_gram index. This method can quickly exclude the vast majority of impossible-to-match sensitive words and narrow the scope of subsequent exact matching.

[0083] Extract the 2-gram segments of the query text Query_Text_2grams: Query_Text_2grams = ["我们", "们一", "一起", "起来", "来赌", "赌$", "$博", "博吧"]

[0084] Quick comparison reveals that the "来赌" in the query text Query_Text_2grams shares the core segment "赌" with the sensitive word "赌博", and at the same time, the edit distance between "赌$" and "$博" and "赌博" is within an acceptable range. Similarly, the segments of "暴力" and "毒品" do not overlap with the query text.

[0085] Generate the candidate sensitive word subset: Candidate_Subset = ["赌博"]. Through N-gram pre-filtering, the system quickly filters out 1 most likely matching candidate word from 3 sensitive words, greatly reducing the subsequent computational workload.

[0086] (3) Input the candidate sensitive word subset Candidate_Subset and the query text Query_Text into the Aho-Corasick automaton using Levenshtein fuzzy matching. The Aho-Corasick automaton using Levenshtein fuzzy matching has been configured with a maximum edit distance Max_Edits=2. The Aho-Corasick automaton using Levenshtein fuzzy matching scans for "gambling" in the query text Query_Text and calculates its Levenshtein distance to the variant "gambling$". Changing "gambling" to "gambling$" requires 1 operation (inserting the character $), and the Levenshtein distance Levenshtein_Distance=1. Since the Levenshtein distance Levenshtein_Distance≤Max_Edits=2, the match is successful, and the final output is: matched sensitive word: "gambling", position in the text: starting from the 6th character and ending at the 8th character ("gambling$"), edit distance: 1.

[0087] Optionally, the named entity recognition module adopts a recognition mechanism that integrates regular rules, a dictionary, and a machine learning model. In this case, step S20 specifically includes:

[0088] Based on the preset regular expression, identify the first entity in the query information;

[0089] The system queries a pre-defined dictionary to identify the second entity in the query information.

[0090] The pre-defined BERT-based NER machine learning model is used to understand the context, identify the third entity in the query information, and verify the first and second entities. The verified first, second and third entities are then classified and integrated to obtain a set of associated entities.

[0091] The system uses preset keywords to replace entities in the query information that are the same as entities in the associated entity set, thus obtaining filtered query information.

[0092] The Named Entity Recognition (NER) module employs a recognition mechanism that integrates regular expressions, a dictionary, and a machine learning model. Specifically: First, it identifies the first entity in the query information, such as names, phone numbers, and email addresses, using a pre-defined regular expression. Second, since regular expressions struggle to accurately identify information like place names, a pre-defined dictionary is used for supplementary recognition, identifying the second entity. Finally, because the pre-defined dictionary has limited ability to recognize newly added or combined entities, a pre-defined BERT-based NER machine learning model is invoked to understand the context and perform deep analysis, identifying the third entity. This model is a BERT-based NER model from existing technology, which supports the recognition of entities such as names, addresses, and organizations.

[0093] The named entity recognition module also verifies the first and second entities, and classifies and integrates the verified first, second, and third entities to obtain a set of associated entities. Specifically, for the first and second entities, the named entity recognition module verifies their rationality within the context; if rational, the verification passes; otherwise, the first or second entity is deleted.

[0094] This filter replaces entities in the query information that are identical to entities in the associated entity set using preset keywords. For example, a person's name is replaced with [Name], a mobile phone number with [Phone Number], an email address with [Email Address], and an address with [Address]. The filtered query information becomes: "My name is [Name], my mobile phone number is [Phone Number], I live at [Address], and my email address is [Email Address]".

[0095] Optionally, the sentiment analysis module is a Roberta-based pre-trained language model, and step S20 specifically includes:

[0096] The sentiment analysis module is used to perform sentiment analysis on the query information and output the sentiment score and content type.

[0097] If the sentiment score is less than or equal to the first preset threshold or the content type is inappropriate, blocking will be triggered.

[0098] If the sentiment score is greater than the first preset threshold and less than or equal to the second preset threshold, the filtered query information will be submitted to the large language model.

[0099] If the sentiment score is greater than the second preset threshold, the filtered query information will be submitted to the large language model.

[0100] In this invention, the Roberta-base pre-trained language model from the prior art can be used to process query information. The Roberta-base pre-trained language model was fine-tuned on 50,000 labeled data points, which included normal, negative sentiment, and inappropriate content. The test dataset contained 10,000 real query messages, with sentiment scores set to the range [0,1], where 0 represents extreme negative and 1 represents extreme positive. The first preset threshold was set to 0.3, and the second preset threshold was set to 0.7. Specifically, if the sentiment score is ≤0.3 or the content type is inappropriate (such as violence, illegality, etc.), interception is triggered; if 0.3 < sentiment score ≤0.7, the filtered query is submitted to the large language model; if the sentiment score >0.7, the filtered query information is submitted to the large language model.

[0101] Step S30: If the multi-layer filtering module does not trigger interception when inputting query information, the filtered query information is submitted to the large language model for inference;

[0102] Optionally, after the multi-layer filtering module for query information input in step S30 fails to trigger interception, the process further includes:

[0103] Obtain the user's historical context determination information, which includes the types of sensitive words that have been triggered, the entities that have been mentioned, the sentiment tendency, and the risk level.

[0104] Based on historical context information, a comprehensive security assessment is performed on the current query information to obtain the current context information.

[0105] If the current context information is greater than or equal to the preset risk threshold, execute the preset intervention operation.

[0106] For example, if user_A has already had two rounds of dialogue, the comprehensive judgment module records and generates the following historical context judgment information: History_Context_Info:

[0107] Python

[0108] History_Context_Info={

[0109] "user_id":"User_A",

[0110] "Triggered Sensitive Word Types": ["Violence", "Weapons"],

[0111] "Entities already mentioned":[],

[0112] "Feeling tendency": "Negative"

[0113] Risk Level: 0.6

[0114] }

[0115] User_A inputs a new query text, namely the current query text Current_Query: Current_Query = "Tell me how to get explosives, I hate that person and want to teach him a lesson." The multi-layer filtering module analyzes the current query text Current_Query and obtains the following preliminary analysis results:

[0116] Sensitive word: "explosives"; Sensitive word type: "weapons / dangerous goods"; Entity: No specific name or address identified. Sentiment: Analyzed as "extremely negative," with clear malicious intent.

[0117] The comprehensive judgment module associates and performs comprehensive calculations with the preliminary analysis results of the current query text Current_Query and the historical context judgment information History_Context_Info to generate the current context judgment information Current_Context_Info.

[0118] Comprehensive judgment logic:

[0119] Sensitive word type upgrade: While "violence" and "weapons" were previously mentioned, the current search for "explosives" indicates an escalation of the risk intent.

[0120] Deteriorating emotional tendencies: A "negative" history and an "extremely negative" present situation with clear aggressive intent significantly increase emotional risk;

[0121] Risk accumulation effect: Combining the historical cumulative risk of 0.6, current high-risk behavior directly triggers a significant increase in risk.

[0122] The comprehensive judgment module performs weighted calculations using a preset algorithm, resulting in: Current_Context_Info = History_Context_Info[“Risk Level”] × Historical Weight + (Current Query Risk Value) × Current Weight + (Risk Upgrade Penalty Score) = 0.6 × 0.4 + 0.85 × 0.6 + 0.1 = 0.85.

[0123] Regarding historical and current weights, those skilled in the art can flexibly set them according to actual needs, and no limitations are imposed here. The current query risk value can be the sentiment score, or it can be the sum of the scores of each dimension multiplied by preset dimension weights, and no limitations are imposed here.

[0124] The comprehensive judgment module compares the current context judgment information Current_Context_Info with the preset risk threshold Preset_Risk_Threshold: if Current_Context_Info(0.85)>=Preset_Risk_Threshold(0.8), the condition is met, and the preset intervention operation is executed, such as stopping the output of filtered query text to the large language model.

[0125] Step S40: The large language model continuously outputs streaming text during the inference process and accumulates the continuously outputting streaming text into the buffer.

[0126] Optionally, before accumulating the continuously outputting text into the buffer in step S40, the method further includes:

[0127] During the reasoning process of generating each streaming output text in the large language model, the real-time activation values ​​of the preset key neuron set in the internal neural network of the large language model are captured in real time; the key neuron set is a set of neurons that are determined by pre-analysis and have high responsiveness to predefined sensitive concept categories;

[0128] Based on the real-time activation values ​​of key neuron sets, calculate the instantaneous risk intensity of the current generation step for each sensitive concept category;

[0129] The instantaneous risk intensity of each sensitive concept category is weighted and accumulated to generate a comprehensive risk trajectory value;

[0130] When the comprehensive risk trajectory value is greater than or equal to the preset intervention threshold, a real-time intervention operation is triggered.

[0131] The key neuron set can be determined using existing differential activation analysis methods. Of course, those skilled in the art can flexibly choose other methods according to actual needs, and this is not limited here. It should be noted that after determining the key neuron set, real-time activation values ​​of the key neuron set can be obtained by embedding software hooks, such as forward propagation hook functions, into the large language model code.

[0132] Among them, real-time intervention can immediately terminate the current inference process of the large language model.

[0133] Sensitive concept categories are violence and illegality; key neuron set: for violence, and Regarding illegal acts, and Intervention threshold: threshold_intervention=2.0 for example.

[0134] (1) Real-time capture of the activation values ​​of a pre-defined set of key neurons in the internal neural network of a large language model, as follows:

[0135] At each generation step t (i.e., at each streaming output text generation), the real-time activation values ​​of the captured neurons are respectively for brute force, , Regarding illegal acts, , .

[0136] (2) Based on the real-time activation values ​​of the key neuron set, calculate the instantaneous risk intensity of the current generation step for each sensitive concept category, as follows:

[0137] For each sensitive concept category, the real-time activation values ​​of the current set of key neurons are input into a pre-trained logistic regression probe model to obtain the instantaneous risk intensity: , ,in, The instantaneous risk intensity of the violence category. For the logistic regression probe model The weights set for key neurons, The results obtained from the pre-training of the logistic regression probe model The weights set for key neurons, The first bias is obtained by pre-training the logistic regression probe model. The results obtained from the pre-training of the logistic regression probe model The weights set for key neurons, For the logistic regression probe model The weights set for key neurons, This is the second bias obtained by pre-training the logistic regression probe model.

[0138] (3) The instantaneous risk intensity of each sensitive concept category is weighted and accumulated to generate a comprehensive risk trajectory value, as follows:

[0139] Based on the pre-set weights: weight_V=0.6 (for violence), weight_I=0.4 (for illegality), calculate the risk contribution value for the current step: current_risk_t=weight_V× +weight_I× ;

[0140] Based on the risk contribution value of each generation step, calculate the comprehensive risk trajectory value: cumulative_risk_t = cumulative_risk_{t-1} + current_risk_t.

[0141] (4) When the cumulative risk trajectory value is greater than or equal to the preset intervention threshold, a real-time intervention operation is triggered, as follows: If the cumulative risk trajectory value cumulative_risk_t is greater than or equal to the preset intervention threshold threshold_intervention=2.0, the current inference process of the large language model is immediately terminated.

[0142] It should be noted that the weights involved in this invention have a value range of [0,1]. Those skilled in the art can select within this value range according to actual needs, and no limitation is made here.

[0143] Step S50: When the accumulated text in the cache meets the preset triggering conditions, perform sensitive content checks on the accumulated text in the cache.

[0144] The preset trigger condition can be that the length of the accumulated text is greater than or equal to the text length threshold, and the text length threshold can be twice the longest sensitive word. Those skilled in the art can set it flexibly according to actual needs, and there is no limitation here.

[0145] Step S60: If sensitive content is detected in the accumulated text, terminate the inference process of the large language model;

[0146] Step S70: If no sensitive content is detected in the accumulated text, output the complete answer text after the reasoning process is completed.

[0147] Optionally, after step S70 where no sensitive content is detected in the accumulated text, the process further includes:

[0148] Detect whether the accumulated text contains content of a preset type that needs to be modified;

[0149] If the accumulated text contains content of a preset type that needs to be modified, the current accumulated text is modified in a safe manner while preserving semantics, resulting in the modified accumulated text.

[0150] The preset types of content to be modified can be flexibly set by those skilled in the art according to actual needs, without limitation here. For example, newly added popular internet slang. By flexibly configuring the types of content to be modified, the system can have dynamic adaptability, quickly respond to emerging risks such as newly added vulgar language, and achieve continuous and effective content security governance.

[0151] The large-scale interactive information security filtering method of this invention firstly achieves efficient and accurate identification of mutated sensitive words by employing the Aho-Corasick algorithm integrating Levinstein fuzzy matching and an N-gram pre-filtering mechanism. Even when processing a sensitive word database of 50,000 words, it maintains a millisecond-level response (approximately 0.012 seconds in actual testing), solving the problem of missed detection of spelling errors and deliberate avoidance behaviors in traditional methods. Secondly, it constructs a dynamic defense system covering the entire process: "input preprocessing - real-time monitoring during inference - pre-output truncation." Through dynamically configurable multi-layered filtering modules (sensitive word filtering module, named entity recognition module, and sentiment analysis module), the input is deeply cleaned. Based on this, continuous content checking is performed on the streaming output text of the large language model, enabling immediate detection and termination of inference during the generation of sensitive content. This blocks risky information before it is fully exposed to the user, achieving real-time intervention and significantly reducing security risks. Thirdly, it innovatively introduces a comprehensive risk assessment mechanism based on historical context. By analyzing the evolution path of sensitive word types, entity associations, and sentiment tendencies, the system can intelligently identify potential escalation of risky behaviors. Finally, the modular design and dynamically configurable parameters of the entire solution enable it to flexibly adapt to different application scenarios, ranging from rapid response to strict control, greatly improving the system's practicality and scalability. Furthermore, it achieves synergistic breakthroughs in accuracy, efficiency, and real-time performance, providing key technical support for building a secure and reliable large-scale model interactive environment.

[0152] Based on the same inventive concept, embodiments of the present invention also provide a large model interaction information security filtering device for implementing the large model interaction information security filtering method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the large model interaction information security filtering device provided below can be found in the limitations of the large model interaction information security filtering method described above, and will not be repeated here.

[0153] like Figure 2a As shown, the present invention provides a large-scale interactive information security filtering device, comprising: an acquisition module 20, a multi-layer filtering module 21, and a large language model 22; wherein,

[0154] The acquisition module 20 is used to acquire the query information input by the user;

[0155] The multi-layer filtering module 21, connected to the acquisition module 20, is used to filter query information. The multi-layer filtering module 21 includes at least a sensitive word filtering module, a named entity recognition module, and a sentiment analysis module. The execution order, activation status, and internal parameters of each module can be dynamically configured. If the query information input to the multi-layer filtering module 21 does not trigger interception, the filtered query text is submitted to the large language model for inference.

[0156] The large language model 22, connected to the multi-layer filtering module 21, is used to continuously output streaming text during the inference process and accumulate the continuously output streaming text into a buffer. When the accumulated text in the buffer meets the preset trigger conditions, a sensitive content check is performed on the accumulated text in the buffer. If sensitive content is detected in the accumulated text, the inference process of the large language model is terminated. If no sensitive content is detected in the accumulated text, the complete answer text is output after the inference process is completed.

[0157] In one alternative implementation, such as Figure 2b As shown, the device also includes: a lightweight risk prediction model 23, which is connected to the acquisition module 20 and the multi-layer filtering module 21 respectively, for performing risk assessment on the query information output by the acquisition module 20 and generating risk values; selecting a target strategy from multiple predefined filtering strategies based on the risk values, and outputting the target strategy to the multi-layer filtering module 21, wherein the multiple predefined filtering strategies include fast channels, standard channels, and strict channels and the channel thresholds corresponding to each channel, the execution order, activation status, and internal parameters of each module in the multi-layer filtering module 21; and adjusting the execution order, activation status, and / or internal parameters of each module in the multi-layer filtering module 21 according to the target strategy.

[0158] The lightweight risk prediction model can be a text convolutional neural network model or a small Transformer model.

[0159] Optionally, the large language model 22 is also used to: detect whether there is content to be modified of a preset type in the accumulated text after no sensitive content is detected in the accumulated text; if there is content to be modified of a preset type in the accumulated text, perform semantically preserved safe modification on the current accumulated text to obtain the modified accumulated text.

[0160] Optionally, the device includes a comprehensive judgment module (not shown in the figure), which is connected to the multi-layer filtering module and the large language model respectively. It is used to obtain the user's historical context judgment information after no interception is triggered. The historical context judgment information includes the types of sensitive words that have been triggered, the entities mentioned, the sentiment tendency, and the risk level. Based on the historical context judgment information, a comprehensive security judgment is made on the current query information to obtain the current context judgment information. If the current context judgment information is greater than or equal to a preset risk threshold, a preset intervention operation is executed.

[0161] Optionally, the sensitive word filtering module is an Aho-Corasick automaton using Lewinstein fuzzy matching. Specifically, it is used to: construct an N-gram index that runs in parallel with the Aho-Corasick automaton using Lewinstein fuzzy matching, to record the mapping relationship between each N-gram segment in the sensitive word library and its corresponding sensitive word; quickly calculate the segment similarity between the query information and each sensitive word by querying the N-gram index; based on the segment similarity, select a candidate sensitive word subset from the full sensitive word library, wherein the candidate sensitive word subset includes all sensitive words that have potential fuzzy matching possibilities with the query information; and perform a fuzzy search on the candidate sensitive word subset using the Aho-Corasick automaton using Lewinstein fuzzy matching.

[0162] Optionally, the named entity recognition module employs a recognition mechanism that integrates regular expressions, a dictionary, and a machine learning model. Specifically, it is used to: identify the first entity in the query information based on a preset regular expression; query a preset dictionary to identify the second entity in the query information; use a preset BERT-based NER machine learning model to understand the context and identify the third entity in the query information, and verify the identified first and second entities. The verified first, second, and third entities are then classified and associated to obtain a set of associated entities; and preset keywords are used to replace entities in the query information that are identical to entities in the set of associated entities, resulting in filtered query information.

[0163] Optionally, the sentiment analysis module is a Roberta-based pre-trained language model, specifically used for: performing sentiment analysis on the query information and outputting sentiment scores and content types; triggering blocking if the sentiment score is less than or equal to a first preset threshold or the content type is inappropriate; submitting the filtered query information to the large language model if the sentiment score is greater than the first preset threshold and less than or equal to a second preset threshold; and submitting the filtered query information to the large language model if the sentiment score is greater than the second preset threshold.

[0164] Optionally, before accumulating the continuously outputting streaming text into the buffer, the large language model 22 is further configured to: capture the real-time activation values ​​of a preset set of key neurons in the internal neural network of the large language model during the inference process of generating each streaming output text; wherein the set of key neurons is a set of neurons that are determined through pre-analysis and have high responsiveness to predefined sensitive concept categories; calculate the instantaneous risk intensity of each sensitive concept category in the current generation step based on the real-time activation values ​​of the key neuron set; perform weighted accumulation of the instantaneous risk intensity of each sensitive concept category to generate a comprehensive risk trajectory value; and trigger a real-time intervention operation when the comprehensive risk trajectory value is greater than or equal to a preset intervention threshold.

[0165] The large-scale interactive information security filtering device of this invention firstly achieves efficient and accurate identification of mutated sensitive words by employing the Aho-Corasick algorithm integrating Levinstein fuzzy matching and an N-gram pre-filtering mechanism. Even when processing a sensitive word database of 50,000 words, it maintains a millisecond-level response (approximately 0.012 seconds in actual testing), solving the problem of missed detection of spelling errors and deliberate avoidance behaviors in traditional methods. Secondly, it constructs a dynamic defense system covering the entire process: "input preprocessing - real-time monitoring during inference - pre-output truncation." Through dynamically configurable multi-layered filtering modules (sensitive word filtering module, named entity recognition module, and sentiment analysis module), the input is deeply cleaned. Based on this, continuous content checks are performed on the streaming output text of the large language model, enabling immediate detection and termination of inference during the generation of sensitive content. This blocks risk information before it is fully exposed to the user, achieving real-time intervention and significantly reducing security risks. Thirdly, it innovatively introduces a comprehensive risk assessment mechanism based on historical context. By analyzing the evolution paths of sensitive word types, entity associations, and sentiment tendencies, the system can intelligently identify potential escalation of risky behaviors. Finally, the modular design and dynamically configurable parameters of the entire solution enable it to flexibly adapt to different application scenarios, ranging from rapid response to strict control, greatly improving the system's practicality and scalability. Furthermore, it achieves synergistic breakthroughs in accuracy, efficiency, and real-time performance, providing key technical support for building a secure and reliable large-scale model interactive environment.

[0166] It should be noted that "multiple" in this invention includes two or more.

[0167] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0168] Each module in the devices of this invention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0169] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for or generated by the large-scale interactive information security filtering method described above. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a large-scale interactive information security filtering method.

[0170] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a large-scale interactive information security filtering method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0171] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0172] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0173] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0175] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0176] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A large model interaction information security filtering method, characterized in that, The method comprises: receiving query information input by a user; inputting the query information into a configurable multi-layer filtering module for filtering processing, wherein the multi-layer filtering module at least comprises a sensitive word filtering module, a named entity recognition module and an emotion analysis module, and the execution order, enabled state and internal parameters of each module can be dynamically configured; if the query information input into the multi-layer filtering module does not trigger interception, submitting the filtered query information to a large language model for reasoning; the large language model continuously outputs streaming output text in the reasoning process, and accumulates the continuously output streaming output text in a cache area; when the accumulated text in the cache area meets a preset triggering condition, performing sensitive content checking on the accumulated text in the cache area; if sensitive content is detected in the accumulated text, terminating the reasoning process of the large language model; if no sensitive content is detected in the accumulated text, outputting a complete answer text after the reasoning process is completed; after the query information input into the multi-layer filtering module does not trigger interception, further comprising: obtaining historical context judgment information of the user, wherein the historical context judgment information includes triggered sensitive word types, mentioned entities, sentiment orientations and risk levels; based on the historical context judgment information, performing comprehensive safety judgment on the current query information to obtain current context judgment information; if the current context judgment information is greater than or equal to a preset risk threshold, performing a preset intervention operation.

2. The method of claim 1, wherein, after receiving the query information input by the user, further comprising: inputting the query information into a lightweight risk prediction model to perform risk assessment on the query information input by the user, and generating a risk value; according to the risk value, selecting a target strategy from a plurality of pre-defined filtering strategies, and outputting the target strategy to the multi-layer filtering module, wherein the plurality of pre-defined filtering strategies include a fast channel, a standard channel and a strict channel, and the execution order, enabled state and internal parameters of each module in the multi-layer filtering module; according to the target strategy, adjusting the execution order, enabled state and / or internal parameters of each module in the multi-layer filtering module.

3. The method of claim 1, wherein, after no sensitive content is detected in the accumulated text, further comprising: detecting whether the accumulated text contains preset types of content to be modified; if the accumulated text contains preset types of content to be modified, performing semantic-preserving security modification on the current accumulated text to obtain modified accumulated text.

4. The method of claim 1, wherein, The sensitive word filtering module is an Aho-Corasick automaton using Levenshtein fuzzy matching, and the filtering processing of inputting the query information into the configurable multi-layer filtering module comprises: building an N-gram index parallel to the Aho-Corasick automaton using Levenshtein fuzzy matching, for recording the mapping relationship between each N-gram segment in the sensitive word library and the sensitive word it belongs to; quickly calculating the segment similarity between the query information and each sensitive word by querying the N-gram index; based on the segment similarity, screening a candidate sensitive word subset from the sensitive word library, wherein the candidate sensitive word subset includes all sensitive words that have potential fuzzy matching possibilities with the query information; For the candidate sensitive word subset, fuzzy search is performed on the Aho-Corasick automaton using Levenshtein fuzzy matching.

5. The method of claim 1, wherein, The named entity recognition module adopts a recognition mechanism that combines regular rules, a dictionary library, and a machine learning model. According to a preset regular expression, a first entity in the query information is identified. A preset dictionary library is queried to identify a second entity in the query information. A preset BERT-based NER machine learning model is used to understand the context and identify a third entity in the query information, and the identified first and second entities are verified. The first, second, and third entities that pass the verification are classified and integrated to obtain a set of associated entities.

6. The method of claim 1, wherein, The same entities in the query information as those in the set of associated entities are replaced using a preset keyword to obtain filtered query information. The sentiment analysis module is a roberta-base pre-trained language model. The sentiment analysis module is used to perform sentiment analysis on the query information, and the sentiment score and content type are output. If the sentiment score is less than or equal to a first preset threshold or the content type is undesirable content, the interception is triggered.

7. The method of claim 1, wherein, If the sentiment score is greater than the first preset threshold and less than or equal to a second preset threshold, or the sentiment score is greater than the second preset threshold, the filtered query information is submitted to the large language model. Before the continuously output stream output text is accumulated to the cache area, it further includes: In the inference process of each stream output text generated by the large language model, the real-time activation values of a preset key neuron set in the internal neural network of the large language model are captured in real time; wherein the key neuron set is determined by pre-analysis and has high response to predefined sensitive concept categories. Based on the real-time activation values of the key neuron set, the instantaneous risk intensity of each sensitive concept category for the current generation step is calculated. The instantaneous risk intensity of each sensitive concept category is weighted and accumulated to generate a comprehensive risk trajectory value.

8. The method of claim 2, wherein, When the comprehensive risk trajectory value is greater than or equal to a preset intervention threshold, the real-time intervention operation is triggered.

9. A large model interaction information security filtering device, characterized in that, The lightweight risk prediction model is a text convolutional neural network model or a small Transformer model. It includes: An acquisition module for acquiring user input query information; A multi-layer filtering module connected to the acquisition module for filtering the query information, wherein the multi-layer filtering module includes at least a sensitive word filtering module, a named entity recognition module, and a sentiment analysis module, and the execution order, enabled state, and internal parameters of each module can be dynamically configured; if the query information input into the multi-layer filtering module does not trigger interception, the filtered query information is submitted to the large language model for inference. The large language model is connected with the multi-layer filtering module, and is used for continuously outputting streaming output text in the reasoning process and accumulating the continuously output streaming output text to a cache area; when accumulated text in the cache area meets a preset triggering condition, sensitive content checking is performed on the accumulated text in the cache area; if sensitive content is detected in the accumulated text, the reasoning process of the large language model is terminated; if no sensitive content is detected in the accumulated text, complete answer text is output after the reasoning process is completed; The comprehensive judgment module is connected with the multi-layer filtering module and the large language model, respectively, and is used for obtaining historical context judgment information of the user after interception is not triggered, wherein the historical context judgment information includes a sensitive word type that has been triggered, an entity that has been mentioned, a sentiment tendency, and a risk level; based on the historical context judgment information, comprehensive security judgment is performed on current query information to obtain current context judgment information; if the current context judgment information is greater than or equal to a preset risk threshold, a preset intervention operation is performed.

Images