Information processing method and system and computer storage medium

Abstract

The invention provides an information processing method and system and a computer storage medium. The information processing method comprises the steps that a preprocessing module preprocesses user input information and inputs obtained target input information into a large language model; the large language model processes target input information, generates lexical elements output in a streaming mode and transmits the lexical elements to the output detection module; the output detection module performs risk assessment on the currently received lexical elements, and determines target output information corresponding to the currently detected lexical elements according to an assessment result; the closed-loop learning module obtains the first processing log of the preprocessing module and the output detection module, and determines the parameter modification strategy of the preprocessing module and the output detection module according to the first processing log, and through the information processing method, the security and compliance of the dialogue system based on the large language model in the dialogue process are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an information processing method, system, and computer storage medium. Background Technology

[0002] With the rapid evolution of artificial intelligence technology, dialogue systems based on large language models have become the core application form in the field of natural language processing. They are widely used in many fields such as intelligent customer service, information consultation, education and healthcare, significantly improving the naturalness and efficiency of human-computer interaction.

[0003] However, in practical applications, at the user input level, users may submit queries or malicious prompts containing sensitive, harmful, or inappropriate content to the system. Since large language models are essentially probabilistic generative models based on training data and lack the ability to make subjective value judgments, the models may still generate responses that do not comply with security mechanisms. For example, they may be maliciously guided by seemingly neutral user input to bypass security mechanisms and generate harmful content.

[0004] Therefore, improving the security and compliance of dialogue systems based on large language models during the dialogue process is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this application proposes an information processing method, a system, and a computer storage medium.

[0006] To address the aforementioned technical problems, this application proposes an information processing method applied to an information processing system, which includes: a preprocessing module, a large language model, an output detection module, and a closed-loop learning module.

[0007] The information processing method includes: the preprocessing module preprocesses the user input information and inputs the obtained target input information into the large language model; the large language model processes the target input information, generates streaming output lexical units, and transmits them to the output detection module; the output detection module performs risk assessment on the currently received lexical units and determines the target output information corresponding to the currently detected lexical units based on the assessment results; the closed-loop learning module obtains the first processing logs of the preprocessing module and the output detection module, and determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing logs.

[0008] The information processing system further includes a configuration management module, and the closed-loop learning module and the configuration management module interact with each other through a bidirectional communication channel. After the closed-loop learning module determines the parameter modification strategies of the preprocessing module and the output detection module based on the first processing log, the method further includes: The closed-loop learning module sends the parameter modification strategy to the configuration management module; The configuration management module performs policy versioning management based on the parameter modification policy, and the configuration management module releases the parameter modification policy to the preprocessing module and the output detection module in a canary manner through the configuration center or message channel.

[0009] The method further includes, after the configuration management module releases the parameter modification strategy to the preprocessing module and the output detection module in a canary manner, the following steps: the configuration management module obtains a second processing log during the operation of the preprocessing module and the output detection module after the parameters are modified; the configuration management module determines the processing efficiency of the preprocessing module and the output detection module based on the second processing log, the processing efficiency including at least one of the following: analyzing the changing trends of interception rate, false positive rate, user experience impact, number of evasion attacks, and compliance indicators; if the processing efficiency does not meet preset conditions, the configuration management module controls the parameters of the preprocessing module and the output detection module to roll back.

[0010] The closed-loop learning module includes: a log access and risk clustering submodule, a strategy automatic optimization submodule, and a strategy effectiveness evaluation submodule. The closed-loop learning module determines the parameter modification strategy for the preprocessing module and the output detection module based on the first processing log, including: the log access and risk clustering submodule receiving the first processing log and processing it using semantic clustering and risk pattern mining algorithms to obtain risk clustering analysis results; the strategy automatic optimization submodule generating the parameter modification strategy based on the risk clustering analysis results; and the strategy effectiveness evaluation submodule quantitatively evaluating the processing efficiency of the preprocessing module and the output detection module after applying the parameter modification strategy to guide the optimization of the next round of parameter modification strategies. The processing efficiency includes at least one of the following: analyzing the changing trends of interception rate, false positive rate, user experience impact, number of evaded attacks, and compliance indicators.

[0011] The closed-loop learning module determines the parameter modification strategy for the preprocessing module and the output detection module based on the first processing log, and further includes: analyzing the risk symbiotic relationship between the user input information and the output information of the large language model based on the first processing log; and determining the parameter modification strategy based on the risk symbiotic relationship.

[0012] The step of analyzing the risk symbiotic relationship between the user input information and the output information of the large language model based on the first processing log includes: obtaining dialogue items with risks in the first processing log, wherein the dialogue items are divided into dialogue input information and dialogue output information; organizing each dialogue item into one or more risk item combinations; filtering out risk item combinations whose frequency of occurrence is greater than a preset threshold among all risk item combinations; and determining the risk symbiotic relationship based on the filtered risk item combinations.

[0013] The output detection module performs a risk assessment on the currently received lexical unit and determines the target output information corresponding to the currently detected lexical unit based on the assessment result. This includes: the output detection module obtaining a risk score for the currently detected lexical unit in the assessment result; when the risk score is greater than or equal to a first threshold, the output detection module controls the large language model to stop generating lexical units and determines preset information as the target output information; when the risk score is less than or equal to a second threshold, the decoded currently detected lexical unit is determined as the target output information; when the risk score is greater than the second threshold and less than the first threshold, the output detection module obtains a target detected lexical unit whose reception time is after the currently detected lexical unit; and determines the target output information based on the assessment result of the target detected lexical unit.

[0014] The output detection module includes a real-time protection module and an output filtering module. The output detection module performs a risk assessment on the currently received lexical units and determines the target output information corresponding to the currently detected lexical unit based on the assessment results. This includes: the real-time protection module performing content classification, sandbox verification, and compliance verification on the currently received lexical units to obtain classification results, verification results, and validation results; determining the assessment result based on the classification results, verification results, and validation results; the real-time protection module determining the preliminary output information of the currently detected lexical unit based on the assessment results; and the output filtering module performing secondary filtering on the preliminary output information through keyword matching to determine the target output information.

[0015] To address the aforementioned technical problems, this application also proposes an information processing system, comprising: a preprocessing module, a large language model, an output detection module, and a closed-loop learning module; wherein, the preprocessing module is used to preprocess user input information and input the obtained target input information into the large language model; the large language model is used to process the target input information, generate streaming output lexical units, and transmit them to the output detection module; the output detection module is used to perform risk assessment on the currently received lexical units and determine the target output information corresponding to the currently detected lexical units based on the assessment results; the closed-loop learning module is used to acquire a first processing log of the preprocessing module and the output detection module, and determine the parameter modification strategy of the preprocessing module and the output detection module based on the first processing log.

[0016] To address the aforementioned technical problems, this application also proposes a computer storage medium for storing program data, which, when executed by a computer, is used to implement the aforementioned information processing method.

[0017] Compared with existing technologies, the beneficial effects of this application are as follows: Through the above embodiments, the preprocessing module preprocesses the user input information and inputs the obtained target input information into the large language model; the large language model processes the target input information, generates streaming output lexical units, and transmits them to the output detection module; the output detection module performs risk assessment on the currently received lexical units and determines the target output information corresponding to the currently detected lexical units based on the assessment results; security detection and processing are performed on the input and output of the large language model, improving the security and compliance of the dialogue system based on the large language model during the dialogue process. In addition, the closed-loop learning module obtains the first processing logs of the preprocessing module and the output detection module, determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing logs, and continuously updates the parameters, enabling the system to have the ability to self-evolve and perpetually protect itself. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the framework of an information processing system provided in one embodiment of this application; Figure 2 This is a flowchart illustrating an embodiment of the information processing method provided in this application; Figure 3This is a schematic diagram of the framework of the information processing system in another embodiment provided in this application; Figure 4 This is a flowchart illustrating the parameter modification process provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the information processing device provided in this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] The information processing method of this application is applied to an information processing apparatus, wherein the information processing apparatus can be a server, a terminal device, or a system in which the server and the terminal device cooperate with each other. Accordingly, the various parts of the information processing apparatus, such as various units, sub-units, modules, and sub-modules, can all be set in the server, all in the terminal device, or separately in the server and the terminal device.

[0022] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed server functionality, or as a single software program or software module; no specific limitations are made here.

[0023] Please refer to details. Figure 1 , Figure 1 This is a schematic diagram of the framework of an information processing system provided in one embodiment of this application, as shown below. Figure 1 As shown, the information processing system includes a preprocessing module, a large language model, an output detection module, and a closed-loop learning module.

[0024] The information processing system provided in this application embodiment is a human-computer dialogue system based on a large language model. The user inputs a question (i.e., user input information), and the large language model predicts and generates the most likely following sequence as output through internal calculations, and displays the output of the large language model to the user.

[0025] In addition to the large language model, the information processing system also includes a preprocessing module, an output detection module, and a closed-loop learning module. For details on the information processing methods based on the information processing system, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the information processing method provided in this application, as shown below. Figure 2 As shown, the specific steps are as follows: Step S11: The preprocessing module preprocesses the user input information and inputs the obtained target input information into the large language model.

[0026] In this embodiment, the preprocessing module is a preprocessing module of the large language model. It preprocesses the user's original input information, i.e., the user input information, and inputs the preprocessed target input information into the large language model so that the large language model can process the target input information.

[0027] The user input information includes at least one of the following: multimodal data such as text, voice, and images; the preprocessing module preprocesses the user input information based on a preprocessing strategy, which includes: performing semantic analysis and intent recognition on the user input information to determine the user intent, and identifying potential risks based on the user intent, that is, judging the potential negative impact based on the identified intent. For example, if the user input information is "Please give me the password for a certain account", the identified user intent is to request sensitive information. Since the password is sensitive authentication information, leaking the password may lead to account theft, causing potential risks such as data leakage and privacy violations.

[0028] The preprocessing strategy also includes hierarchical desensitization of user input information. Specifically, it involves determining the sensitivity level of multiple words in the user input information and replacing or blurring words with a sensitivity level greater than a preset sensitivity threshold. For example, if the user input information includes an ID card number, then the ID card number in the user input information is replaced with a preset character, and the replaced user input information becomes the target input information.

[0029] After preprocessing the original user input information, target input information with a certain degree of security is obtained. The target input information is then input into a large language model to generate a response to the user input.

[0030] Step S12: The large language model processes the target input information, generates streaming word units, and transmits them to the output detection module.

[0031] In this embodiment of the application, the large language model receives the preprocessed target input information, predicts and generates the most likely following sequence through internal calculations, and generates an initial output message.

[0032] When generating a response, the large language model does not generate all words and assemble them into a complete output all at once. Instead, it generates and outputs words one by one in a continuous manner, that is, it generates words for streaming output. Each time the large language model generates a word, it transmits the real-time output word to the output detection module, and the initial output message is the sum of all the words in the streaming output.

[0033] Step S13: The output detection module performs a risk assessment on the currently received lexical unit and determines the target output information corresponding to the currently detected lexical unit based on the assessment result.

[0034] In this embodiment, the output detection module is a proxy middleware deployed between the large language model and the user. It is used to receive the word units output by the large language model. The output detection module does not directly send the word units to the user, but manages the received word units. For example, it performs a risk assessment on the currently received word units and determines whether to push the word units to the user based on the assessment results, i.e., whether to intercept the word units.

[0035] The output detection module receives the words from the streaming output of the large language model one after another. The output detection module provides a buffer to cache the received words in chronological order.

[0036] The detection program in the output detection module is triggered to perform a risk assessment on at least a portion of the tokens in the buffer. Notably, the detection program is not triggered after all tokens have been generated and received to perform a risk assessment on the completed output content, but rather it is triggered multiple times during the token generation process. For example, the detection program is triggered to perform a risk assessment on each token generated.

[0037] Preferably, when a preset triggering condition is met, the triggering detection program performs a risk assessment, wherein the preset triggering condition includes at least one of the following: triggering the detection program based on a time window, for example, triggering the detection program to perform a risk assessment every predetermined time period by storing the cached words in the buffer; triggering the detection program based on the number of words, for example, triggering the detection program once after receiving a preset number of words; triggering the detection program based on sentence boundaries, for example, triggering the detection program when a sentence boundary (period, question mark, newline character, etc.) is detected.

[0038] The output detection module integrates multiple detection technologies. After the detection program is triggered, the multiple detection technologies perform detection (i.e., risk assessment) on the current detection term. The detection results of the multiple detection technologies are merged and a risk score and risk label are returned. Based on the risk score and / or risk label, a real-time decision is made, i.e., whether to send the current detection term to the user.

[0039] Multiple detection technologies include: classification models, which quickly categorize the current detected words into multiple labels, such as hate speech, violence, self-harm, pornography, and vulnerability exploitation; regular expression and keyword matching technology, which performs regular expression matching on known specific harmful patterns (such as phone numbers, email addresses, etc.) to classify harmful content; heuristic rule technology, which quickly evaluates based on preset heuristic rules, for example, checking whether a code snippet contains preset dangerous functions or commands; semantic similarity analysis, which calculates the vector similarity between the current detected words and known harmful content templates; execution detection of the current detected words through a sandbox to discover malicious code or vulnerabilities; and compliance verification technology.

[0040] The output detection module performs a risk assessment on the currently received lexical unit and determines the target output information corresponding to the currently detected lexical unit based on the assessment result. This includes: the output detection module obtaining a risk score for the currently detected lexical unit in the assessment result; when the risk score is greater than or equal to a first threshold, the output detection module controls the large language model to stop generating lexical units and determines preset information as the target output information; when the risk score is less than or equal to a second threshold, the decoded currently detected lexical unit is determined as the target output information; when the risk score is greater than the second threshold and less than the first threshold, the output detection module obtains a target detected lexical unit whose reception time is after the currently detected lexical unit; and determines the target output information based on the assessment result of the target detected lexical unit.

[0041] In this embodiment, risk scores are used to classify the risk into high risk, low risk, and medium risk. When the risk score is greater than or equal to the first threshold, the currently detected word is considered high risk. A termination signal is immediately sent to the large language model to control the subsequent generation of words by the large language model. This is the most critical interception action. At this time, the original content (currently detected word) is not returned to the user. Instead, a preset message is returned, such as "The content has been removed by the security filter".

[0042] When the risk score is greater than the second threshold and less than the first threshold, the current detected word is considered medium risk. At this time, we wait for more words to be generated, that is, we receive the target detected word after the current detected word to obtain a more complete context. Based on the evaluation result of the target detected word, we determine the target output information to reduce the false alarm rate.

[0043] When the risk score is less than or equal to the second threshold, the currently detected word is considered low-risk, and the detected word is returned to the user. The target output information is the information ultimately returned to the user.

[0044] The output detection module's calls to the detection program and the reception of data from the large language model are asynchronous, avoiding blocking the reception of word units. The detection program is deployed on a dedicated AI inference chip to accelerate the computation speed.

[0045] The output detection module and the large language model communicate via a high-speed intranet.

[0046] Step S14: The closed-loop learning module obtains the first processing logs of the preprocessing module and the output detection module, and determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing logs.

[0047] In this embodiment of the application, an independent intelligent hub, namely a closed-loop learning module, is established in the information processing system. The closed-loop learning module collects the full-link log of information from input to output, namely the first processing log, which includes the processing log of the preprocessing module and the processing log of the output detection module.

[0048] The closed-loop learning module can use cluster analysis to automatically discover new attack patterns and abnormal behaviors from the first processing log, and then determine the parameter modification strategy; it can also use reinforcement learning to simulate the game process of attack and defense based on the first processing log, and automatically derive the optimal parameter modification strategy.

[0049] The preprocessing module's processing logs and the output detection module's processing logs have a unified log format standard, ensuring that all key information is recorded in a structured manner.

[0050] The preprocessing module's processing log includes: the original user input (i.e., user input information), the processed security input (i.e., target input information), the detected risk type, the actions performed, and the version of the preprocessing strategy currently used by the preprocessing module.

[0051] The logs of the output detection module include: the raw output of the large language model, the risk assessment results, the actions performed, the content finally returned to the user, and the version of the risk assessment strategy currently used by the output detection module.

[0052] All processing logs are transmitted to the closed-loop learning module in real time via a high-throughput message queue. The closed-loop learning module periodically processes the large amount of collected log data, namely the first processing logs. The learning process of the closed-loop learning module may include at least one of the following: full-link correlation analysis, unsupervised learning, reinforcement learning, and supervised learning.

[0053] Optionally, the closed-loop learning module includes: a log access and risk clustering submodule, a strategy automatic optimization submodule, and a strategy effectiveness evaluation submodule; the closed-loop learning module determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing log, including: the log access and risk clustering submodule receives the first processing log, processes the first processing log using semantic clustering and risk pattern mining algorithms to obtain risk clustering analysis results; the strategy automatic optimization submodule generates the parameter modification strategy based on the risk clustering analysis results; the strategy effectiveness evaluation submodule is used to quantitatively evaluate the processing efficiency of the preprocessing module and the output detection module after applying the parameter modification strategy, so as to guide the optimization of the next round of parameter modification strategy, wherein the processing efficiency includes at least one of the following: analyzing the changing trends of interception rate, false positive rate, user experience impact, number of evaded attacks, and compliance indicators.

[0054] In this embodiment, unsupervised learning can be used to obtain parameter modification strategies. Specifically, semantic clustering and risk pattern mining algorithms are used to identify new attack patterns, avoidance strategy prompts, abnormal token trajectories, and high-frequency false alarms in the first processing log to obtain risk clustering analysis results. For example, from the intercepted user input information, unauthorized requests that are not in the rule base corresponding to the original interception strategy but are semantically similar are found. The target attack vector corresponding to the unauthorized request is determined, which is the clustering analysis result. The parameter modification strategy is determined based on the target attack vector. For example, the target attack vector is added to the original rule base of the preprocessing module, and the unauthorized request can be intercepted in the preprocessing module.

[0055] The generated parameter modification strategies include at least one of the following: automatic expansion of the sensitive word library, addition of new violation warning words to the rule library, sandbox rule upgrade, optimization of filtering strategy matching mode, and automatic calibration of risk threshold.

[0056] Optionally, the closed-loop learning module determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing log, and further includes: analyzing the risk symbiotic relationship between the user input information and the output information of the large language model based on the first processing log; and determining the parameter modification strategy based on the risk symbiotic relationship.

[0057] In this embodiment of the application, the parameter modification strategy obtained by unsupervised learning can also learn risk symbiosis relationships based on association rules. For example, the risk symbiosis relationship can be represented as follows: when the user input information includes A and B appears in the output of the large language model, the probability of a violation of type C exceeds the probability threshold.

[0058] The parameter modification strategy is determined based on the risk symbiosis relationship. For example, the newly added risk symbiosis relationship is added to the database of the output detection module, and the corresponding information is quickly determined based on the risk symbiosis relationship.

[0059] The step of analyzing the risk symbiotic relationship between the user input information and the output information of the large language model based on the first processing log includes: obtaining dialogue items with risks in the first processing log, wherein the dialogue items are divided into dialogue input information and dialogue output information; organizing each dialogue item into one or more risk item combinations; filtering out risk item combinations whose frequency of occurrence is greater than a preset threshold among all risk item combinations; and determining the risk symbiotic relationship based on the filtered risk item combinations.

[0060] In this embodiment, based on the dialogue items with risks in the first processing log, a dialogue item includes dialogue input information (e.g., user input information or information output by a large language model) and dialogue output information (e.g., preprocessed information or target output information output by an output detection module). Each dialogue item is organized into one or more risk item combinations. For example, the dialogue input information is divided into multiple input sub-information, and the dialogue output information is divided into multiple output sub-information. A risk item combination includes one input sub-information, one output sub-information, and a risk category. Risk item combinations with a frequency greater than a preset threshold are filtered, and risk symbiosis relationships are determined based on the filtered risk item combinations.

[0061] Specifically, by using a unique identifier, the processing logs of the same user input information in the preprocessing module and the output detection module are linked together to obtain the first processing log, which constitutes a full-link view of input-detection-output. The full-link view is analyzed to determine the parameter modification strategy. For example, if it is found that a certain user input information bypasses the interception of the preprocessing module, but is identified as high-risk and thus intercepted in the output detection module, the interception parameters of the preprocessing module are modified according to the analysis of the user input information, so that the user input information can be intercepted in the preprocessing stage.

[0062] Optionally, the closed-loop learning module is defined as an agent; the parameters of the preprocessing module and the output detection module are defined as actions; the security posture of the information processing system is defined as a state; a reward function is determined; and a reinforcement learning environment is determined based on the agent, actions, state, and establishment function. During the reinforcement learning process, the parameter modification strategy is determined according to the reward corresponding to different actions. The reward function includes positive rewards and negative rewards. Positive rewards are given when the number of effective interceptions increases and the number of false alarms decreases, while negative rewards are given when the number of missed alarms increases and good user requests are mistakenly intercepted.

[0063] The classification model in the output detection module is optimized through supervised learning to obtain multiple processing items that are false positives or false negatives in the first processing log; corresponding labels are added to the multiple processing items, and a training set is constructed based on the multiple processing items and labels; the classification model in the output detection module is trained based on the training set to determine the parameter modification strategy of the classification model.

[0064] Unreported items include, for example, an output that should have been blocked but was ultimately reported by the user, while false alarms include, for example, security content that was incorrectly blocked and was discovered during an audit.

[0065] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an information processing system provided in another embodiment of this application, such as... Figure 3 As shown, in addition to the preprocessing module, large language model, output detection module, and closed-loop learning module, the information processing system also includes a configuration management module. The output detection module includes a real-time protection module and an output filtering module. The closed-loop learning module and the configuration management module interact with each other through a bidirectional communication channel.

[0066] Optionally, after the closed-loop learning module determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing log, the method further includes: the configuration management module performs strategy versioning management based on the parameter modification strategy, and the configuration management module releases the parameter modification strategy to the preprocessing module and the output detection module in a canary manner through a configuration center or message channel.

[0067] In this embodiment, the parameter modification strategy may include updating regular expression rules, keyword lists, publishing newly trained classification models, and threshold adjustments.

[0068] After generating the parameter modification policy, it is distributed. First, the policy is uniformly distributed to the configuration management module, which is responsible for version management, unified distribution, and status monitoring. Then, the policy is canary-released to the preprocessing and output detection modules. Canary-release means that when there are multiple distributed preprocessing and output detection modules, the policy is first sent to a subset of them. After verifying the policy's effectiveness, it is then officially released and applied to all preprocessing and output detection modules. The configuration management module records each distributed policy, generating a unique version identifier for each policy. This supports version rollback, comparison, and switching, ensuring that parameter modifications in the preprocessing and output detection modules are traceable and controllable, guaranteeing system configuration consistency and change security.

[0069] The preprocessing module and output detection module have a hot-load parameter modification strategy that can be applied without restarting.

[0070] After the configuration management module canary releases the parameter modification strategy to the preprocessing module and the output detection module, the method further includes: the configuration management module obtaining a second processing log during the operation of the preprocessing module and the output detection module after the parameters are modified; the configuration management module determining the processing efficiency of the preprocessing module and the output detection module based on the second processing log, the processing efficiency including at least one of the following: analyzing the changing trends of interception rate, false positive rate, user experience impact, number of evasion attacks, and compliance indicators; if the processing efficiency does not meet preset conditions, the configuration management module controlling the parameter rollback of the preprocessing module and the output detection module.

[0071] After the parameter modification strategy is applied to the preprocessing module and the output detection module, a new interception strategy is formed. The new interception strategy is verified, and the second processing logs of the preprocessing module and the output detection module after the application of the new interception strategy are collected. The parameter modification strategy is evaluated based on the second processing logs to determine the processing efficiency of the preprocessing module and the output detection module. Based on the processing efficiency, it is determined whether the parameter modification strategy is effective and whether to use the parameter modification strategy or roll back.

[0072] Processing efficiency includes at least one of the following: analyzing the trend of changes in interception rate, false positive rate, user experience impact, number of evasion attacks, and compliance indicators. Processing efficiency not meeting the preset conditions can be due to an increase in false positive rate, a decrease in interception rate, etc.

[0073] The above embodiments achieve a closed-loop strategy, which can not only deal with known threats, but also adaptively deal with unknown threats by continuously learning from the operation process, realizing the transformation from static protection to dynamic immunity, and from passive response to proactive evolution.

[0074] Please see Figure 4 , Figure 4 This is a flowchart illustrating the parameter modification process provided in this application, such as... Figure 4 As shown, the preprocessing module and the output detection module send processing logs to the closed-loop learning module. The closed-loop learning module determines the parameter modification strategy based on the received processing logs. The closed-loop learning module then sends the parameter modification strategy to the configuration management module. The configuration management module then uniformly releases the parameter modification strategy to the preprocessing module and the output detection module in a canary manner.

[0075] In an optional embodiment, the output detection module performs a risk assessment on the currently received lexical units and determines the target output information corresponding to the currently detected lexical units based on the assessment results. This includes: the real-time protection module performing content classification, sandbox verification, and compliance verification on the currently received lexical units to obtain classification results, verification results, and validation results; determining the assessment results based on the classification results, verification results, and validation results; the real-time protection module determining the preliminary output information of the currently detected lexical units based on the assessment results; and the output filtering module performing secondary filtering on the preliminary output information through keyword matching to determine the target output information.

[0076] In this embodiment, the output detection module is divided into a real-time protection module and an output filtering module. The initial output information is determined based on the evaluation results of the real-time protection module, and the output filtering module uses keyword matching for secondary filtering to determine the final target output information.

[0077] The real-time protection module includes content classification, sandbox verification, and compliance verification. The evaluation result is obtained by combining the detection results of the three (classification results, verification results, and validation results).

[0078] The processing logs of the real-time protection module and the output filtering module are identified as the processing logs of the output detection module.

[0079] The output filtering module also provides an auditing function to audit whether the processing logs of the preprocessing module, real-time protection module and secondary filtering are compliant, for compliance supervision, incident review and model optimization.

[0080] Through the above embodiments, the preprocessing module preprocesses the user input information and inputs the obtained target input information into the large language model; the large language model processes the target input information, generates streaming output lexical units, and transmits them to the output detection module; the output detection module performs risk assessment on the currently received lexical units and determines the target output information corresponding to the currently detected lexical units based on the assessment results; security detection and processing are performed on the input and output of the large language model, improving the security and compliance of the dialogue system based on the large language model during the dialogue process. In addition, the closed-loop learning module obtains the first processing logs of the preprocessing module and the output detection module, determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing logs, and continuously updates the parameters, enabling the system to have the ability to self-evolve and perpetually protect itself.

[0081] To implement the above information processing method, this application also proposes an information processing device, please refer to the details below. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the information processing device provided in this application.

[0082] The information processing device 400 in this embodiment includes a processor 41, a memory 42, an input / output device 43, and a bus 44.

[0083] The processor 41, memory 42, and input / output device 43 are respectively connected to the bus 44. The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the information processing method described in the above embodiments.

[0084] In this embodiment, processor 41 can also be referred to as a CPU (Central Processing Unit). Processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 41 can be any conventional processor.

[0085] This application also provides a computer storage medium; please refer to the following: Figure 6 , Figure 6This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores a computer program 61, which, when executed by a processor, is used to implement the information processing method of the above embodiment.

[0086] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An information processing method, characterized in that, It is applied to an information processing system, which includes: a preprocessing module, a large language model, an output detection module, and a closed-loop learning module; The information processing method includes: The preprocessing module preprocesses the user input information and inputs the obtained target input information into the large language model; The large language model processes the target input information, generates streaming word units, and transmits them to the output detection module. The output detection module performs a risk assessment on the currently received lexical unit and determines the target output information corresponding to the currently detected lexical unit based on the assessment result. The closed-loop learning module obtains the first processing logs of the preprocessing module and the output detection module, and determines the parameter modification strategy of the preprocessing module and the output detection module based on the first processing logs.

2. The information processing method according to claim 1, characterized in that, The information processing system also includes a configuration management module, and the closed-loop learning module and the configuration management module interact with each other through a bidirectional communication channel. After the closed-loop learning module determines the parameter modification strategies of the preprocessing module and the output detection module based on the first processing log, the method further includes: The closed-loop learning module sends the parameter modification strategy to the configuration management module; The configuration management module performs policy versioning management based on the parameter modification policy, and the configuration management module releases the parameter modification policy to the preprocessing module and the output detection module in a canary manner through the configuration center or message channel.

3. The information processing method according to claim 2, characterized in that, After the configuration management module releases the parameter modification strategy to the preprocessing module and the output detection module in a canary manner, The method further includes: The configuration management module obtains the second processing log during the operation of the preprocessing module and the output detection module after the parameters have been modified; The configuration management module determines the processing efficiency of the preprocessing module and the output detection module based on the second processing log. The processing efficiency includes at least one of the following: analyzing the changing trends of interception rate, false alarm rate, user experience impact, number of evasion attacks, and compliance indicators. If the processing efficiency does not meet the preset conditions, the configuration management module controls the parameters of the preprocessing module and the output detection module to roll back.

4. The information processing method according to claim 1, characterized in that, The closed-loop learning module includes: a log access and risk clustering submodule, an automatic strategy optimization submodule, and a strategy effectiveness evaluation submodule; The closed-loop learning module determines the parameter modification strategies for the preprocessing module and the output detection module based on the first processing log, including: The log access and risk clustering submodule receives the first processed log, processes the first processed log using semantic clustering and risk pattern mining algorithms, and obtains risk clustering analysis results. The automatic strategy optimization submodule generates the parameter modification strategy based on the risk clustering analysis results. The strategy effectiveness evaluation submodule is used to quantitatively evaluate the processing efficiency of the preprocessing module and the output detection module after applying the parameter modification strategy, so as to guide the optimization of the next round of parameter modification strategy. The processing efficiency includes at least one of the following: analyzing the changing trends of interception rate, false alarm rate, user experience impact, number of evasion attacks, and compliance indicators.

5. The information processing method according to claim 1 or 4, characterized in that, The closed-loop learning module determines the parameter modification strategies for the preprocessing module and the output detection module based on the first processing log, and further includes: Based on the analysis of the first processing log, a risk symbiotic relationship is analyzed between the user input information and the output information of the large language model; The parameter modification strategy is determined based on the aforementioned risk symbiosis relationship.

6. The information processing method according to claim 5, characterized in that, Based on the analysis of the first processing log, the risk symbiotic relationship between the user input information and the output information of the large language model is analyzed, including: Obtain the dialogue items in the first processing log that are at risk, wherein the dialogue items are divided into dialogue input information and dialogue output information; Each of the aforementioned dialogue items is organized into one or more risk item combinations; Filter out risk item combinations that occur more frequently than a preset threshold from all risk item combinations; The risk symbiosis relationship is determined based on the selected combinations of risk items.

7. The information processing method according to claim 1, characterized in that, The output detection module performs a risk assessment on the currently received lexical unit, and determines the target output information corresponding to the currently detected lexical unit based on the assessment result, including: The output detection module obtains the risk score of the currently detected word in the evaluation result; When the risk score is greater than or equal to the first threshold, the output detection module controls the large language model to stop generating word units and determines the preset information as the target output information; When the risk score is less than or equal to the second threshold, the decoded current detected word is determined as the target output information; When the risk score is greater than the second threshold and less than the first threshold, the output detection module acquires the target detection word whose reception time is after the current detection word; and determines the target output information based on the evaluation result of the target detection word.

8. The information processing method according to claim 1, characterized in that, The output detection module includes a real-time protection module and an output filtering module; The output detection module performs a risk assessment on the currently received lexical unit, and determines the target output information corresponding to the currently detected lexical unit based on the assessment result, including: The real-time protection module performs content classification, sandbox verification, and compliance verification on the currently received lexical units, and obtains classification results, verification results, and validation results. Based on the classification results, verification results, and validation results, the evaluation result is determined. The real-time protection module determines the preliminary output information of the currently detected word based on the evaluation results; The output filtering module performs secondary filtering on the initial output information through keyword matching to determine the target output information.

9. An information processing system, characterized in that, The information processing system includes: a preprocessing module, a large language model, an output detection module, and a closed-loop learning module; The preprocessing module is used to preprocess user input information and input the obtained target input information into the large language model. The large language model is used to process the target input information, generate streaming word units, and transmit them to the output detection module; The output detection module is used to perform risk assessment on the currently received lexical units and determine the target output information corresponding to the currently detected lexical units based on the assessment results. The closed-loop learning module is used to obtain the first processing logs of the preprocessing module and the output detection module, and to determine the parameter modification strategy of the preprocessing module and the output detection module based on the first processing logs.

10. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the information processing method as described in any one of claims 1 to 8.

Images