Dynamic security auditing method and system for output content of AI model and related equipment

Abstract

The invention discloses a dynamic security auditing method and system for output content of an AI model and related equipment, and the method comprises the steps: carrying out the preliminary security screening and deep semantic recognition of first information inputted by a user, and determining the risk level of the first information based on a recognition result; inputting the first information into an AI model, wherein the AI model outputs second information in a streaming manner; and in the process of streaming output of the second information by the AI model, determining an auditing frequency based on the risk level, performing security auditing on the generated second information by adopting the auditing frequency, and executing a corresponding security response strategy based on an auditing result. Therefore, according to the method, the safety of the output content in the industrial fault diagnosis scene is improved, the generation and output of the risk content can be intervened in real time, the processing mode of the AI model on the sensitive data is optimized, and reliable safety guarantee is provided for equipment fault diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial fault diagnosis, and in particular to an AI model output content dynamic security auditing method and system and related equipment. BACKGROUND

[0002] Currently, in the field of industrial fault diagnosis, an artificial intelligence (AI) model (such as a chat robot using an AI model) often needs to process sensitive content related to device operation, technical parameters, safety procedures, etc. Traditional security solutions are mainly based on rule matching, and only one-time filtering is performed at the input or output link, which cannot cope with dynamic security threats in the AI generation process, and lacks consideration of risk content and user experience in risk response, and cannot adapt to the AI model in the scene of a large number of content generation, including: unable to detect the intermediate results in the AI model generation process; the AI model lacks accurate identification of professional scene risks; the response mechanism for risk content is single, affecting user experience, etc.

[0003] Therefore, how to realize real-time and accurate security auditing in the process of AI model content generation has become a problem to be solved. SUMMARY

[0004] The present application provides an AI model output content dynamic security auditing method, system and related equipment, which ensures that the user obtains safe and reliable output content through real-time auditing of AI model generated content. The technical solution is as follows: In a first aspect, an AI model output content dynamic security auditing method is provided, which includes: performing preliminary security screening and deep semantic recognition on user input first information to determine the risk level of the first information; inputting the first information into an AI model to control the AI model to generate second information in a streaming manner; during the generation of the second information, determining an auditing frequency based on the risk level, using the auditing frequency to perform real-time security auditing on the segmented generated second information, and executing corresponding security response strategies based on the auditing results.

[0005] In combination with the first aspect, the risk level includes a low risk level, a medium risk level and a high risk level, and the auditing frequency is positively correlated with the risk level, specifically including: the risk level and the auditing frequency have a preset mapping relationship, and the mapping relationship includes: the high risk level corresponds to a first detection frequency, the medium risk level corresponds to a second detection frequency, and the low risk level corresponds to a third detection frequency, wherein the first detection frequency is higher than the second detection frequency, and the second detection frequency is higher than the third detection frequency.

[0006] In combination with the first aspect, the audit result includes a low-risk result, a medium-risk result, a high-risk result, and an emergency-risk result, and is obtained by weighting the second information in at least two risk dimensions, including an operation safety risk dimension, a technical leakage risk dimension, a compliance risk dimension, and an operation execution risk dimension.

[0007] In combination with the first aspect, in some embodiments of the first aspect, the corresponding security response strategy is executed based on the audit result, and specifically includes: if the audit result is the low-risk result, the security response strategy is a first response strategy of adding a security prompt in the second information; if the audit result is the medium-risk result, the security response strategy is a second response strategy of returning a restrictive operation suggestion of the second information; if the audit result is the high-risk result, the security response strategy is a third response strategy of rejecting output of the second information; and if the audit result is the emergency-risk result, the security response strategy is a fourth response strategy of immediately interrupting the AI model to generate the second information.

[0008] In combination with the first aspect, in the process of weighting the second information in at least two risk dimensions to obtain the audit result, the weight of the risk dimension is obtained by self-adaptive adjustment based on historical data in the industrial fault diagnosis field.

[0009] In combination with the first aspect, the deep semantic recognition is implemented by a language model fine-tuned in the industrial fault diagnosis field, and the preliminary security screening is based on a sensitive word library to screen the first information, wherein the sensitive word library at least includes three dimensions of equipment safety, technical secret, and compliance risk.

[0010] It should be noted that the features in the various embodiments of the first aspect can be combined with each other without conflict, and any combination of features in different embodiments is also within the protection scope of the present application, that is, the above-described multiple embodiments can also be combined arbitrarily according to actual needs.

[0011] In a second aspect, a system for dynamically auditing the output content of an AI model is provided, including: The preliminary audit module is configured to perform preliminary security screening and deep semantic recognition on the first information input by the user, and determine a risk level of the first information; the reply generation module is configured to input the first information into an AI model, and control the AI model to generate second information in a streaming manner; the security audit module is configured to determine an audit frequency based on the risk level during generation of the second information, perform real-time security audit on the second information generated in segments using the audit frequency, and execute a corresponding security response strategy based on an audit result; and the multi-dimensional risk assessment engine is configured to determine the risk level of the first information in the preliminary audit module, and perform real-time security audit on the second information generated in segments using the audit frequency in the security audit module.

[0012] In a third aspect, a computer is provided, including one or more memories, one or more processors; the memory is coupled to the one or more processors, the memory is configured to store computer program code, the computer program code includes computer instructions, the one or more processors invoke the computer instructions, so that the computer implements the method of the first aspect or any one of the implementation manners of the first aspect.

[0013] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions, the computer instructions are executed by a processor to implement the method of the first aspect or any one of the implementation manners of the first aspect.

[0014] In a fifth aspect, a chip is provided, and the chip is applied to a computer, and the chip includes one or more processors, and the processor is configured to invoke computer instructions to make the computer execute the method of the first aspect or any one of the implementation manners of the first aspect.

[0015] In the embodiments of the present application, the method provided by the present application improves the security of output content in an industrial fault diagnosis scene (such as a scene in which a user interacts with a fault diagnosis chat robot) by constructing a dynamic security audit system in a streaming generation process, can intervene in the generation and output of risk content in real time, optimizes the processing mode of sensitive data by an AI model, and provides reliable security guarantee for equipment fault diagnosis (such as vehicle fault diagnosis).

[0016] In the embodiments of the present application, preliminary security screening of input content can realize efficient interception of explicit and implicit high-risk content input; security audit of output content, determination of an audit frequency based on a risk level, and precise intervention in the risk of generated content can improve the security and reliability of AI model generated content in professional and high-risk field applications. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0018] Figure 1 is a schematic diagram of a communication system provided by an embodiment of the present application; Figure 2 is a schematic diagram of a system architecture for dynamically and safely auditing AI model output content provided by an embodiment of the present application; Figure 3 is a general method flow of dynamically and safely auditing AI model output content provided by an embodiment of the present application; Figure 4 is another method flow chart of dynamically and safely auditing AI model output content provided by an embodiment of the present application; Figure 5 is a module schematic diagram of a system for dynamically and safely auditing AI model output content provided by an embodiment of the present application; Figure 6 is a hardware structure schematic diagram of a computer device provided by an embodiment of the present application; Figure 7 is a schematic diagram of a computer readable storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0020] It should be understood that the "multiple" mentioned in the present application refers to two or more. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in this paper is only a description of the association between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and roles are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0021] The phrase "one embodiment" or "some embodiments" appearing in the present application are intended to refer to one or more embodiments of the present application including the particular feature, structure, or characteristic being described. Thus, appearance of the phrase "in one embodiment" or "in some embodiments" at different locations within the specification are not necessarily all referring to the same embodiment, although it can. Furthermore, the terms "comprise," "have," "contain," and "include," or variations thereof, are intended to be open-ended and mean that the object or entities listed are not exhaustive, unless otherwise expressly specifically recited.

[0022] Embodiments of the present application provide a method, system and related device for dynamically auditing AI model output content, which comprises: performing preliminary security screening and deep semantic recognition on the first information input by the user, determining the risk level of the first information based on the recognition result; inputting the first information into the AI model, and the AI model outputs the second information in a streaming manner; during the process of the AI model outputting the second information in a streaming manner, determining the auditing frequency based on the risk level, performing security auditing on the generated second information using the auditing frequency, and executing the corresponding security response strategy based on the auditing result.

[0023] In embodiments of the present application, the risk level includes a low risk level, a medium risk level and a high risk level, the risk level and the auditing frequency have a preset mapping relationship, the auditing frequency is positively correlated with the risk level, and the mapping relationship includes: the high risk level corresponds to a first detection frequency, the medium risk level corresponds to a second detection frequency, and the low risk level corresponds to a third detection frequency, wherein the first detection frequency is higher than the second detection frequency, and the second detection frequency is higher than the third detection frequency.

[0024] In embodiments of the present application, the auditing result includes a low risk result, a medium risk result, a high risk result and an emergency risk result, which is obtained by weighting and scoring the second information in at least two risk dimensions, and the risk dimensions include an operation safety risk dimension, a technology leakage risk dimension, a compliance risk dimension and an operation execution risk dimension.

[0025] In embodiments of the present application, the corresponding security response strategy is executed based on the auditing result, specifically including: the security response strategy corresponding to the low risk result is a first response strategy of attaching a guiding safety prompt in the second information; the security response strategy corresponding to the medium risk result is a second response strategy of returning a restrictive operation suggestion and refusing to output specific steps in the second information; the security response strategy corresponding to the high risk result is a third response strategy of replacing the output second information with a safety warning sentence; and the security response strategy corresponding to the emergency risk result is a fourth response strategy of immediately interrupting the AI model to generate the second information.

[0026] Thus, the dynamic security audit method for AI model output content provided in this application improves the security of output content in industrial fault diagnosis scenarios (such as scenarios where users interact with fault diagnosis chatbots) by constructing a dynamic security audit system in the streaming generation process. It can intervene in the generation and output of risky content in real time, optimize the way AI models process sensitive data, and provide reliable security for equipment fault diagnosis (such as vehicle fault diagnosis).

[0027] The following three embodiments describe the method, system, and related equipment for dynamic security auditing of AI model output content provided in this application. Embodiment 1 describes the communication system for the dynamic security auditing method for AI model output content. Embodiment 2 describes the method flow for implementing dynamic security auditing of AI model output content. Embodiment 3 describes the system module structure, computer equipment hardware structure, and computer storage medium for executing the dynamic security auditing method for AI model output content.

[0028] Example 1 Figure 1 This is a schematic diagram of a communication system provided in an embodiment of this application. For example... Figure 1 As shown, the communication system includes a user 100 and an electronic device 200.

[0029] In this embodiment, user 100 can directly send query information (i.e., first information) to electronic device 200, or input query information through the user interface of terminal device (such as chat box interface, application APP interface, etc.) and send the query information to electronic device 200 through communication network (such as Hypertext Transfer Protocol (HTTP) / Hypertext Transfer Protocol Secure (WebSocket) etc.).

[0030] In this embodiment of the application, user 100 may receive response content (i.e., second information) or security response from electronic device 200 and present it to user 100 in real time in a streaming or non-streaming manner.

[0031] In this embodiment, the electronic device 200 includes a communication interface, which is responsible for network communication with the terminal device, receiving user queries, distributing tasks to the backend module, and returning the final generated reply or security response to the terminal device.

[0032] In the embodiment of the present application, the electronic device 200 receives the query information, calls the sensitive word library and the semantic analysis model for screening. If the query information does not pass the screening, the electronic device 200 directly returns a preset response to the terminal device of the user 100 through the communication interface, and ends the process; if the query information passes the screening, the electronic device 200 sends the query and the calculated risk level to the AI model for obtaining the reply content.

[0033] In the embodiment of the present application, the electronic device 200 includes a multi-dimensional risk assessment engine responsible for real-time, multi-dimensional risk (such as operation safety risk dimension, technical leakage risk dimension, compliance risk dimension, etc.) assessment and level determination of the stream-generated content segment (i.e., the second information). The multi-dimensional risk assessment engine can realize full-process risk assessment and security audit of the input content and output content in the method and process of dynamic security audit of AI model output content.

[0034] In the embodiment of the present application, the electronic device 200 includes an AI model responsible for generating specific reply content for the query information that passes the screening. Meanwhile, in the process of the AI model generating the reply content in a streaming manner, the electronic device 200 performs real-time security audit on the content segment and executes a progressive security response strategy. The electronic device 200 can dynamically adjust the audit frequency of the content segment according to the risk level and the real-time audit result.

[0035] In the embodiment of the present application, the electronic device 200 can set an initial audit frequency for the reply content according to the risk level. The AI model starts to generate the reply content in a streaming manner, and the segments of the reply content can be audited in real time and in batches. Specifically, the electronic device 200 can send the generated reply content segment to the multi-dimensional risk assessment engine for real-time security audit based on the current audit frequency. The multi-dimensional risk assessment engine calculates and returns the audit result. The electronic device 200 determines whether to continue streaming the reply content segment to the terminal device of the user 100 or trigger a security response (i.e., the electronic device 200 executes the corresponding security response strategy based on the audit result) according to the audit result and the preset strategy.

[0036] In the embodiment of the present application, if the electronic device 200 detects that the audit result of the reply content is a low or medium risk result, it will generate a security prompt or a restrictive operation suggestion. These suggestions can be inserted into the data stream through the communication interface and sent to the user terminal together; if the electronic device 200 detects that the audit result of the reply content is a high risk result, it will immediately send a warning (such as a termination signal) to the AI model, forcibly interrupting the AI model to generate subsequent content. At the same time, the electronic device 200 returns the final security warning statement to the user terminal through the communication interface.

[0037] In the embodiments of this application, the electronic device 200 may include user equipment (UE), mobile device, mobile station, mobile unit, machine-to-machine (M2M) terminal, wireless unit, terminal device, remote unit, terminal agent, mobile client, etc.

[0038] In this way, the communication system establishes a communication network between user 100 and electronic device 200 that can provide real-time feedback and control, and realizes real-time security review decisions in the process of electronic device 200 generating content responses, thereby efficiently and reliably ensuring the security of AI-generated content.

[0039] In the embodiments of this application, Figure 1 The communication system described herein uses the communication between a user 100 and an electronic device 200 as an example. In practical applications, the communication system can have more or fewer devices. For example, the number of users 100 and electronic devices 200 can be even greater, and a user 100 can communicate with multiple electronic devices 200 simultaneously. In the following description, the dynamic security auditing method for AI model output content involved in the embodiments of this application can be applied to… Figure 1 The communication system shown.

[0040] Figure 2 This is a schematic diagram of a dynamic security auditing system architecture for AI model output content provided in an embodiment of this application, which is applied to, for example... Figure 1 The communication system shown. (As shown) Figure 2 As shown, the architecture of the dynamic security review system for AI model output content includes a data input layer, a semantic review layer, a streaming content generation control layer, and an output content detection and processing layer.

[0041] In this embodiment, the data input layer performs preliminary security screening on the first information input by the user. This includes receiving the user's input content (i.e., the first information), configuring a sensitive word library for the input content (which may contain words related to device security, technical secrets, compliance risks, etc.), and using a text matching algorithm to detect keywords in the user input for preliminary screening. The input content that passes the preliminary screening is then input into the semantic review layer for further analysis. Specifically, when the user input content matches a word in the sensitive word library, the subsequent detection process is immediately terminated and a preset response is returned.

[0042] In the embodiments of the present application, the semantic review layer includes a semantic analysis model for deep semantic recognition of the first information input by the user. The semantic analysis model is a language model fine-tuned in the field of industrial fault diagnosis, and further semantic analysis is performed on the input content screened by the sensitive word library. The semantic analysis model, for example, a large language model LLM, can realize user intent query of the input content, and based on the context detection, evaluate whether the semantic safety of the input content passes or not. If it does not pass, the subsequent detection process is terminated and a preset response is returned; if it passes, the input content is further analyzed for risk level.

[0043] In the embodiments of the present application, the semantic review layer further includes a risk assessment module for obtaining a risk assessment matrix of the input content through the semantic analysis model. The risk assessment matrix includes risk level, risk type and confidence, and is used for joint analysis of the input risk obtained by the semantic analysis model.

[0044] In the embodiments of the present application, the streaming content generation control layer includes an AI model, and the semantic review layer sends the first information recognized by the deep semantic recognition to the streaming content generation control layer. The AI model generates reply information (i.e. second information) input by the user in a streaming manner.

[0045] In the embodiments of the present application, the AI model includes a large language model LLM based on a Transformer decoder architecture, which is used to understand the natural language query information of the user and generate fluent, accurate and valuable reply content.

[0046] In the embodiments of the present application, in order to adapt the AI model to a specific field (such as the field of industrial fault diagnosis), the AI model is fine-tuned during its construction process, including continuous pre-training and instruction fine-tuning. For example, a dedicated material (such as professional textbooks, equipment manuals, maintenance case library, technical papers, etc.) can be used to continue pre-training the basic model, so that it can absorb and master the professional terms, concepts and knowledge logic in the field; question and answer pairs composed of instructions and answers can also be used for fine-tuning. These question and answer pairs are used to simulate real user queries, for example: the instruction is to query the diagnosis steps of the first type of vehicle engine overheating fault, and the corresponding answers are to confirm whether the cooling system is normally started, check the cooling liquid temperature, and at the same time give a safety warning: before proceeding to the next step, make sure that the vehicle is completely powered off, etc.; finally, experts in the field can be introduced to score and sort the model generation results, and further align with the professional judgment and value standard of human beings, so that the generated AI model is more inclined to generate safe, reliable and helpful replies.

[0047] In the embodiments of the present application, the streaming content generation control layer also acquires the risk assessment matrix, and determines the audit frequency of the reply information based on the risk assessment matrix (such as risk level). Specifically, the risk level includes a low risk level, a medium risk level, and a high risk level, the audit frequency is positively correlated with the risk level, and the risk level and the audit frequency have a preset mapping relationship, which includes: the high risk level corresponds to a first detection frequency (which can be referred to as a close detection strategy), the medium risk level corresponds to a second detection frequency (which can be referred to as a standard detection strategy), and the low risk level corresponds to a third detection frequency (which can be referred to as a relaxed detection strategy), wherein the first detection frequency is higher than the second detection frequency, and the second detection frequency is higher than the third detection frequency.

[0048] In some embodiments, during the process of generating the second information by the AI model in a streaming manner, the streaming content generation control layer can also dynamically adjust the audit frequency based on the risk level of the second information, and establish a real-time content buffering mechanism in the streaming generation process.

[0049] In the embodiments of the present application, the output content detection and processing layer is configured to receive the second information output by the streaming content generation control layer, and the audit frequency information of the second information. At the same time, based on the audit frequency, the second information generated in segments is subjected to real-time security audit, and a responsive security response strategy is executed on the audit result.

[0050] In the embodiments of the present application, the output content detection and processing layer includes a security audit module, a security response strategy, and a final output process, wherein the security audit module is configured to perform security audit on the second information and obtain an audit result; the security response strategy module corresponds to the audit result and is configured to be executed in the final output process.

[0051] In the embodiments of the present application, the audit result includes a low-risk result, a medium-risk result, a high-risk result, and an emergency risk result, which is obtained by weighting the second information in at least two risk dimensions (such as operation safety risk dimension and technical leakage risk dimension). The risk dimensions include operation safety risk dimension, technical leakage risk dimension, compliance risk dimension, and operation execution risk dimension.

[0052] In the embodiments of the present application, the corresponding security response strategy is executed based on the audit result, specifically including: if the audit result is a low-risk result, the security response strategy is a first response strategy of attaching a security prompt in the second information; if the audit result is a medium-risk result, the security response strategy is a second response strategy of returning a limited operation suggestion of the second information; if the audit result is a high-risk result, the security response strategy is a third response strategy of rejecting the output of the second information; and if the audit result is an emergency risk result, the security response strategy is a fourth response strategy of immediately interrupting the generation of the second information by the AI model.

[0053] It can be understood that the function division between the modules shown in the embodiments of the present application is only illustrative and does not constitute a function limitation of the AI model output content dynamic security auditing system architecture. In other embodiments of the present application, the AI model output content dynamic security auditing system architecture can also be implemented in a manner of different modules or a combination of multiple modules to realize the functions in the AI model output content dynamic security auditing system architecture.

[0054] Embodiment two Figure 3 is a general method flowchart for dynamically auditing the output content of an AI model provided by an embodiment of the present application, applied to a communication system as shown in Figure 1 and an AI model output content dynamic security auditing system architecture as shown in Figure 2 , and specifically includes the following steps. S101. Obtain first information.

[0055] In the embodiments of the present application, the AI model output content dynamic security auditing system architecture (hereinafter referred to as system architecture) can be deployed in an electronic device 200, and based on the communication interface of the electronic device 200, the natural language query information sent by the terminal device of the user is received. The query information is the first information, which can be in the form of text, voice (which can be converted after voice recognition) or picture containing text (which can be converted after optical character recognition OCR). After receiving the information, the communication interface performs preliminary analysis and standardization to ensure that the first information meets the input requirements of the subsequent processing module and prepares for security detection.

[0056] S102. Perform preliminary security screening and deep semantic recognition to determine the risk level.

[0057] In the embodiments of the present application, the first information input by the user is subjected to preliminary security screening and deep semantic recognition to determine the risk level of the first information.

[0058] Specifically, the preliminary security screening quickly matches the first information with a high-frequency and high-confidence sensitive word library. If an accurate matching high-risk word (such as an unauthorized illegal instruction) is found, the system architecture can immediately determine that the first information is unsafe information and directly execute a preset response step (such as returning information that the input information contains an unauthorized illegal instruction), and terminate the subsequent processing flow.

[0059] Specifically, for the first information screened through the preliminary screening, the system architecture can perform deep semantic recognition and multi-dimensional risk assessment on the first information by calling a semantic analysis model. The semantic analysis model can obtain the real intention of the user inputting the first information, and output a preliminary score of the first information on the preset dimensions such as operation safety, technology leakage, compliance, etc. Finally, the system architecture can aggregate the scores according to the weights of the preset dimensions to obtain a comprehensive score, and finally map to a preset risk level (including a low risk level, a medium risk level and a high risk level), which is used to provide prior guidance for subsequent security audit of the second information generated in a streaming manner.

[0060] In the embodiments of the present application, the weights of the preset dimensions can be obtained by self-adaptive adjustment based on historical data in the field of industrial fault diagnosis, and the risk threshold is determined based on the weights of the preset dimensions, specifically including: collecting historical dialogue data in the field of industrial fault diagnosis and performing multi-dimensional risk labeling by domain experts; based on the multi-dimensional risk labeled data, the initial weights of each dimension are trained by a machine learning model (such as logistic regression, decision tree, etc.), and the final weight distribution strategy is determined by combining expert decision and calibration; by analyzing the comprehensive risk score distribution of the labeled samples, the best risk level threshold is determined by using clustering algorithm, and is confirmed by expert review, and at the same time, the periodic iteration mechanism is used to continuously optimize the weights and threshold according to the newly generated feedback data.

[0061] S103. Input the first information into the AI model and output the second information in a streaming manner.

[0062] In the embodiments of the present application, based on the first information screened by the step S102, the AI model in the system architecture is input and the second information is output in a streaming manner, wherein the AI model can be a language model trained based on a large amount of corpus, which can understand the user's intention and generate corresponding reply content, i.e. the second information.

[0063] In the embodiments of the present application, the streaming output mode of the AI model means that the AI model does not generate all the content at once, but generates and outputs in segments and sentences in units of tokens (Token). The content segments generated in these segments can be audited in real time through a data pipeline, which creates technical conditions for subsequent segmented security audit of the second information.

[0064] S104. Determine the audit frequency, perform security audit on the second information, and execute the corresponding security response strategy.

[0065] In the embodiments of the present application, the system architecture determines the audit frequency based on the risk level, adopts the audit frequency to perform real-time security audit on the second information generated by segmentation, and executes the corresponding security response strategy based on the audit result, wherein the system architecture determines the initial audit frequency based on the risk level obtained in step S102. For example, a first detection frequency (such as auditing each output sentence in the second information) is adopted for high-risk content, a second detection frequency (such as auditing one or more output sentences in the second information at intervals) is adopted for medium-risk content, and a third detection frequency (such as auditing one or more extracted sentences in the second information) is adopted for low-risk content.

[0066] In the embodiments of the present application, the system architecture performs real-time interception on the second information segment generated in a streaming manner according to the determined audit frequency, and calls a multi-dimensional risk assessment engine for rapid analysis (i.e., security audit). The multi-dimensional risk assessment engine can combine the context of the generated content to calculate the real-time risk level and risk type in real time.

[0067] In the embodiments of the present application, the system architecture executes a preset gradual security response strategy according to the real-time security audit result, specifically including: Low-risk result: The system architecture allows the second information segment content to be normally streamed to the user terminal.

[0068] Medium-risk result: While returning the second information segment content, the system architecture attaches a security warning or a restrictive suggestion (such as an instruction such as “this operation suggestion needs to be carried out under the guidance of professional personnel”).

[0069] High-risk result: The system architecture terminates the output of the second information segment content, and can return a security warning information to the user terminal.

[0070] Emergency risk result: The system architecture immediately sends a termination signal to the AI model, interrupts the subsequent content generation, and returns a forced security warning information to the user, and timely terminates the generation of high-risk content.

[0071] Based on the method shown in the foregoing steps S101-S104, the technical solution of the present application realizes the security management of the AI generated content through the processing flow of dynamic feedback. The flow starts from the first information acquisition request input by the user, outputs accurate security response through real-time detection of the AI model generated content, and guarantees the security of the output content. At the same time, the method realizes the security control of the AI model output content through the close combination of streaming processing and dynamic regulation, and finally provides the user with a safe and reliable and smooth and efficient human-computer interaction experience.

[0072] It should be understood that, although the above Figure 3The steps in the flowchart involved are displayed in sequence according to the direction of the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, as described above Figure 3 At least some of the steps in the flowchart involved can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps in other steps.

[0073] Figure 4 is another method flowchart for dynamically auditing the output content of an AI model provided by an embodiment of the present application, applied to a communication system as shown in Figure 1 and a system architecture for dynamically auditing the output content of an AI model as shown in Figure 2 , specifically comprising: S201. Audit the keywords of the input content.

[0074] In an embodiment of the present application, the system architecture obtains the input content (i.e. the first information), and performs preliminary security screening by quickly matching it with a pre-constructed sensitive word library. For details, please refer to the foregoing Figure 3 S101-S102 steps, which will not be described here.

[0075] In an embodiment of the present application, the sensitive word library contains words and phrases that have high-risk operations on devices or users, and its matching rules can be exact matching or support fuzzy matching (such as case-insensitive detection).

[0076] S202. Determine whether there is a sensitive word.

[0077] In an embodiment of the present application, based on the result of the preliminary security screening in the foregoing S201 step, it is determined whether the input content has a sensitive word: if it is determined to be yes (i.e. there is a sensitive word), it means that the input content has explicit and extremely high risk, and the flow jumps to the subsequent S205 step; if it is determined to be no, it means that the input content has passed the preliminary security screening, and the flow continues to execute the subsequent S203 step for more in-depth intent analysis.

[0078] S203. Audit the semantics of the input content and determine the risk level of the input content.

[0079] In an embodiment of the present application, for the content that passes the keyword screening, the system architecture calls a semantic analysis model (such as a lightweight text classification or natural language understanding model) to analyze it.

[0080] In the embodiment of the present application, the semantic analysis model can understand the true intention, context and potential implied risks of the query beyond the literal meaning, wherein the semantic analysis model can score the input content on multiple preset risk dimensions (such as operation safety, information compliance, and technology leakage), and the system architecture calculates the comprehensive risk level according to the preset weight and aggregation algorithm. The specific description of the semantic analysis model can also refer to the foregoing S102 step, which will not be repeated here.

[0081] S204. Determine whether the semantics are safe.

[0082] In the embodiment of the present application, based on the risk level obtained in the foregoing S203 step, the system architecture determines whether the semantics of the input content are safe: if it is determined to be no (i.e., the semantics are unsafe, for example, the risk level is a high risk level), the flow jumps to the subsequent S205 step, and the subsequent content generation request is rejected; if it is determined to be yes, the input content is also determined to be safe or controllable in terms of intention, and the flow performs the subsequent S206 step, and prepares to enter the content generation phase.

[0083] S205. Return a preset rejection response.

[0084] In the embodiment of the present application, the system architecture returns a preset rejection response to the input content that does not pass the security screening. Specifically, when the input content is determined to be unsafe in the S202 step or the S204 step, the system architecture will terminate the subsequent flow, and return a preset security response to the terminal device of the user through the communication interface, for example, prompting the user that the input content involves security risks and cannot provide related answers.

[0085] S206. Determine the detection intensity of the output content based on the risk level.

[0086] In the embodiment of the present application, for the input content that passes the screening (usually with a low risk level or a medium risk level), the system architecture enters the preparation phase of generating reply information (i.e., the second information). Wherein, the system architecture can preset the initial audit frequency and depth (i.e., the detection intensity) for the second information to be generated based on the risk level determined in the S203 step. For example, for medium risk content, set to audit the output content every interval sentence (i.e., the second detection frequency); for low risk content, set to audit one or more sentences extracted from the output content (i.e., the third detection frequency), which can realize the on-demand allocation of computing resources and avoid waste of computing power. The specific description of determining the detection intensity of the output content can also refer to the foregoing S104 step, which will not be repeated here. Figure 3

[0087] S207. Generate output content and perform real-time security audit.​

[0088] In the embodiment of the present application, the system architecture generates output content in a streaming manner based on the AI model, that is, text segments are generated word by word and sentence by sentence, and meanwhile, based on the detection intensity determined in the S206 step, the content segments in the generation are intercepted in real time, and a multi-dimensional risk assessment engine is called to quickly analyze them, and the risk level and risk type of the output content are calculated in real time. The specific description of the security audit of the generated output content can also be referred to the S104 step in the foregoing Figure 3 , which will not be repeated here.

[0089] S208. Based on the security audit result, execute the security response strategy on the output content.

[0090] In the embodiment of the present application, the system architecture executes the progressive security response strategy based on the audit result obtained in the S207 step, which can be specifically referred to the S104 step in the foregoing Figure 3 , which will not be repeated here.

[0091] S209. Output the content determined by the security response strategy.

[0092] In the embodiment of the present application, the system architecture returns the content determined by the security response strategy to the terminal device of the user through the communication interface, which is, for example, when the whole process is risk-free, the generated content without modification is output; when a low-risk result is detected, the generated content with a security prompt is output; when a medium-risk result is detected, the generated content which is partially truncated and replaced with a security warning is output; when a high-risk result or an emergency risk result is detected, a rejection information composed of a preset security response is output.

[0093] Based on the method shown in the foregoing S201-S209 steps, the present application realizes a dynamic security audit method for AI model output content. Among them, the S201-S205 steps are used for screening input content, realizing efficient interception of explicit and implicit high-risk content input; the S206-S209 steps are used for security auditing of output content, determining the auditing frequency based on the risk level, and implementing accurate intervention on the risk of the generated content. The method process improves the safety and reliability of the application of AI model generated content in professional and high-risk fields.

[0094] Embodiment Three Figure 5 is a module schematic diagram of a dynamic security audit system for AI model output content provided by the embodiment of the present application. As Figure 5 shown, the dynamic security audit system for AI model output content 500 specifically includes the following modules: The preliminary audit module 510 is used for performing preliminary security screening and deep semantic recognition on the first information input by the user, and determining the risk level of the first information.

[0095] The reply generation module 520 is configured to input the first information into an AI model, and control the AI model to generate second information in a streaming manner.

[0096] The security audit module 530 is configured to determine an audit frequency based on the risk level during generation of the second information, perform real-time security audit on the second information generated in segments using the audit frequency, and execute a corresponding security response strategy based on an audit result.

[0097] The multi-dimensional risk assessment engine 540 is configured to determine the risk level of the first information in the preliminary audit module, and perform real-time security audit on the second information generated in segments using the audit frequency in the security audit module.

[0098] It can be understood that the function division between the modules illustrated in the embodiments of the present application is only illustrative, and does not constitute a function limitation of the AI model output content dynamic security audit system 500. In some other embodiments of the present application, the AI model output content dynamic security audit system 500 can also be implemented in a manner of different modules or a combination of multiple modules to realize the function of the AI model output content dynamic security audit system 500.

[0099] Figure 6 is a hardware structure schematic diagram of a computer device provided by an embodiment of the present application. The computer device 600 can include the electronic device 200 shown in Figure 1 and the AI model output content dynamic security audit system 500 shown in Figure 5 . As shown in Figure 6 , the computer device 600 includes a processor 601, a memory 602, a communication module 604, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps of the AI model output content dynamic security audit shown in Figure 3-4 are implemented. For example, the computer program 603 can be divided into one or more units / modules, which are stored in the memory 602 and executed by the processor 601 to complete the present application.

[0100] The one or more units / modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 603 in the computer device 600. For example, the computer program 603 can be used to execute the steps of the AI model output content dynamic security audit shown in Figure 3The method, specific function or mechanism shown in steps S101-S104 is described in the above embodiments, and thus is not described here again.

[0101] Those skilled in the art can understand that, Figure 6 The computer device 600 is only an example, and does not constitute a limitation on the computer device 600, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the computer device 600 can also include an input / output device, a network access device, a bus, and the like.

[0102] The processor 601 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable logic array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0103] In some embodiments, the processor 601 can include one or more interfaces. The interfaces can include an I2C interface, an I2S interface, a PCM interface, a UART interface, an MIPI interface, a GPIO interface, an OBD interface, and / or a USB interface, and the like. It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative, and does not constitute a limitation on the structure of the computer device 600. In another embodiment of the present application, the computer device 600 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0104] In some embodiments, the computer device 600 can connect the internal devices and modules through one or more interfaces. The memory 602 can be an internal storage unit of the computer device 600, such as a hard disk or a memory of the computer device 600. The memory 602 can also include both the internal storage unit of the computer device 600 and an external storage device. The memory 602 is used to store the computer program and other programs and data required by the computer device 600. The memory 602 can also be used to temporarily store data that has been output or will be output.

[0105] The communication module 604 can provide a solution for wireless communication including wireless local area network (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The communication module 604 can be one or more device communication modules integrated with at least one communication processing module, receive electromagnetic waves via an antenna, demodulate and filter the electromagnetic wave signals, and send the processed signals to the processor 601. The communication module 604 can also receive signals to be sent from the processor 601, frequency modulate them, amplify them, and radiate them as electromagnetic waves via an antenna.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the above described functions.

[0107] The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software functional unit.

[0108] In the embodiments of the present application, the specific names of the functional units and modules are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. It should be understood that each step in the above method embodiments provided by the present application can be completed by integrated logic circuit of hardware or instruction in the form of software in the processor. The method steps disclosed in the embodiments of the present application can be directly embodied as hardware processor execution completion, or executed by hardware and software module combination in the processor.

[0109] The present application also provides a computer program product, which comprises a computer program (also referred to as code or instruction), which, when executed, causes a computer to execute the method performed by the AI model output content dynamic security audit in the above embodiments.

[0110] The embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0111] In the embodiments provided in the present application, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of computer program product in whole or in part.

[0112] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0113] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instructions). When the computer program is executed, the computer performs the method executed by the computer device in any one of the preceding embodiments.

[0114] Figure 7 is a schematic diagram of a computer-readable storage medium provided by an embodiment of the present application. As shown in Figure 7 The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0115] The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital versatile disc DVD), or a semiconductor medium (for example, a solid state disk SSD) and the like.

[0116] Those of ordinary skill in the art can understand that all or part of the processes in the foregoing embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the foregoing method embodiments. The foregoing storage medium includes a readable memory ROM or a random memory RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0117] In summary, the above description is only an embodiment of the technical solution of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present application shall be included in the protection scope of the present application.

Claims

1. A method for dynamic security auditing of AI model output content, characterized in that, The method includes: The first information input by the user is subjected to preliminary security screening and deep semantic recognition to determine the risk level of the first information. The first information is input into the AI ​​model, and the AI ​​model is controlled to generate the second information. During the generation of the second information, the review frequency is determined based on the risk level, and the second information is reviewed in real time using the review frequency. Based on the review results, the corresponding security response strategy is executed.

2. The method according to claim 1, characterized in that, The risk levels include low risk, medium risk, and high risk, and the review frequency is positively correlated with the risk level, specifically including: The risk level and the review frequency have a preset mapping relationship, which includes: the high risk level corresponds to the first detection frequency, the medium risk level corresponds to the second detection frequency, and the low risk level corresponds to the third detection frequency, wherein the first detection frequency is higher than the second detection frequency, and the second detection frequency is higher than the third detection frequency.

3. The method according to claim 1, characterized in that, The audit results include low-risk, medium-risk, high-risk, and emergency-risk results, which are obtained by weighting the second information on at least two risk dimensions, including operational security risk, technology leakage risk, compliance risk, and operational execution risk.

4. The method according to claim 3, characterized in that, The execution of the corresponding security response strategy based on the audit results specifically includes: If the audit result is the low-risk result, then the security response strategy is the first response strategy of adding a security prompt to the second information; If the audit result is a medium-risk result, then the security response strategy is a second response strategy that returns restrictive operational suggestions to the second information; If the audit result is a high-risk result, then the security response strategy is a third response strategy that rejects the output of the second information; If the audit result is the emergency risk result, then the security response strategy is the fourth response strategy of immediately interrupting the AI ​​model from generating the second information.

5. The method according to claim 1, characterized in that, In the process of obtaining the audit result by weighting the second information across at least two risk dimensions, the weights of the risk dimensions are adaptively adjusted based on historical data in the field of industrial fault diagnosis.

6. The method according to claim 1, characterized in that, The deep semantic recognition is achieved through a language model fine-tuned in the field of industrial fault diagnosis. The preliminary security screening is based on a sensitive word library to screen the first information. The sensitive word library includes at least three dimensions: equipment security, technical secrets, and compliance risks.

7. A dynamic security review system for the output content of an AI model, characterized in that, include: The preliminary review module is used to perform preliminary security screening and deep semantic recognition on the first information input by the user to determine the risk level of the first information. The response generation module is used to input the first information into the AI ​​model and control the AI ​​model to generate the second information. The security audit module is used to determine the audit frequency based on the risk level during the generation of the second information, perform real-time security audits on the second information using the audit frequency, and execute corresponding security response strategies based on the audit results.

8. The dynamic security review system for AI model output content according to claim 7, characterized in that, It also includes a multi-dimensional risk assessment engine, which is used to determine the risk level of the first information in the preliminary review module, and to perform real-time security review of the second information using the review frequency in the security review module.

9. A computer device, characterized in that, The device includes one or more memories and one or more processors; the memories are coupled to the one or more processors, the memories are used to store computer program code, the computer program code including computer instructions, and the one or more processors invoke the computer instructions to cause the computer device to implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1 to 6.

Images