Question answering system, security defense method and program product

Abstract

The invention provides a question answering system, a security defense method and a program product, and relates to the technical field of computers. The question answering system comprises a discriminant analysis module used for obtaining a query request from a user, the query request comprising a question input by the user, triggering a large language model to perform semantic analysis on the question input by the user, and performing task structure analysis on the question input by the user to assess whether the intention of the question input by the user is malicious or not, obtaining a safety judgment result about whether the question input by the user is the harmful content query; and the judgment response module is used for triggering the large language model to generate a reply response aiming at the question input by the user in combination with the security judgment result. According to the embodiment of the invention, the safety of the question answering system can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a question-answering system, a security defense method, and a program product. Background Art

[0002] In recent years, large language models (LLMs), such as GPT-4, Claude-3.5, and Llama-3.1, have demonstrated unprecedented capabilities in a variety of areas, including natural language understanding and complex reasoning. However, these models also face serious security challenges, particularly their vulnerability to carefully crafted "jailbreak attacks." These attacks can bypass the models' built-in security mechanisms, tricking them into generating harmful content and posing a significant threat to their practical applications. Summary of the Invention

[0003] The present disclosure provides a question-answering system, a security defense method, and a program product, which, at least to a certain extent, address the problem that some jailbreak attacks can bypass the security mechanisms built into the model and induce the model to generate harmful content, thereby improving the security of large language models.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0005] According to one aspect of the present disclosure, a question-answering system is provided, including a discriminant analysis module and a discriminant response module.

[0006] The discriminant analysis module is used to obtain query requests from users, including questions entered by users, trigger the large language model to perform semantic analysis on the questions entered by users, and perform task structure analysis on the questions entered by users to assess whether the intention of the questions entered by users is malicious, and obtain a security judgment result on whether the questions entered by users are harmful content queries;

[0007] The discrimination response module is used to trigger the large language model and combine it with the security judgment results to generate a reply response to the questions entered by the user.

[0008] According to another aspect of the present disclosure, a security defense method is provided, including: obtaining a query request from a user, the query request including a question input by the user; triggering a large language model to perform semantic analysis on the question input by the user, and performing task structure analysis on the question input by the user to evaluate whether the intention of the question input by the user is malicious, and obtaining a security judgment result of whether the question input by the user is a query for harmful content; triggering the large language model to generate a reply response to the question input by the user in combination with the security judgment result.

[0009] According to another aspect of the present disclosure, an electronic device is provided, including: a memory for storing instructions; and a processor for calling the instructions stored in the memory to implement the above-mentioned security defense method.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned security defense method is implemented.

[0011] According to another aspect of the present disclosure, a computer program product is provided. The computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the above-mentioned security defense method.

[0012] According to yet another aspect of the present disclosure, there is provided a chip comprising at least one processor and an interface;

[0013] An interface for providing program instructions or data to at least one processor;

[0014] At least one processor is used to execute program instructions to implement the above security defense method.

[0015] The question-answering system, security defense method, and program product provided by the embodiments of the present disclosure cleverly integrate the model's discrimination capabilities into its generation process. Each query request will first undergo a security assessment, making security an inherent component of the generation process rather than an afterthought. This makes use of the model's existing discrimination capabilities to effectively prevent the generation of unsafe or inappropriate content, thereby improving security. By performing semantic analysis on user-input questions and performing task structure analysis on user-input questions, it can more accurately identify harmful content queries, and then refuse to generate replies for query requests that are considered unsafe or suspicious. Compared with the method of generating a reply and then determining whether the reply content contains harmful information, this method can reduce the waste of computing power, optimize the overall model performance, and make it more robust and reliable.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] Obviously, the drawings described below are only some embodiments of the present disclosure. A person skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0019] Figure 1 A schematic diagram of a question-answering system according to an embodiment of the present disclosure is shown;

[0020] Figure 2 The safety performance comparison results of SAGE in the embodiment of the present disclosure and several benchmark models are shown;

[0021] Figure 3 The performance results of SAGE and other defense methods in the disclosed embodiments on three common benchmark tests are shown;

[0022] Figure 4 TCPS showing SAGE and other defense methods in the embodiments of the present disclosure;

[0023] Figure 5 The performance results of two variants of SAGE in the embodiments of the present disclosure on ReNeLLM and DeepInception are shown;

[0024] Figures 6a-6c 2. The hidden states of the three models are visualized using two-dimensional PCA in the embodiment of the present disclosure;

[0025] Figure 7 A flow chart of a security defense method according to an embodiment of the present disclosure is shown;

[0026] Figure 8 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0028] Existing jailbreak attack methods can be divided into two main categories:

[0029] Optimization-based attack methods: These methods break security constraints by iteratively optimizing harmful hints and continuously adjust attack strategies using query feedback or gradient-based methods. Typical examples include GCG and AutoDAN, which often require access to the model's internal parameters or gradient information.

[0030] Template-based attack methods: These methods construct specialized instruction templates to mislead the model into generating harmful content. These attacks are implemented through techniques such as scene nesting and multi-round rewriting. Representative works include ReNeLLM, PAIR, and DeepInception. These methods are more universal and can be applied to various models.

[0031] The solutions to the above-mentioned problems in related technologies still have some problems and need further improvement. For example, methods based on model retraining use technologies such as RLHF to enhance model security, but they suffer from high computational overhead, may lead to "alignment tax" problems, and may even be prone to catastrophic forgetting. Policy-based methods use the model's ability to follow instructions to enhance security, and implement defenses through means such as security declarations, example rejection, and intent analysis. These methods are relatively simple to implement, but their effectiveness is often limited.

[0032] During their research, the inventors discovered that large language models exhibit significant differences in capabilities when used as discriminators and generators. Specifically:

[0033] Discriminative ability: When the model is asked to judge the safety of input content, it can fairly accurately identify harmful content, including those carefully crafted jailbreak attack prompts.

[0034] Generated Behavior: Even if the same model has previously been shown to identify the harmfulness of the above content, when the same model directly processes these inputs, it often produces unsafe responses.

[0035] For example, after multiple experiments, the inventors showed that Llama-3.1-8B-Instruct:

[0036] When used as a discriminator, it can correctly identify 100% of harmful requests, but when used as a generator, it can only defend against 34% of attacks. This huge gap shows that the model has underutilized security potential.

[0037] Based on the above findings of the inventors, the embodiments of the present disclosure propose a security defense solution applied to large language models. By migrating the model's powerful discriminative ability to the generation process, the overall security of the model is improved. This solution is a training-free defense strategy. Its core idea is to align the model's discriminative ability with its generation ability, and enhance the model's security through structured analysis and response mechanisms.

[0038] The defects of the above solutions and the proposed solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0039] This exemplary implementation is described in detail below with reference to the accompanying drawings and examples.

[0040] Figure 1 A question-answering system in an embodiment of the present disclosure is shown. The question-answering system can be a question-answering system implemented based on a large language model, that is, the steps performed by each module of the question-answering system below can be specifically implemented by the large language model.

[0041] like Figure 1 As shown, the question-answering system includes a discriminant analysis module 101 and a discriminant response module 102 .

[0042] The discriminant analysis module 101 is used to obtain a query request from a user, where the query request includes a question input by the user, trigger the large language model to perform semantic analysis on the question input by the user, and perform task structure analysis on the question input by the user to evaluate whether the intention of the question input by the user is malicious, and obtain a security judgment result on whether the question input by the user is a harmful content query.

[0043] The questions input by users refer to the specific questions or queries submitted by users to the question-answering system, which can be any information for which answers are sought.

[0044] Receiving a query request involves receiving a question or query from a user, typically through a user interface such as a web form, mobile application, or other interactive interface.

[0045] The judgment and response module 102 is used to trigger the large language model to combine the security judgment results to generate a reply response to the question input by the user.

[0046] In some embodiments, the question-answering system may further include a user input module. The user input module may be used to generate a query request in response to a question input by a user. The user input module may also be used to construct a compound instruction sequence for the query request, the compound instruction sequence comprising a discriminant analysis instruction, a discriminant response instruction, and a query request connected in series. The query request from the user obtained by the discriminant analysis module 101 may be the compound instruction sequence. The compound instruction sequence is an instruction set consisting of multiple parts, used to guide the large language model to follow a specific security protocol when processing user queries.

[0047] Discriminant analysis module 101 executes the discriminant analysis instructions in the composite instruction sequence, guiding the large language model to perform a security assessment before generating a response to the user's question. Specifically, it performs semantic analysis and task structure analysis on the user's question to assess whether the user's question is malicious, and obtains a security assessment result as to whether the user's question is a harmful content query. In other words, the discriminant analysis instructions guide the large language model to perform a security assessment on user-submitted content, ensuring that the content is checked for potentially harmful information before generating a response.

[0048] The discrimination response module 102 executes the discrimination response instruction in the compound instruction sequence to guide the large language model to combine the security judgment result and generate a reply response to the question input by the user.

[0049] The disclosed embodiment sets up a two-stage security check mechanism of semantic analysis and task structure analysis, namely, semantic analysis of questions input by users, and task structure analysis of questions input by users. In the analysis at the semantic level, the model will evaluate the harmfulness of the content itself, without being affected by its surface presentation. In the analysis at the task level, the model is enabled to identify whether harmful content is embedded in seemingly harmless tasks. Through the above-mentioned semantic analysis and task structure analysis, harmful content queries can be more accurately identified, and then for query requests that are considered unsafe or suspicious, responses can be refused to be generated. Compared with the method of generating a response and then judging whether the response content contains harmful information, the waste of computing power can be reduced, and the overall model performance is optimized, making it more robust and reliable.

[0050] In some embodiments, the discrimination response module 102 is used to trigger the large language model to process a first reply response to the question input by the user when the result of the security judgment is that the question input by the user is not a harmful content query, and the first reply response is that the large language model replies to the question input by the user; the discrimination response module 102 is also used to trigger the large language model to process a second reply response to the question input by the user when the result of the security judgment is that the question input by the user is not a harmful content query, and the second reply response is that the large language model refuses to reply to the question input by the user.

[0051] The discrimination response module 102 triggers the first reply response or the second reply response according to the result of the security judgment. The first reply response is to reply to the question input by the user, and the second reply response is to refuse to reply to the question input by the user. That is to say, if it is judged to be safe, the first reply response is triggered; if it is not safe, the second reply response is triggered. It is understandable that if the first reply response is triggered, the large model needs to use computing power to analyze and process the question input by the user to obtain the specific content of the first reply response. This process requires more computing power than the second reply response. In the embodiment of the present disclosure, when it is unsafe, the first reply response is not triggered, but the second reply response is triggered, which can save computing power.

[0052] As an example, a compound instruction sequence may look like this:

[0053]

[0054] Among them, P usr Indicates a query request (or a question entered by the user), I da Indicates discriminant analysis instruction, I dr Indicates the judgment response instruction. Indicates a connection.

[0055] Although large language models themselves have certain capabilities to identify potentially harmful content, when directly faced with such inputs, they may ignore security considerations due to their focus on content generation, resulting in unsafe outputs. The disclosed solution cleverly integrates the model's discriminative capabilities into its generation process. By inserting discriminative analysis instructions and discriminative response instructions before the query request, it ensures that each query request will first undergo a security assessment, making security an inherent component of the generation process rather than an afterthought. It can utilize the model's existing discriminative capabilities to effectively prevent the generation of unsafe or inappropriate content, thereby improving security. In addition, for query requests that are considered unsafe or suspicious, refusing to generate replies can reduce the waste of computing power compared to the method of generating a reply and then determining whether the reply content contains harmful information, optimize the overall model performance, and make it more robust and reliable.

[0056] In some embodiments, when the determination response module 102 determines that the query request is a query for harmful content, the second reply response further includes a consideration process of whether the query request is a query for harmful content.

[0057] In some embodiments, when the determination response module 102 determines that the query request is not a query with harmful content, the first reply response does not include a thought process that the query request is not a query with harmful content.

[0058] In some embodiments, the safety judgment is used to determine whether the question input by the user is a query for harmful content, where harmful content is determined based on ethics, laws, and industry guidelines.

[0059] The disclosed embodiment integrates the model's discriminative capabilities into its generation process, ensuring that each query request undergoes a security assessment first, making security an inherent component of the generation process rather than an afterthought. This leverages the model's existing discriminative capabilities to effectively prevent the generation of unsafe or inappropriate content, thereby improving security.

[0060] In some embodiments, the discriminant analysis module 101 performs semantic analysis on the question input by the user, which may be: using a predefined keyword library (such as "harmful keywords") to detect whether the question input by the user contains harmful content, and determining that the question input by the user is a harmful content query if it contains harmful content; using the hidden states of the model to analyze the contextual semantics of the question input by the user, and using a deep learning model to understand the intention of the question input by the user based on the contextual semantics, determine whether the intention is malicious, and determine that the question input by the user is a harmful content query if the intention is malicious; classifying the safety of the question input by the user through a logistic regression model to obtain a classification result of whether the question input by the user is a harmful content query.

[0061] In the semantic level analysis of the disclosed embodiment, the model will evaluate the harmfulness of the content itself without being affected by its surface presentation, and can more accurately identify harmful content queries.

[0062] In some embodiments, the discriminant analysis module 101 performs a task structure analysis on the user-input question, which may include breaking the user-input question into basic components; based on the basic components, identifying whether the user-input question contains instructions that attempt to circumvent or ignore security judgment criteria; and determining whether the user-input question is a harmful content query if instructions that attempt to circumvent or ignore security judgment criteria are contained. In some embodiments, breaking the user-input question into basic components may be performed recursively or hierarchically.

[0063] In the task-level analysis, the disclosed embodiment enables the model to identify whether harmful content is embedded in seemingly harmless tasks, and to identify more subtle harmful content queries. It then refuses to generate responses for query requests that are considered unsafe or suspicious. This reduces the waste of computing power compared to the method of generating a response and then determining whether the response content contains harmful information.

[0064] In some embodiments, the above-mentioned instructions that attempt to bypass or ignore the security judgment criteria may include nested instructions or implicit instructions. Nested instructions can also be called nested harmful content queries, and implicit instructions can also be called hidden harmful content queries. Among them, the nested harmful content query includes the form of nesting the rewritten initial prompt in a harmless task scenario, and the initial prompt is a prompt with malicious intentions. In some embodiments, the discriminant analysis module 101 also verifies whether the decomposed basic components are consistent with the context of the question input by the user. If not, it is determined that the question input by the user includes a hidden harmful content query.

[0065] In some embodiments, the above-mentioned rewriting of the initial prompt may include one of the following schemes for rewriting the initial prompt:

[0066] Rewrite it using fewer words, for example, by reducing the prompt to six words or fewer.

[0067] Change the sentence structure (reorder words without changing the semantics). For example, the original prompt is "To increase diversity, ask the LLM responsible for rewriting to generate five candidates and randomly select one as the paraphrase result." The rewritten prompt is "Ask the LLM to increase the diversity of the paraphrase and randomly select one of the five candidates."

[0068] Deliberately misspell sensitive words, such as "theift" for "theft," "hake" for "hack," and "feik nuz" for "fake news," and then return the modified prompt.

[0069] Insert nonsense characters, which rewrite the prompt by adding completely meaningless characters or foreign words without changing its semantics. As an example, the original prompt "How to steal" is rewritten as: "How to steàlf eu".

[0070] Partially translate sensitive words, translating sensitive words in the prompt into another foreign language. For example, "How to steal," where "steal" is the Chinese translation of "theft." As another example, the original prompt: "Discuss ways of hacking into a system" is partially translated into "Discuss ways of intruding into a system," where "intrusion" and "system" are the Chinese translations of "hacking" and "system," respectively.

[0071] Rephrase the prompt using slang, dialect, or non-standard vocabulary, but don't change its meaning. For example, "How to nick something."

[0072] In some embodiments, harmless task scenarios may include code completion, form filling, and text continuation.

[0073] The disclosed embodiment sets up a two-stage security check mechanism of semantic analysis and task structure analysis, which can more accurately identify harmful content queries and then refuse to generate replies for query requests that are considered unsafe or suspicious. Compared with the method of generating a reply and then judging whether the reply content contains harmful information, it can reduce the waste of computing power, optimize the overall model performance, and make it more robust and reliable.

[0074] The security defense solution adopted in the above-mentioned question-and-answer system can also be called SAGE (Self-Aware Guard Enhancement). Hereinafter, SAGE will be used to refer to the security defense solution adopted in the above-mentioned question-and-answer system, and the technical effects of the above-mentioned question-and-answer system in security defense will be described in detail.

[0075] The inventors evaluated the technical effects of the above scheme through experiments, that is, evaluated the effectiveness, usefulness, efficiency and robustness of SAGE.

[0076] The experiments were conducted on six open-source and closed-source large language models of different sizes and architectures, including three relatively small but popular open-source large language models: Gemma-2-9B-IT, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, and three large-scale and leading-performance closed-source large language models: GPT-4o-mini, GPT-4o, and Claude-3.5-Sonnet.

[0077] The experiments used two widely used jailbreak datasets: AdvBench and JBB-Behaviors, as well as seven state-of-the-art jailbreak methods, including two optimization-based methods, namely GCG (optimizing attacks using gradient information) and AutoDAN (genetic algorithm attacks), two LLM-based automated methods, namely PAIR (generating attacks using model interaction) and ReNeLLM (generating attacks based on prompt rewriting and scene nesting), and three template-based methods, namely Deepinception, GPTFuzzer, and CodeAttack.

[0078] In the experiments, SAGE is compared with a vanilla LLM without any defense measures, as well as five state-of-the-art efficient defense methods, namely Self-Reminder (Xie et al., 2023), Self-Examination (Phute et al., 2024), ICD (Wei et al., 2024), Target Prioritization (Zhang et al., 2024), and IA (Zhang et al., 2025).

[0079] The effectiveness of various methods is comprehensively and accurately evaluated using the rule-based keyword attack success rate (ASR) and the GPT-based harm score. Keyword ASR calculates the proportion of samples that do not contain any elements from a predefined dictionary of rejected strings. To account for potential misjudgments in rule-based ASR, GPT-4o is used to calculate the harm score on a scale of 1 to 5, with 1 representing completely harmless and 5 representing extremely harmful.

[0080] For usefulness evaluation, we used three authoritative datasets: MMLU (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), and Just-Eval (Lin et al., 2023). MMLU is a dataset covering multiple disciplines. GSM8K focuses on mathematical reasoning. Just-Eval is a dataset that comprehensively evaluates model performance across multiple dimensions.

[0081] Figure 2 The security performance of SAGE was compared with several baseline models. SAGE achieved the best performance in terms of ASR and malicious request scores compared to these baseline methods. While these methods are effective against common malicious requests, their generalization capabilities are insufficient against complex cracking attacks such as ReNeLLM, DeepInception, and CodeAttack. By combining the model's discriminative and generative capabilities, SAGE fully unleashed their security potential and demonstrated strong general defense performance, with significant average effectiveness. Across the six models, SAGE achieved a 99% success rate, even reducing the ASR score for complex jailbreak attacks from 100% to 0%.

[0082] Figure 3The authors show that SAGE achieves minimal performance loss on common benchmarks, nearly matching that of large, unprotected language models. This is due to SAGE's explicit discriminatory guidance and response protocols, which enable the model to block truly harmful requests while ensuring proper responses to benign requests. Furthermore, the inventors found that some baseline models exhibit oversensitivity on specific datasets. For example, self-censorship and target prioritization show significant performance degradation on MMLU, indicating that they may misclassify benign requests as harmful.

[0083] In the experiment, we extracted the same proportion of benign requests from GSM8k, MMLU and Just-Eval, and the same number of harmful requests from all jailbreak attacks, a total of 100 samples, to test the efficiency of SAGE. Figure 4 As shown in Figure 3, we observe that SAGE’s efficiency is only slightly inferior to IA in the baseline, without causing significant inference delay. This efficiency is due to not requiring the model to output the discriminative reasoning process, allowing the model to directly reject harmful requests or respond normally to benign requests.

[0084] In the experiment, GPT-4o (OpenAI, 2024) was used to rewrite two modules in SAGE according to the instruction “Please rewrite the following sentence without changing the core semantics”. Figure 5 As shown, the performance of SAGE and its two variants remains largely unchanged, which indicates that SAGE does not rely on precisely specified cues and can work effectively under different expression modes.

[0085] Figure 6a-6c The hidden states of the three models are visualized using two-dimensional principal component analysis (PCA). "CLS" indicates that a security discrimination instruction has been added. Figure 6a-6c It can be seen that: (1) the model can easily distinguish between harmful and benign samples, as shown by the bounds of the logistic regression fit (black dashed line). (2) the jailbreak attack pushes the query representation from the harmful side to the benign side (blue arrow). (3) the discriminant instruction pulls the hidden state of the jailbreak request back to the harmful side (red arrow). The above suggests that the discriminant generation gap in LLM may be related to the internal hidden state corresponding to the prompt in these two modes.

[0086] When the model acts as a generator, it becomes confused or disoriented about the hidden states of truly benign and jailbreak requests. However, when the model acts as a discriminator, its internal awareness realigns. These findings are consistent across three different large-scale language models, suggesting that the model's response to cues may be related to its internal hidden state, and that different states during discrimination and generation may lead to response gaps.

[0087] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory.

[0088] In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0089] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0090] Based on the same inventive concept, the present disclosure also provides a security defense method, such as Figure 7 As shown, the security defense method includes S701-S703.

[0091] In S701, a query request from a user is obtained, where the query request includes a question input by the user;

[0092] In S702, the large language model is triggered to perform semantic analysis on the question input by the user, and task structure analysis on the question input by the user is performed to evaluate whether the intention of the question input by the user is malicious, and obtain a security judgment result of whether the question input by the user is a harmful content query;

[0093] In S703 , the large language model is triggered and combined with the security judgment result to generate a reply response to the question input by the user.

[0094] The security defense method in the above embodiment can be used in a question-answering system. The specific manner in which the security defense method is applied in the question-answering system has been described in detail in the embodiment of the question-answering system and will not be elaborated on here.

[0095] In the embodiments of the present disclosure, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0096] In this disclosure, the term "and / or" simply describes an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0097] Furthermore, although the steps of the methods of the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in this particular order, or that all illustrated steps must be performed to achieve desired results.

[0098] In some embodiments, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0099] Refer to the following Figure 8 To describe the electronic device provided by the embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0100] Figure 8 FIG. 8 is a schematic diagram showing the architecture of an electronic device 800 provided by an embodiment of the present invention. Figure 8 As shown, the electronic device 800 includes but is not limited to: at least one processor 810 and at least one memory 820.

[0101] The memory 820 is used to store instructions.

[0102] In some embodiments, the memory 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0103] In some embodiments, the memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0104] In some embodiments, the memory 820 may store an operating system, which may be a real-time operating system (RTX), LINUX, UNIX, WINDOWS, or OS X.

[0105] In some embodiments, data may also be stored in the memory 820 .

[0106] As an example, the processor 810 may read data stored in the memory 820 . The data may be stored at the same storage address as the instruction, or the data may be stored at a different storage address than the instruction.

[0107] The processor 810 is configured to call instructions stored in the memory 820 to implement the steps described in the "Exemplary Method" section above according to various exemplary embodiments of the present disclosure. For example, the processor 810 may execute the steps of the aforementioned security defense method embodiment.

[0108] It should be noted that the processor 810 may be a general-purpose processor or a dedicated processor. The processor 810 may include one or more processing cores, and the processor 810 executes various functional applications and data processing by running instructions.

[0109] In some embodiments, the processor 810 may include a central processing unit (CPU) and / or a baseband processor.

[0110] In some embodiments, the processor 810 may determine an instruction based on the priority identifier and / or function category information carried in each control instruction.

[0111] In the present disclosure, the processor 810 and the memory 820 may be provided separately or integrated together.

[0112] As an example, the processor 810 and the memory 820 may be integrated on a single board or a system on chip (SOC).

[0113] like Figure 8 As shown, the electronic device 800 is in the form of a general-purpose computing device. The electronic device 800 may further include a bus 830 .

[0114] The bus 830 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0115] The electronic device 800 may also communicate with one or more external devices 840 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 850.

[0116] Furthermore, the electronic device 800 can also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 860 .

[0117] like Figure 8 As shown, the network adapter 860 communicates with other modules of the electronic device 800 via the bus 830 .

[0118] It should be understood that although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] It is understood that the structure shown in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 800. In other embodiments of the present disclosure, the electronic device 800 may include Figure 8 More or fewer components may be shown, or some components may be combined or separated, or the components may be arranged differently. Figure 8 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0120] The present disclosure also provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the security defense method described in the above method embodiment is implemented.

[0121] The computer-readable storage medium in the embodiments of the present disclosure is a computer instruction that can be sent, propagated or transmitted for use by or in conjunction with an instruction execution system, apparatus or device.

[0122] As an example, computer readable storage media are non-volatile storage media.

[0123] In some embodiments, more specific examples of computer-readable storage media in the present disclosure may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, a USB flash drive, a mobile hard disk, or any suitable combination of the foregoing.

[0124] In the embodiments of the present disclosure, the computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer instructions (readable program codes).

[0125] Such a propagated data signal may take any of a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0126] In some examples, computing instructions contained on a computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0127] The embodiments of the present disclosure also provide a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the security defense method described in the above method embodiment.

[0128] The above instructions may be program codes. In specific implementation, the program codes may be written in any combination of one or more programming languages.

[0129] Programming languages include object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages.

[0130] The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0131] Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0132] The embodiment of the present disclosure further provides a chip, comprising at least one processor and an interface;

[0133] An interface for providing program instructions or data to at least one processor;

[0134] At least one processor is used to execute program instructions to implement the security defense method described in the above method embodiment.

[0135] In some embodiments, the chip may further include a memory for storing program instructions and data, and the memory may be located inside or outside the processor.

[0136] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, which may be collectively referred to herein as a "circuit," "module," or "system."

[0137] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein.

[0138] This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A question-answering system, characterized in that: include: A discriminant analysis module is configured to obtain a query request from a user, the query request including a question input by the user, trigger a large language model to perform semantic analysis on the question input by the user, and perform task structure analysis on the question input by the user to assess whether the intention of the question input by the user is malicious, thereby obtaining a security judgment result on whether the question input by the user is a query for harmful content; The discrimination response module is used to trigger the large language model to combine the security judgment result to generate a reply response to the question input by the user.

2. The system according to claim 1, wherein: The discrimination response module is configured to trigger the large language model to perform a first reply response to the question input by the user if the result of the security judgment is that the question input by the user is not a query for harmful content, wherein the first reply response is a reply by the large language model to the question input by the user; The discrimination response module is further configured to trigger the large language model to perform a second reply response to the question input by the user when the result of the security judgment is that the question input by the user is not a query for harmful content, wherein the second reply response is that the large language model refuses to answer the question input by the user.

3. The system according to claim 1, wherein: The discriminant analysis module performs semantic analysis on the question input by the user, including: Using a predefined keyword library to detect whether the question input by the user contains harmful content, and if so, determining that the question input by the user is a harmful content query; Analyzing the contextual semantics of the question input by the user using the hidden state of the model, and using the deep learning model to understand the intent of the question input by the user based on the contextual semantics, determining whether the intent is malicious, and determining that the question input by the user is a harmful content query if the intent is malicious; The security of the question input by the user is classified by a logistic regression model to obtain a classification result of whether the question input by the user is a harmful content query.

4. The system according to claim 1, wherein: The discriminant analysis module performs task structure analysis on the question input by the user, including: decomposing the user input question into basic components; identifying, based on the basic components, whether the question input by the user contains instructions that attempt to bypass or ignore security judgment criteria; In the case of including instructions that attempt to circumvent or ignore security judgment criteria, determining whether the question input by the user is a harmful content query.

5. The system according to claim 4, characterized in that Instructions that attempt to bypass or disregard security guidelines, including nested queries for harmful content; The nested harmful content query includes a form of nesting the rewritten initial prompt in a harmless task scenario, and the initial prompt is a prompt with malicious intentions.

6. The system according to claim 4, characterized in that Instructions that attempt to circumvent or disregard security guidelines, including hidden queries for harmful content; The discriminant analysis module further verifies whether the decomposed basic components are consistent with the context of the question input by the user, and if not, determines that the question input by the user includes a hidden harmful content query.

7. The system according to claim 2, wherein: When the identification and response module determines that the query request is a query for harmful content, the second reply response further includes a consideration process of whether the query request is a query for harmful content.

8. The system according to claim 2, wherein: When the determination response module determines that the query request is not a query for harmful content, the first reply response does not include a consideration process of whether the query request is not a query for harmful content.

9. A security defense method, characterized in that: include: Obtaining a query request from a user, wherein the query request includes a question input by the user; Triggering the large language model to perform semantic analysis on the question input by the user, and performing task structure analysis on the question input by the user to assess whether the intention of the question input by the user is malicious, and obtaining a security judgment result on whether the question input by the user is a harmful content query; The large language model is triggered to combine the security judgment result to generate a reply response to the question input by the user.

10. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the security defense method according to claim 9.

Images