Method and device for identifying malicious query intent of large model
By analyzing the logical coherence of query sequences and average probing depth in large-scale model interactions, and utilizing medical entity relationship networks and semantic similarity, malicious query intent in large-scale models can be identified. This solves the problem of identifying malicious query intent in existing technologies and achieves a more efficient protection effect.
Patent Information
- Application Number
- CN202610573469.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies struggle to effectively identify malicious query intent in large models, especially when attackers mimic real user interaction patterns, conventional detection methods fail to recognize their malicious intent.
By acquiring the query sequences of target users interacting with the large model, we analyze the logical coherence and average follow-up depth of the query statements. Using medical entity relationship networks and semantic similarity calculations, we divide topic clusters and calculate multi-dimensional indicator values to determine whether the query sequences have malicious query intentions.
It significantly improves the effectiveness of identifying malicious query intent in large models, reduces the false negative rate and false positive rate, and enhances the protection against new attack strategies.
Smart Images
Figure CN122451889A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the field of artificial intelligence, and more particularly to methods and apparatus for identifying malicious query intent for large models. Background Technology
[0002] With the widespread application of large-scale models in vertical industries, protecting their core knowledge has become a critical challenge. Currently, large-scale models face intense and continuous external probing pressure. Attackers attempt to reverse engineer or replicate the model's expertise through queries targeting the model. These queries often harbor malicious intent, such as attempting knowledge distillation through knowledge traversal, knowledge base sampling, or anthropomorphic dialogue. Model knowledge distillation is an AI model compression technique based on a teacher-student model. By transferring knowledge from a large teacher model to a small student model, it aims to reduce deployment costs and improve inference efficiency.
[0003] In existing technologies, conventional detection methods primarily rely on surface-level characteristics such as request frequency, Internet Protocol (IP) address clustering, and network attack features of query statements to formulate detection rules. Malicious query intent targeting the large model is identified based on whether the query conforms to these rules. However, to circumvent conventional detection, attackers deliberately mimic the interaction patterns of real users, making their queries sent to the large model highly similar to those of normal users in terms of surface-level characteristics. In such cases, conventional detection methods fail to identify malicious query intent.
[0004] Therefore, an improved solution is needed to enhance the effectiveness of identifying malicious query intent for large models. Summary of the Invention
[0005] This specification describes one or more embodiments of a method and apparatus for identifying malicious query intent for large models, which can improve the effectiveness of identifying malicious query intent for large models.
[0006] Firstly, a method for identifying malicious query intent in large models is provided, including:
[0007] Obtain the query sequence of the target user's interaction with the large model, which includes multiple query statements;
[0008] For each query statement, the jump degree index of a single statement is determined based on its first similarity with adjacent query statements and its maximum similarity with the query sequence; based on the jump degree index of each query statement, the first index value reflecting the logical coherence of the query sequence is determined.
[0009] The query sequence is divided into several topic clusters, and each topic cluster contains several consecutive query statements; at least based on the cluster size of each topic cluster, a second indicator value reflecting the average follow-up depth of the query sequence is determined.
[0010] Based on the first indicator value and the second indicator value, it is determined whether the query sequence has a malicious query intent against the large model.
[0011] In one possible implementation, determining the jump index of a single query statement based on its first similarity to adjacent query statements and its maximum similarity to the query sequence includes:
[0012] The maximum similarity between the current query statement and each historical query statement in the query sequence is calculated and used as the maximum similarity.
[0013] Calculate the ratio of the first similarity to the maximum similarity, and use this ratio as the jump index of the current query statement.
[0014] In one possible implementation, determining the first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement includes:
[0015] Count the first number of query statements in the query sequence whose jump index is lower than the first threshold.
[0016] Calculate the proportion of the first quantity to the total number of query statements in the query sequence, and use the proportion as the first indicator value.
[0017] In one possible implementation, determining the first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement includes:
[0018] Calculate the average value of each jump index in the query sequence, and use the average value as the first index value.
[0019] In one possible implementation, the first similarity is determined as follows:
[0020] Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities;
[0021] Based on the medical entity relationship network, the medical entities involved in the current query statement and its adjacent query statements are extracted to form a first entity set and a second entity set;
[0022] Calculate the ratio of the number of elements in the intersection of the first entity set and the second entity set to the number of elements in the union set, and use this ratio as the first similarity.
[0023] In one possible implementation, the determination of the jump rate index for a single sentence includes:
[0024] Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities;
[0025] In the medical entity relationship network, the medical entity involved in the current query statement is taken as the starting node and the medical entity involved in the adjacent query statement is taken as the ending node. A graph traversal search is performed to determine whether there is a direct or indirect connection path from the starting node to the ending node.
[0026] Based on the judgment results, the ratio of the first similarity to the maximum similarity is adjusted to obtain the jump index of a single sentence.
[0027] In one possible implementation, dividing the query sequence into several topic clusters includes:
[0028] For any two adjacent query statements in the query sequence, determine whether the similarity between the two query statements is less than a preset similarity threshold;
[0029] If the similarity between two query statements is greater than or equal to the similarity threshold, the two query statements will be grouped into the same topic cluster.
[0030] If the similarity between two query statements is less than the similarity threshold, the two query statements will be classified into different topic clusters.
[0031] In one possible implementation, determining a second metric reflecting the average follow-up depth of the query sequence, at least based on the cluster size of each topic cluster, includes:
[0032] For each topic cluster, evaluate whether the queries within the cluster present a deepening pattern from general concepts to specific details;
[0033] Based on the cluster size and the deepening pattern of query statements within the cluster, a second index value reflecting the average probing depth of the query sequence is calculated.
[0034] In one possible implementation, it also includes:
[0035] Traverse the query sequence and extract the original medical terms from each query statement to obtain an initial term set;
[0036] Map the original medical terms in the initial terminology set to standard medical terms, and merge duplicate standard medical terms to construct a standard terminology set;
[0037] Calculate the number of standard medical terms in the set of standard terms, and the information entropy of the frequency of occurrence of each standard medical term;
[0038] The number of standard medical terms is weighted and summed with the information entropy to obtain a third index value that reflects the topic dispersion of the query sequence.
[0039] Determining whether the query sequence has a malicious query intent targeting the large model includes:
[0040] Based on the first indicator value, the second indicator value, and the third indicator value, it is determined whether the query sequence has a malicious query intent.
[0041] Furthermore, the extraction of raw medical terms from each query statement includes:
[0042] Each query statement is matched with a pre-built medical terminology dictionary using regular expressions to obtain the original medical terms in each query statement. The medical terminology dictionary includes medical terms for diseases, symptoms, drugs, and examination items.
[0043] Secondly, a malicious query intent identification device for large models is provided, including:
[0044] The acquisition unit is used to acquire the query sequence of the target user's interaction with the large model, which includes multiple query statements;
[0045] The first determining unit is used to determine the jump degree index of a single query statement based on its first similarity with adjacent query statements and its maximum similarity with the query sequence for each query statement obtained by the obtaining unit; and to determine a first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement.
[0046] The second determining unit is used to divide the query sequence obtained by the obtaining unit into several topic clusters, each topic cluster containing several consecutive query statements; and to determine a second index value reflecting the average follow-up depth of the query sequence based at least on the cluster size of each topic cluster.
[0047] The identification unit is used to determine whether the query sequence has a malicious query intent against the large model based on the first indicator value obtained by the first determining unit and the second indicator value obtained by the second determining unit.
[0048] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.
[0049] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.
[0050] The method and apparatus provided in the embodiments of this specification first obtain a query sequence of interaction between a target user and a large model, including multiple query statements; then, for each query statement, a jump index is determined based on its first similarity with adjacent query statements and its maximum similarity with the query sequence; based on the jump index of each query statement, a first index value reflecting the logical coherence of the query sequence is determined; next, the query sequence is divided into several topic clusters, each topic cluster containing several consecutive query statements; at least based on the cluster size of each topic cluster, a second index value reflecting the average follow-up depth of the query sequence is determined; finally, based on the first index value and the second index value, it is determined whether the query sequence has a malicious query intent against the large model. As can be seen from the above, the embodiments of this specification, by determining a first indicator value reflecting the logical coherence of the query sequence, can distinguish between the natural topic switching of normal users and the illogical jumps of malicious users, thus enabling the identification of attackers even when they are disguised as normal users, significantly reducing the false negative rate. By determining a second indicator value reflecting the average probing depth of the query sequence, it can accurately distinguish between deep probing by normal users and shallow traversal by malicious users, reducing the false positive rate. The joint determination of the first and second indicator values forms a complementary detection loop, which can only determine malicious query intent when a user exhibits both logically irregular jumps and shallow probing. This multi-dimensional cross-validation method greatly enhances the generalization protection capability against new attack strategies, prevents attackers from bypassing detection by disguising query statements, and improves the effectiveness of identifying malicious query intent for large models. Attached Figure Description
[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;
[0053] Figure 2 A flowchart illustrating a method for identifying malicious query intent for a large model according to one embodiment is shown.
[0054] Figure 3 A schematic diagram illustrating a method for determining a first index value according to one embodiment is shown;
[0055] Figure 4 A schematic diagram illustrating a method for determining a first index value according to another embodiment is shown;
[0056] Figure 5 This diagram illustrates a multi-dimensional determination method according to one embodiment.
[0057] Figure 6 A schematic block diagram of a malicious query intent identification device for large models is shown according to one embodiment. Detailed Implementation
[0058] The solution provided in this specification will now be described with reference to the accompanying drawings.
[0059] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves identifying malicious query intent against a large model. Specifically, the large model can be a Large Language Model (LLM), which refers to a deep learning model trained on massive amounts of data with a huge number of parameters, possessing powerful natural language understanding, generation, and logical reasoning capabilities. Taking the medical vertical field as an example, after fine-tuning with professional corpora, this model can simulate the role of a doctor, providing accurate disease diagnosis suggestions, symptom analysis, and treatment plan consultations, exhibiting high professional capabilities and knowledge density. The aforementioned malicious query intent may include, but is not limited to, the intent of model knowledge distillation. For example, the aforementioned malicious query intent also includes: resource exhaustion attack intent, i.e., queries made by attackers to consume the model's computing resources, causing service unavailability; and hallucination inducement intent, i.e., attackers intentionally guide the model to fabricate facts through queries, causing it to experience severe hallucinations, thereby spreading false information, damaging the model's reputation, or misleading the public. Many other possible malicious query intents exist, which will not be listed here.
[0060] Reference Figure 1In a user-model interaction, the user inputs a query, and the model outputs a response. Typically, this interaction involves multiple rounds, with the user inputting a query in each round. These queries are arranged chronologically to form a query sequence. For example, in a medical consultation scenario, a query sequence with malicious intent might disguise itself as a standard inquiry about a series of common symptoms, making it difficult for conventional detection methods to identify its malicious intent. For instance, a query sequence might include the following queries: Query 1 "Q1. What are the symptoms of a cold?", Query 2 "Q2. What are the symptoms of a fever?", Query 3 "Q3. What are the symptoms of a stomachache?", Query 4 "Q4. What are the symptoms of a headache?", and Query 5 "Q5. What are the symptoms of a cough?". These queries are highly similar to normal user requests in their surface features, making it impossible to identify their malicious intent based on surface characteristics.
[0061] The embodiments in this specification analyze multi-dimensional deep features, including the logical coherence of the query sequence and the average probing depth, in order to improve the effectiveness of identifying malicious query intent for large models, thereby more accurately blocking query behaviors with malicious intent and protecting the core knowledge and service security of the model.
[0062] Figure 2 This diagram illustrates a method for identifying malicious query intent for a large model according to one embodiment. The method can be based on... Figure 1 The implementation scenario is shown. For example... Figure 2 As shown, the method for identifying malicious query intent against a large model in this embodiment includes the following steps: Step 21, obtaining the query sequence of the target user's interaction with the large model, including multiple query statements; Step 22, for each query statement, determining the jump degree index of a single sentence based on its first similarity with adjacent query statements and its maximum similarity with the query sequence; determining a first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement; Step 23, dividing the query sequence into several topic clusters, each topic cluster containing several consecutive query statements; determining a second index value reflecting the average follow-up depth of the query sequence based at least on the cluster size of each topic cluster; Step 24, determining whether the query sequence has a malicious query intent against the large model based on the first index value and the second index value. The specific execution method of each of the above steps is described below.
[0063] First, in step 21, the query sequence of the target user's interaction with the large model is obtained, which includes multiple query statements. It can be understood that these multiple query statements are entered sequentially during the multi-turn dialogue between the target user and the large model, and are arranged in chronological order.
[0064] Among them, the target users mentioned above may be normal users or malicious users. The identification of whether the query sequence has malicious query intent is to identify whether the target user is a malicious user.
[0065] In the embodiments described in this specification, individual query statements are typically highly similar to normal user requests in terms of surface features. Therefore, by extracting deeper features from the query sequence, it is easier to identify the user's malicious query intent. To accommodate situations where the surface features of individual query statements have obvious attack characteristics, conventional detection methods can be used first to determine whether the user has malicious query intent based on the surface features of individual query statements. When conventional detection methods fail to identify the malicious intent, then identification of the query sequence can be performed.
[0066] Then, in step 22, for each query statement, a jump index is determined based on its first similarity to adjacent query statements and its maximum similarity to the query sequence; based on the jump index of each query statement, a first index value reflecting the logical coherence of the query sequence is determined. It is understandable that by analyzing the logical coherence of the query sequence, abrupt jumps in medical topics can be identified. The worse the logical coherence of the query sequence, the more likely it is to have malicious query intent targeting the larger model.
[0067] In one example, determining the jump index of a single query statement based on its first similarity with adjacent query statements and its maximum similarity with the query sequence includes:
[0068] The maximum similarity between the current query statement and each historical query statement in the query sequence is calculated and used as the maximum similarity.
[0069] Calculate the ratio of the first similarity to the maximum similarity, and use this ratio as the jump index of the current query statement.
[0070] In this example, the aforementioned similarity specifically refers to semantic similarity. The first similarity reflects the semantic relevance of two consecutive queries, while the maximum similarity reflects the strongest semantic association between the current query and the entire historical dialogue context. Calculating the ratio of the first similarity to the maximum similarity reveals that when a user naturally continues the current topic, both the first and maximum similarities are high, with the ratio approaching 1. When a user suddenly jumps to a completely new topic, the first similarity is very low, but the maximum similarity may still be high because the new topic might be related to an earlier topic, resulting in a ratio much less than 1. By calculating this ratio, this change in relative relationship can be readily detected, effectively distinguishing between natural semantic extensions and malicious, illogical jumps.
[0071] In one example, determining the first indicator value reflecting the logical coherence of the query sequence based on the jump index of each query statement includes:
[0072] Count the first number of query statements in the query sequence whose jump index is lower than the first threshold.
[0073] Calculate the proportion of the first quantity to the total number of query statements in the query sequence, and use the proportion as the first indicator value.
[0074] In this example, besides the first query statement in the query sequence, the similarity between each query statement and the previous query statement is calculated. The total number of query statement pairs can be calculated by subtracting 1 from the total number of query statements. By statistically analyzing the proportion of query statements with a jump index below a first threshold, discrete single-sentence logical judgments are transformed into a global logical coherence index, quantifying the overall logical coherence of user dialogue. It can effectively identify malicious traversal behavior that appears fluent only locally but contains numerous abrupt topic changes. This statistical proportion-based judgment method not only avoids single-point misjudgments but also provides an intuitive and quantifiable threshold basis, thereby accurately distinguishing between naturally divergent normal consultations and malicious sampling without logical jumps.
[0075] Figure 3 A schematic diagram illustrating a method for determining a first index value according to one embodiment is shown. (Refer to...) Figure 3 The query sequence consists of five query statements, denoted as Q1, Q2, Q3, Q4, and Q5. The first query statement, Q1, does not have a jump index. Subsequent query statements have their jump indices calculated: Q2 has a jump index of 0.8, Q3 0.1, Q4 0.3, and Q5 0.6. A first threshold of 0.5 is set. When the calculated jump index of the current query statement is less than this first threshold, the query statement is considered a logical jump point, indicating that the user's topic switching deviates from normal conversation logic. The number of all query statements identified as logical jump points in the query sequence (i.e., the first number) is counted as 2. The ratio of this first number to the total number of query statements in the query sequence (4) is calculated, and this ratio of 0.5 is directly used as the first index value. This ratio can also be called the logical jump ratio; the higher the value, the worse the logical coherence of the query sequence, and the more likely it is to have malicious intent.
[0076] In one example, determining the first indicator value reflecting the logical coherence of the query sequence based on the jump index of each query statement includes:
[0077] Calculate the average value of each jump index in the query sequence, and use the average value as the first index value.
[0078] In this example, by calculating the average value of the jump index as the first indicator, the overall logical coherence of the query sequence can be quantified globally. Compared to counting the aforementioned first quantity, the average value is more sensitive to outliers and can more delicately reflect the degree of logical discontinuity exhibited by the attacker throughout the session. This approach, through continuous magnitude features, can more sensitively capture those hidden attack intentions that, although they have not completely exceeded the single sentence threshold, exhibit overall traversal characteristics.
[0079] Figure 4 A schematic diagram illustrating a method for determining a first index value according to another embodiment is shown. (Refer to...) Figure 4 The query sequence consists of five query statements, denoted as Q1, Q2, Q3, Q4, and Q5. The first query statement, Q1, does not have a jump index. Subsequent query statements have their jump indices calculated: Q2 has a jump index of 0.8, Q3 has 0.1, Q4 has 0.3, and Q5 has 0.6. The average value of each jump index is calculated, and this average of 0.45 is used as the first index value. This average value, also known as the average logical deviation, reflects the overall coherence of the query sequence. The lower the value, the worse the logical coherence of the query sequence, and the more likely it is to indicate malicious intent.
[0080] It should be noted that the aforementioned logical jump ratio and average logical deviation can also be combined to determine the first indicator value. For example, a weighted fusion approach can be used, specifically, the first indicator value = α × logical jump ratio + β × average logical deviation, where α and β are preset weights, to simultaneously quantify the frequency and severity of abnormal switching; or a product-coupled approach can be used, specifically, the first indicator value = logical jump ratio × average logical deviation. This combination can more sensitively detect extreme malicious behavior that involves both frequent jumps and significant logical breaks. There are many possible combinations, which will not be elaborated here.
[0081] In one example, the first similarity is determined as follows:
[0082] Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities;
[0083] Based on the medical entity relationship network, the medical entities involved in the current query statement and its adjacent query statements are extracted to form a first entity set and a second entity set;
[0084] Calculate the ratio of the number of elements in the intersection of the first entity set and the second entity set to the number of elements in the union set, and use this ratio as the first similarity.
[0085] In this example, by introducing a medical entity relationship network, traditional text-based word matching is upgraded to semantic association computation based on domain knowledge. Its core effect lies in its ability to accurately identify logically related medical dialogues. Specifically, it utilizes the intersection and union ratio of entity sets to capture the overlap of medical entities in two query statements, using this ratio as the first similarity score. This effectively distinguishes between the substitution of professional terminology in normal medical consultations and illogical topic jumps in malicious queries. This mechanism significantly improves the accuracy of judging logical coherence in complex medical scenarios, avoids misjudgments caused by synonym differences, and enhances the ability to identify covert attacks.
[0086] The aforementioned medical entities may include entities such as diseases, symptoms, and drugs.
[0087] Furthermore, it should be noted that there are many ways to determine the first similarity, including but not limited to the methods described in the previous examples. For instance, a pre-trained word vector model can be used to map the words in the query statement into vectors in a high-dimensional vector space; all word vectors in the query statement can be aggregated (e.g., averaged) to generate the semantic vector of the query statement; and the cosine similarity between the semantic vectors of two adjacent query statements can be calculated as the first similarity.
[0088] In one example, the metric for determining the jumpiness of a single sentence includes:
[0089] Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities;
[0090] In the medical entity relationship network, the medical entity involved in the current query statement is taken as the starting node and the medical entity involved in the adjacent query statement is taken as the ending node. A graph traversal search is performed to determine whether there is a direct or indirect connection path from the starting node to the ending node.
[0091] Based on the judgment results, the ratio of the first similarity to the maximum similarity is adjusted to obtain the jump index of a single sentence.
[0092] In this example, by introducing graph traversal search of medical entity relationship networks, simple text similarity calculation is upgraded to deep logic verification based on medical knowledge graphs. This can accurately identify complex scenarios that are "superficially unrelated but intrinsically related" or "superficially related but logically disjointed." Specifically, even if the current statement and adjacent statements have low lexical overlap, if the entities involved in both have direct or indirect paths in the medical network (such as the association from "cold" to "fever reducer"), it can be judged as logically coherent and the jump level can be corrected. Conversely, if the words seem similar but there is no medical connection between the entities (such as "fever" and "hypertension"), it is judged as a logical jump. This mechanism effectively overcomes the limitations of simply relying on word vector matching, significantly improves the ability to understand the implicit logical relationships in medical professional dialogues, and thus more accurately identifies malicious queries that use synonyms to disguise themselves or perform illogical knowledge traversal.
[0093] In the embodiments of this specification, the relevance of adjacent query statements in a session can be analyzed from multiple dimensions: one dimension is entity overlap, which calculates the proportion of the same medical entities in the two query statements; another dimension is relational connectivity, which queries the medical entity relationship network to determine whether there is a direct or indirect known medical relationship between the entities in the two query statements; and another dimension is intent coherence, which analyzes whether the intent of the two query statements constitutes a reasonable medical dialogue sequence, such as "symptom description → diagnosis inquiry → treatment consultation".
[0094] Next, in step 23, the query sequence is divided into several topic clusters, each containing several consecutive query statements; at least based on the cluster size of each topic cluster, a second indicator value reflecting the average follow-up depth of the query sequence is determined. It is understood that a lower average follow-up depth indicates a simpler user query, which is more likely to be a simple knowledge traversal and may indicate malicious query intent.
[0095] In one example, dividing the query sequence into several topic clusters includes:
[0096] For any two adjacent query statements in the query sequence, determine whether the similarity between the two query statements is less than a preset similarity threshold;
[0097] If the similarity between two query statements is greater than or equal to the similarity threshold, the two query statements will be grouped into the same topic cluster.
[0098] If the similarity between two query statements is less than the similarity threshold, the two query statements will be classified into different topic clusters.
[0099] In this example, by setting a similarity threshold to divide topic clusters, automated clustering from continuous dialogue streams to discrete logical units is achieved, enabling precise quantification of the depth of user follow-up questions. Specifically, when adjacent query statements are semantically highly related, they are grouped into the same topic cluster, simulating the behavior of normal users engaging in multiple rounds of in-depth discussion around a single medical topic (such as "cold symptoms"). Once the semantic association breaks, cluster splitting is triggered, indicating a topic switch. This mechanism transforms the abstract "follow-up questioning behavior" into a statistically measurable "cluster size" metric, effectively distinguishing between the deep vertical mining by normal users and the shallow traversal by malicious attackers, providing a clear, objective, and structured basis for subsequent determination of malicious query intent.
[0100] In one example, determining the second metric reflecting the average follow-up depth of the query sequence, at least based on the cluster size of each topic cluster, includes:
[0101] For each topic cluster, evaluate whether the queries within the cluster present a deepening pattern from general concepts to specific details;
[0102] Based on the cluster size and the deepening pattern of query statements within the cluster, a second index value reflecting the average probing depth of the query sequence is calculated.
[0103] In this example, by introducing a depth-of-concept assessment, the simple statistical dimension is upgraded to a dual verification of the quality and logical depth of the probing questions, accurately distinguishing between purposeful in-depth questioning and meaningless mechanical piling up. Specifically, normal users' medical consultations often follow a progressive logic of "symptom description → etiology analysis → treatment plan," and high scores are awarded after identifying this depth-of-concept pattern. Malicious attackers, even with numerous statements within a topic cluster, will be judged as having low depth if they lack logical progression, such as repeatedly asking the same facts or randomly listing details. This mechanism effectively prevents attackers from masquerading as in-depth probing by simply repeating the same topic, ensuring that the second indicator value truly reflects the rationality and professionalism of the user's intent, thereby significantly improving the accuracy of identifying concealed malicious query behavior.
[0104] For example, lightweight semantic similarity calculation methods (such as the Jaccard similarity algorithm) can be used to calculate the semantic similarity between adjacent query statements. The similarity value ranges from 0 to 1, with higher values indicating greater semantic similarity between the two queries. A similarity threshold t (e.g., 0.7) is set. When the similarity between adjacent queries is greater than or equal to t, they are considered to belong to the same topic cluster; when the similarity is less than t, they are considered the boundary between two different topic clusters. Based on the identified cluster boundaries, the entire query sequence is divided into several consecutive topic clusters. Each topic cluster contains multiple semantically related queries, representing a series of follow-up questions from the user on a specific topic. For each medical question-and-answer topic cluster, the depth is evaluated from two dimensions: cluster size (calculated based on the number of queries contained in a single topic cluster) and follow-up question quality (whether the questions within the cluster exhibit a deepening pattern from general to specific, from phenomenon to cause). Combining these two dimensions, the average depth of the user's follow-up questions is calculated.
[0105] The assessment of the deepening model can, but is not limited to, the following methods: First, based on the medical knowledge graph, a predefined abstract hierarchy of entities is established, such as "disease" as the high-level, "specific symptoms / medications" as the mid-level, and "dosage / contraindications" as the low-level. Then, statements within the cluster are scanned, and the sequence of changes in the entity hierarchy is statistically analyzed. If the sequence shows a unidirectional increasing trend of "high-level → mid-level → low-level," such as from "cold" to "fever" and then to "ibuprofen dosage," it is determined to conform to the deepening model. Conversely, if the hierarchy jumps repeatedly or stagnates, such as repeatedly asking for the names of different diseases, it is determined to have no deepening. This method only requires simple hierarchy mapping and sequence direction judgment, without the need for complex models, to efficiently identify logical depth.
[0106] Finally, in step 24, based on the first and second indicator values, it is determined whether the query sequence has a malicious query intent targeting the large model. It is understood that the first indicator value reflects the logical coherence of the query sequence, and the second indicator value reflects the average probing depth of the query sequence. Combining these two indicator values can more accurately identify malicious query intent.
[0107] In one example, it also includes:
[0108] Traverse the query sequence and extract the original medical terms from each query statement to obtain an initial term set;
[0109] Map the original medical terms in the initial terminology set to standard medical terms, and merge duplicate standard medical terms to construct a standard terminology set;
[0110] Calculate the number of standard medical terms in the set of standard terms, and the information entropy of the frequency of occurrence of each standard medical term;
[0111] The number of standard medical terms is weighted and summed with the information entropy to obtain a third index value that reflects the topic dispersion of the query sequence.
[0112] Determining whether the query sequence has a malicious query intent targeting the large model includes:
[0113] Based on the first indicator value, the second indicator value, and the third indicator value, it is determined whether the query sequence has a malicious query intent.
[0114] In this example, a three-dimensional defense system of "logical coherence + inquiry depth + topic breadth" is constructed by introducing a third indicator reflecting the topic dispersion of the query sequence. These three dimensions correspond to the "logical jumps," "shallow questions," and "breadth traversal" characteristics of malicious query intent, forming a multi-faceted and complementary detection system that accurately identifies malicious query intent targeting large models. Specifically, normal medical consultations typically revolve around a few core diseases, with a small number of terms and low information entropy. However, malicious attackers, in order to steal knowledge, deliberately traverse a large number of different categories of medical entities, such as continuously asking about unrelated diseases like colds, diabetes, and heart disease, resulting in a surge in the number of standard terms with a uniform distribution and high information entropy. By calculating the weighted value of the number of terms and information entropy, this disordered jump characteristic across topics can be keenly captured. In addition to logic and depth, this further blocks the attack path of using topic switching to disguise oneself as a normal user, significantly improving the overall robustness of the detection.
[0115] Figure 5 A schematic diagram illustrating a multi-dimensional determination method according to one embodiment is shown. (Refer to...) Figure 5 This embodiment constructs a multi-dimensional collaborative malicious query intent determination framework. First, by calculating the first indicator value L, the second indicator value D, and the third indicator value T, the feature performance of the query sequence in three dimensions—logical coherence, probing depth, and topic dispersion—is independently quantified. Then, these three indicators are weighted and fused using preset weight coefficients W1, W2, and W3 to generate a comprehensive score R = W1×L + W2×D + W3×T. This scoring mechanism effectively overcomes the limitations of single-dimensional detection, simultaneously capturing composite attack features such as "logical jumps," "shallow questions," and "breadth traversal." Based on the threshold comparison of the score R, it accurately determines whether a user is executing malicious query intents such as knowledge distillation targeting a large model. Furthermore, the weight coefficients can be dynamically adjusted according to actual business scenarios. For example, when defending against highly concealed attacks, the weight of the first indicator value L can be increased to enhance sensitivity to subtle semantic breaks, ensuring a balance between high recall and low false positive rates under different attack strategies, achieving adaptive security protection.
[0116] Furthermore, the extraction of raw medical terms from each query statement includes:
[0117] Each query statement is matched with a pre-built medical terminology dictionary using regular expressions to obtain the original medical terms in each query statement. The medical terminology dictionary includes medical terms for diseases, symptoms, drugs, and examination items.
[0118] In this example, by combining regular expression matching with a pre-built medical terminology dictionary, high accuracy and recall of medical entity extraction are achieved. It can accurately filter noise interference in natural language, ensuring that all key medical information is captured completely and systematically from the user's unstructured query. This not only provides a reliable data foundation for subsequent standard terminology mapping and topic dispersion calculation, but also effectively avoids entity omissions caused by differences in synonym expression or colloquial omissions, thus ensuring the reliability of malicious query intent identification in complex medical scenarios.
[0119] The process of determining the aforementioned third indicator value is explained in stages below:
[0120] The first stage involves obtaining an initial terminology set. A lightweight medical terminology dictionary is predefined, covering core categories such as diseases, symptoms, medications, and examination items. Upon receiving the user's query sequence, each query is iterated through, and regular expression matching technology is used to extract all original medical terms that match the dictionary definition from the unstructured natural language text, forming the initial terminology set.
[0121] The second stage involves constructing a standard terminology set. The original medical terms in the initial terminology set are mapped to standard medical terms in a pre-defined standard medical terminology database to eliminate ambiguity caused by synonyms, near-synonyms, and colloquial expressions. For example, "stomach ache" and "abdominal discomfort" are uniformly mapped to the standard medical term "abdominal pain." Subsequently, the mapped standard medical terms are merged, and duplicates are removed, thus constructing a standard terminology set that reflects the overall scope of the conversation.
[0122] In the third stage, based on the constructed set of standard terms, two core dimensional indicators are calculated: Total number of unique terms: This represents the total number of elements in the set of standard terms, i.e., the scale of non-repeating medical entities involved in the conversation; Distribution information entropy: This calculates the distribution of the frequency of each standard medical term in the conversation and uses the Shannon entropy formula to calculate its information entropy value. Malicious traversal behavior typically manifests as uniform sampling of a large number of different topics, resulting in a high distribution entropy value; while normal consultations tend to focus on a few relevant entities, resulting in a relatively low entropy value.
[0123] In the fourth stage, the total number of unique terms mentioned above is weighted and summed with the distribution information entropy to obtain a third indicator value reflecting the topic dispersion of the query sequence. The higher this indicator value, the more dispersed and disjointed the topics of the user's questions are, and the more likely it is to be judged as a malicious query intent.
[0124] For example, a user inputs a query sequence of ["What to do if I have a cold", "Should I take antipyretics for a fever", "What might cause a stomachache", "Can a cold cause a cough"]. For this query sequence, the original terms are first extracted as {cold}, {fever, antipyretics}, {stomachache}, and {cold, cough}. Then, "stomachache" is mapped to "abdominal pain". "Cold" is treated as the same standard term in different statements; after merging and deduplication, the standard term set {cold, fever, antipyretics, abdominal pain, cough} is obtained. Next, the total number of unique terms is counted as 5; the frequency of each term is calculated (cold: 2, fever: 1, antipyretics: 1, abdominal pain: 1, cough: 1), and then the distribution information entropy is calculated. Finally, the total number of unique terms and the distribution information entropy are weighted according to preset weights to generate the final topic dispersion score. In this example, because it involves multiple unrelated diseases and symptoms with a relatively uniform distribution, its topic dispersion score will be at a high level, indicating a potential risk of malicious traversal.
[0125] The method provided in the embodiments of this specification first obtains the query sequence of the target user's interaction with the large model, including multiple query statements; then, for each query statement, a jump index is determined based on its first similarity with adjacent query statements and its maximum similarity with the query sequence; based on the jump index of each query statement, a first index value reflecting the logical coherence of the query sequence is determined; next, the query sequence is divided into several topic clusters, each topic cluster containing several consecutive query statements; at least based on the cluster size of each topic cluster, a second index value reflecting the average follow-up depth of the query sequence is determined; finally, based on the first index value and the second index value, it is determined whether the query sequence has a malicious query intent against the large model. As can be seen from the above, the embodiments of this specification, by determining a first indicator value reflecting the logical coherence of the query sequence, can distinguish between the natural topic switching of normal users and the illogical jumps of malicious users, thus enabling the identification of attackers even when they are disguised as normal users, significantly reducing the false negative rate. By determining a second indicator value reflecting the average probing depth of the query sequence, it can accurately distinguish between deep probing by normal users and shallow traversal by malicious users, reducing the false positive rate. The joint determination of the first and second indicator values forms a complementary detection loop, which can only determine malicious query intent when a user exhibits both logically irregular jumps and shallow probing. This multi-dimensional cross-validation method greatly enhances the generalization protection capability against new attack strategies, prevents attackers from bypassing detection by disguising query statements, and improves the effectiveness of identifying malicious query intent for large models.
[0126] According to another embodiment, a malicious query intent identification device for large models is also provided, which is used to perform the methods provided in the embodiments of this specification. Figure 6 A schematic block diagram of a malicious query intent identification apparatus for large models is shown according to one embodiment. Figure 6 As shown, the device 600 includes:
[0127] Acquisition unit 61 is used to acquire the query sequence of the target user's interaction with the large model, which includes multiple query statements;
[0128] The first determining unit 62 is used to determine the jump degree index of a single query statement based on its first similarity with adjacent query statements and its maximum similarity with the query sequence for each query statement obtained by the obtaining unit 61; and to determine a first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement.
[0129] The second determining unit 63 is used to divide the query sequence obtained by the obtaining unit 61 into several topic clusters, each topic cluster containing several consecutive query statements; and to determine a second index value reflecting the average follow-up depth of the query sequence based at least on the cluster size of each topic cluster.
[0130] The identification unit 64 is used to determine whether the query sequence has a malicious query intent against the large model based on the first index value obtained by the first determining unit 62 and the second index value obtained by the second determining unit 63.
[0131] Optionally, as an embodiment, the first determining unit 62 includes:
[0132] The first calculation subunit is used to calculate the maximum value of the similarity between the current query statement and each historical query statement in the query sequence, as the maximum similarity.
[0133] The second calculation subunit is used to calculate the ratio of the first similarity to the maximum similarity obtained by the first calculation subunit, and use this ratio as the jump degree index of the current query statement.
[0134] Optionally, as an embodiment, the first determining unit 62 includes:
[0135] The statistics subunit is used to count the first number of query statements in the query sequence whose jump index is lower than the first threshold.
[0136] The third calculation subunit is used to calculate the proportion of the first quantity obtained by the statistics subunit to the total number of query statements in the query sequence, and to use the proportion as the first indicator value.
[0137] Optionally, as an embodiment, the first determining unit 62 includes:
[0138] The fourth calculation subunit is used to calculate the average value of each jump index in the query sequence, and use the average value as the first index value.
[0139] Optionally, as an embodiment, the first similarity is determined in the following manner:
[0140] Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities;
[0141] Based on the medical entity relationship network, the medical entities involved in the current query statement and its adjacent query statements are extracted to form a first entity set and a second entity set;
[0142] Calculate the ratio of the number of elements in the intersection of the first entity set and the second entity set to the number of elements in the union set, and use this ratio as the first similarity.
[0143] Optionally, as an embodiment, the first determining unit 62 includes:
[0144] The sub-unit is used to obtain a pre-built medical entity relationship network, where nodes represent medical entities and edges represent the relationships between entities;
[0145] The first judgment subunit is used to perform a graph traversal search in the medical entity relationship network obtained by the acquisition subunit, taking the medical entity involved in the current query statement as the starting node and the medical entity involved in the adjacent query statement as the ending node, to determine whether there is a direct or indirect connection path from the starting node to the ending node.
[0146] The adjustment subunit is used to adjust the ratio of the first similarity to the maximum similarity based on the judgment result obtained from the first judgment subunit, so as to obtain the jump degree index of a single sentence.
[0147] Optionally, as an embodiment, the second determining unit 63 includes:
[0148] The second judgment subunit is used to determine whether the similarity between any two adjacent query statements in the query sequence is less than a preset similarity threshold.
[0149] The partitioning unit is used to classify two query statements into the same topic cluster if the judgment result of the second judgment subunit is that the similarity between the two query statements is greater than or equal to the similarity threshold; and to classify the two query statements into different topic clusters if the judgment result of the second judgment subunit is that the similarity between the two query statements is less than the similarity threshold.
[0150] Optionally, as an embodiment, the second determining unit 63 includes:
[0151] The evaluation subunit is used to evaluate whether the query statements within each topic cluster present a deepening pattern from general concepts to specific details.
[0152] The fifth calculation subunit is used to calculate a second index value that reflects the average probing depth of the query sequence by combining the cluster size and the deepening pattern of the intra-cluster query statements obtained by the evaluation subunit.
[0153] Optionally, as an embodiment, it also includes:
[0154] An extraction unit is used to traverse the query sequence and extract the original medical terms from each query statement to obtain an initial term set.
[0155] The mapping unit is used to map the original medical terms in the initial term set obtained by the extraction unit to standard medical terms, and to merge duplicate standard medical terms to construct a standard term set.
[0156] The calculation unit is used to calculate the number of standard medical terms in the standard term set obtained by the mapping unit, and the information entropy of the frequency of occurrence of each standard medical term; the number of standard medical terms and the information entropy are weighted and summed to obtain a third index value reflecting the topic dispersion of the query sequence;
[0157] The identification unit 64 is specifically used to determine whether the query sequence has a malicious query intent based on the first indicator value, the second indicator value, and the third indicator value.
[0158] Furthermore, the extraction unit is specifically used to match each query statement with a pre-built medical terminology dictionary using regular expressions to obtain the original medical terms in each query statement. The medical terminology dictionary includes medical terms for diseases, symptoms, drugs, and examination items.
[0159] Using the apparatus provided in the embodiments of this specification, firstly, the acquisition unit 61 acquires the query sequence of the target user's interaction with the large model, including multiple query statements; then, the first determination unit 62 determines the jump degree index of each query statement based on its first similarity with adjacent query statements and its maximum similarity with the query sequence; based on the jump degree index of each query statement, a first index value reflecting the logical coherence of the query sequence is determined; next, the second determination unit 63 divides the query sequence into several topic clusters, each topic cluster containing several consecutive query statements; based at least on the cluster size of each topic cluster, a second index value reflecting the average follow-up depth of the query sequence is determined; finally, the identification unit 64 determines whether the query sequence has a malicious query intent against the large model based on the first index value and the second index value. As can be seen from the above, the embodiments of this specification, by determining a first indicator value reflecting the logical coherence of the query sequence, can distinguish between the natural topic switching of normal users and the illogical jumps of malicious users, thus enabling the identification of attackers even when they are disguised as normal users, significantly reducing the false negative rate. By determining a second indicator value reflecting the average probing depth of the query sequence, it can accurately distinguish between deep probing by normal users and shallow traversal by malicious users, reducing the false positive rate. The joint determination of the first and second indicator values forms a complementary detection loop, which can only determine malicious query intent when a user exhibits both logically irregular jumps and shallow probing. This multi-dimensional cross-validation method greatly enhances the generalization protection capability against new attack strategies, prevents attackers from bypassing detection by disguising query statements, and improves the effectiveness of identifying malicious query intent for large models.
[0160] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.
[0161] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.
[0162] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0163] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for identifying malicious query intent in large models, comprising: Obtain the query sequence of the target user's interaction with the large model, which includes multiple query statements; For each query statement, the jump index of a single statement is determined based on its first similarity with adjacent query statements and its maximum similarity with the query sequence; based on the jump index of each query statement, the first index value reflecting the logical coherence of the query sequence is determined. The query sequence is divided into several topic clusters, and each topic cluster contains several consecutive query statements; at least based on the cluster size of each topic cluster, a second indicator value reflecting the average follow-up depth of the query sequence is determined. Based on the first indicator value and the second indicator value, it is determined whether the query sequence has a malicious query intent against the large model.
2. The method as described in claim 1, wherein, For each query statement, the jump index of a single statement is determined based on its first similarity with adjacent query statements and its maximum similarity with the query sequence, including: The maximum similarity between the current query statement and each historical query statement in the query sequence is calculated and used as the maximum similarity. Calculate the ratio of the first similarity to the maximum similarity, and use this ratio as the jump index of the current query statement.
3. The method as described in claim 1, wherein, The step of determining the first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement includes: Count the first number of query statements in the query sequence whose jump index is lower than the first threshold. Calculate the proportion of the first quantity to the total number of query statements in the query sequence, and use the proportion as the first indicator value.
4. The method of claim 1, wherein, The step of determining the first indicator value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement includes: Calculate the average value of each jump index in the query sequence, and use the average value as the first index value.
5. The method as described in claim 1 or 2, wherein, The first similarity is determined in the following way: Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities; Based on the medical entity relationship network, the medical entities involved in the current query statement and its adjacent query statements are extracted to form a first entity set and a second entity set; Calculate the ratio of the number of elements in the intersection of the first entity set and the second entity set to the number of elements in the union set, and use this ratio as the first similarity.
6. The method as described in claim 1 or 2, wherein, The index for determining the jump degree of a single sentence includes: Obtain a pre-constructed network of relationships between medical entities, where nodes represent medical entities and edges represent the relationships between entities; In the medical entity relationship network, the medical entity involved in the current query statement is taken as the starting node and the medical entity involved in the adjacent query statement is taken as the ending node. A graph traversal search is performed to determine whether there is a direct or indirect connection path from the starting node to the ending node. Based on the judgment results, the ratio of the first similarity to the maximum similarity is adjusted to obtain the jump index of a single sentence.
7. The method of claim 1, wherein, The step of dividing the query sequence into several topic clusters includes: For any two adjacent query statements in the query sequence, determine whether the similarity between the two query statements is less than a preset similarity threshold; If the similarity between two query statements is greater than or equal to the similarity threshold, the two query statements will be grouped into the same topic cluster. If the similarity between two query statements is less than the similarity threshold, the two query statements will be classified into different topic clusters.
8. The method of claim 1, wherein, The determination of a second metric reflecting the average follow-up depth of the query sequence, based at least on the cluster size of each topic cluster, includes: For each topic cluster, evaluate whether the queries within the cluster present a deepening pattern from general concepts to specific details; Based on the cluster size and the deepening pattern of query statements within the cluster, a second index value reflecting the average probing depth of the query sequence is calculated.
9. The method of claim 1, wherein, Also includes: Traverse the query sequence and extract the original medical terms from each query statement to obtain an initial term set; Map the original medical terms in the initial terminology set to standard medical terms, and merge duplicate standard medical terms to construct a standard terminology set; Calculate the number of standard medical terms in the set of standard terms, and the information entropy of the frequency of occurrence of each standard medical term; The number of standard medical terms is weighted and summed with the information entropy to obtain a third index value that reflects the topic dispersion of the query sequence. Determining whether the query sequence has a malicious query intent targeting the large model includes: Based on the first indicator value, the second indicator value, and the third indicator value, it is determined whether the query sequence has a malicious query intent.
10. The method of claim 9, wherein, The extraction of raw medical terms from each query statement includes: Each query statement is matched with a pre-built medical terminology dictionary using regular expressions to obtain the original medical terms in each query statement. The medical terminology dictionary includes medical terms for diseases, symptoms, drugs, and examination items.
11. A device for identifying malicious query intent in large models, comprising: The acquisition unit is used to acquire the query sequence of the target user's interaction with the large model, which includes multiple query statements; The first determining unit is used to determine the jump degree index of a single query statement based on its first similarity with adjacent query statements and its maximum similarity with the query sequence for each query statement obtained by the obtaining unit; and to determine a first index value reflecting the logical coherence of the query sequence based on the jump degree index of each query statement. The second determining unit is used to divide the query sequence obtained by the obtaining unit into several topic clusters, each topic cluster containing several consecutive query statements; and to determine a second index value reflecting the average follow-up depth of the query sequence based at least on the cluster size of each topic cluster. The identification unit is used to determine whether the query sequence has a malicious query intent against the large model based on the first indicator value obtained by the first determining unit and the second indicator value obtained by the second determining unit.
12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-10.