Malicious text generation detection method and device for large model application
By constructing a multi-layered collaborative detection architecture that combines feature extraction, semantic analysis, and behavioral pattern recognition, the accuracy problem of malicious text detection in large model applications is solved, enabling real-time, accurate detection and automated alerts for content generated by large models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-04-03
AI Technical Summary
Existing large-scale model applications lack effective malicious text detection mechanisms, resulting in a large amount of false information and harmful content spreading through the platform. Traditional methods are unable to identify semantically disguised malicious content, leading to missed detections and false alarms.
A multi-layered collaborative detection architecture is constructed, which combines feature extraction, semantic analysis, and behavioral pattern recognition with deep semantic understanding and adaptive learning to achieve real-time detection of content generated by large models.
It significantly improves the accuracy and robustness of malicious text detection, enables effective monitoring and automated alerting of large-scale model application content, and reduces false positive and false negative rates.
Smart Images
Figure CN121786831A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information security technology, and in particular to a method and apparatus for detecting malicious text generation in large-scale model applications. Background Technology
[0002] Existing large-scale model applications lack effective mechanisms for detecting malicious text in the content generated by these models. Traditional content filtering methods primarily rely on keyword matching and simple rule filtering, lacking deep semantic understanding capabilities. This makes it difficult to identify semantically disguised malicious content, leading to numerous missed detections and false positives. This allows false information and harmful content to spread through large-scale model application platforms, posing potential harm to users and society. Therefore, how to achieve real-time and accurate detection of malicious text generated by large-scale model applications has become a pressing technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0003] The purpose of this invention is to provide a method and apparatus for detecting malicious text generation in large-scale model applications. By constructing a multi-layered collaborative detection architecture, it enables effective monitoring of the security of content in large-scale model applications.
[0004] According to one aspect of the present invention, a method for detecting malicious text generation for large model applications is provided, the method comprising: Obtain the text to be detected generated by a large model application; Extract the basic feature information of the text to be detected and quantize it to obtain the basic feature vector; Based on the text to be detected and the basic feature vector, a pre-trained semantic analysis model is used to obtain semantic feature vectors and malicious type probability distribution results; wherein, the malicious type probability distribution results represent the probability value of the text to be detected under various malicious text types; Based on the text to be detected, the semantic feature vector, and the probability distribution of the malicious type, a malicious text classification result is obtained using a pre-trained malicious behavior recognition model; The basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text are fused using a preset fusion algorithm to obtain the final detection result of the text to be detected.
[0005] Optionally, the step of extracting and quantizing the basic feature information of the text to be detected to obtain a basic feature vector specifically includes: Basic feature information is extracted from the text to be detected; wherein, the basic feature information includes: lexical features, grammatical structure features, and text sentiment features; Based on the aforementioned basic feature information, the index value of each basic feature index is calculated according to the quantitative formula associated with each basic feature index; wherein, the basic feature index includes: keyword density, sentence complexity, and sentiment polarity; The values of all basic feature indicators are combined to form the basic feature vector.
[0006] Optionally, the step of obtaining semantic feature vectors and malicious type probability distribution results based on the text to be detected and the basic feature vectors using a pre-trained semantic analysis model specifically includes: The text to be detected and the basic feature vector are input together into the semantic analysis model, and the semantic feature vector representing the deep semantics of the text to be detected is obtained through the feature extraction module in the semantic analysis model. The classifier in the semantic analysis model calculates the probability values of the text to be detected under the categories of false information, offensive language, and illegal content based on the semantic feature vector, so as to obtain the probability distribution results of the malicious type.
[0007] Optionally, the step of obtaining the malicious text classification result based on the text to be detected, the semantic feature vector, and the probability distribution result of the malicious type using a pre-trained malicious behavior recognition model specifically includes: The text to be detected, the semantic feature vector, and the probability distribution result of the malicious type are input together into the malicious behavior recognition model; The output of the malicious behavior recognition model is obtained, and the output includes: a malicious text classification result that characterizes the malicious text type to which the text to be detected belongs, and at least one of the following: the confidence level corresponding to the malicious text classification result, the risk level of the text to be detected, and the malicious behavior pattern label involved in the text to be detected; wherein, each malicious text type corresponds to multiple malicious behavior pattern labels.
[0008] Optionally, after fusing the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text using a preset fusion algorithm to obtain the final detection result of the text to be detected, the method further includes: The final classification result of the text to be detected and the comprehensive confidence level corresponding to the final classification result are extracted from the final detection result. When the overall confidence level is greater than the preset warning threshold, a warning response operation is executed; wherein, the warning response operation includes at least one of the following: performing content interception operation on the text to be detected, sending a real-time warning message to a designated management terminal, and generating and recording a security event log corresponding to the final detection result.
[0009] To achieve the above objectives, the present invention also provides a malicious text generation detection device for large-scale model applications, the device comprising: The text acquisition module is used to acquire the text to be detected generated by the large model application; The feature extraction module is used to extract and quantize the basic feature information of the text to be detected, and obtain the basic feature vector. The semantic analysis module is used to obtain semantic feature vectors and malicious type probability distribution results based on the text to be detected and the basic feature vectors using a pre-trained semantic analysis model; wherein, the malicious type probability distribution results represent the probability values of the text to be detected under various malicious text types; The behavior recognition module is used to obtain the malicious text classification result based on the text to be detected, the semantic feature vector, and the probability distribution result of the malicious type using a pre-trained malicious behavior recognition model; The information fusion module is used to fuse the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text using a preset fusion algorithm to obtain the final detection result of the text to be detected.
[0010] Optionally, the feature extraction module is specifically used for: Basic feature information is extracted from the text to be detected; wherein, the basic feature information includes: lexical features, grammatical structure features, and text sentiment features; Based on the aforementioned basic feature information, the index value of each basic feature index is calculated according to the quantitative formula associated with each basic feature index; wherein, the basic feature index includes: keyword density, sentence complexity, and sentiment polarity; The values of all basic feature indicators are combined to form the basic feature vector.
[0011] Optionally, the semantic analysis module is specifically used for: The text to be detected and the basic feature vector are input together into the semantic analysis model, and the semantic feature vector representing the deep semantics of the text to be detected is obtained through the feature extraction module in the semantic analysis model. The classifier in the semantic analysis model calculates the probability values of the text to be detected under the categories of false information, offensive language, and illegal content based on the semantic feature vector, so as to obtain the probability distribution results of the malicious type.
[0012] To achieve the above objectives, the present invention also provides a computer device, which specifically includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the malicious text generation and detection method for large model applications described above.
[0013] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the malicious text generation and detection method for large model applications described above.
[0014] The present invention provides a method and apparatus for detecting malicious text generation in large model applications. By constructing a multi-layered malicious text generation detection architecture for large model applications, and through the collaborative work of feature extraction, semantic analysis and behavioral pattern recognition, the method performs real-time detection and analysis of the text output by the large model. When malicious text generation behavior is detected, an alarm mechanism is automatically triggered, thereby achieving effective monitoring of the security of the content of large model applications. Attached Figure Description
[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of an optional process for a malicious text generation and detection method for large model applications provided in Example 1. Figure 2 This is a schematic diagram of another optional component structure for the malicious text generation and detection method for large model applications provided in Example 1. Figure 3 This is a schematic diagram of an optional component structure of the malicious text generation detection device for large model applications provided in Embodiment 2. Figure 4 This is a schematic diagram of an optional hardware structure for the computer device provided in Embodiment 3. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.
[0017] Example 1 This invention provides a method for detecting malicious text generation in large-scale model applications, such as... Figure 1 As shown, the method specifically includes the following steps: Step S101: Obtain the text to be detected generated by the large model application.
[0018] Large-scale model applications refer to software or systems developed based on large language models that possess text generation capabilities. Their core function is to receive textual queries input by users and automatically generate textual responses that conform to semantic logic. Common large-scale model applications include: dialogue interaction applications, content creation applications, and information consulting applications.
[0019] Additionally, a data interception module can be deployed at the text output interface of the large model application. When the large model application generates text and pushes it to the user through the output interface, the data interception module captures the text data in real time and directly passes it into the subsequent detection process as the text to be detected.
[0020] Step S102: Extract the basic feature information of the text to be detected and quantize it to obtain the basic feature vector.
[0021] In this embodiment, conventional text processing methods such as word segmentation, part-of-speech tagging, and statistics are used to extract basic surface feature information from the text to be detected and convert it into a numerical basic feature vector. First, natural language processing techniques such as word segmentation, part-of-speech tagging, and syntactic analysis are used to extract a series of basic feature information (basic feature indicators) from the text to be detected, which can reflect its surface statistical characteristics and basic linguistic attributes. These basic feature information only reflect the basic attributes of the text and do not involve the deep semantics of the text. Common basic feature information includes: lexical features (e.g., word frequency of specific keywords or sensitive words, word usage norms), grammatical and structural features (e.g., average sentence length, number of clauses), and text sentiment features (e.g., positive sentiment, negative sentiment). Second, the above basic feature information is converted into a unified and comparable numerical form. For example, a pre-set dictionary, statistical rules, mapping rules, and feature extraction models are used to convert the basic feature information into indicator values that meet the numerical requirements of subsequent model inputs. Finally, the indicator values of multiple quantified basic feature indicators are arranged and combined in a predetermined order to form a multi-dimensional basic feature vector, for example: [keyword density = 0.4, sentence complexity = 0.2, sentiment polarity = -0.8].
[0022] Step S103: Based on the text to be detected and the basic feature vector, obtain the semantic feature vector and the probability distribution result of malicious type using a pre-trained semantic analysis model; wherein, the probability distribution result of malicious type represents the probability value of the text to be detected under various malicious text types.
[0023] The semantic feature vector refers to a high-dimensional numerical vector (e.g., 768-dimensional) generated after mining the deep semantics of text through a semantic analysis model. This vector represents the text's implicit semantics, contextual relationships, and semantic tendencies. The probability distribution of malicious types is typically represented as a probability vector, where each dimension corresponds to a type of malicious text. The value (between 0 and 1) represents the model's confidence in classifying the text as belonging to that category, and the sum of the probabilities of all dimensions is 1. For example, a possible probability distribution of malicious types might be [False information: 0.75, Aggressive language: 0.20, Illegal content: 0.05], indicating that the model believes there is a 75% probability that the text is false information. Furthermore, the semantic analysis model can be based on the Transformer architecture, for example, trained using the BERT model.
[0024] Step S104: Based on the text to be detected, the semantic feature vector, and the probability distribution of the malicious type, obtain the malicious text classification result using a pre-trained malicious behavior recognition model.
[0025] The malicious text classification result refers to a definitive conclusion regarding the type of malicious text the text to be detected belongs to, rather than a probabilistic tendency. It's important to note that the "malicious type probability distribution result" focuses on a multi-dimensional, fine-grained quantitative assessment of the text's semantic tendency. It displays the likelihood or degree of suspicion of the text in each predefined malicious text type; it is a probability distribution that may show the text has a certain probability in multiple types. In contrast, the "malicious text classification result" is a definitive conclusion that can be directly used as a preliminary decision-making basis. For example, the malicious text classification result might be: the text to be detected belongs to the false information category.
[0026] Step S105: Use a preset fusion algorithm to fuse the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text to obtain the final detection result of the text to be detected.
[0027] The final detection result is the final judgment of the text to be detected: clearly determining whether the text to be detected "belongs" or "does not belong" to malicious text; if it is determined to "belong", then further clarifying the specific type of malicious text it belongs to (for example, whether it is "false information" or "offensive language").
[0028] This step organically integrates the basic feature vectors representing textual appearance, the probability distribution results representing semantic tendencies, and the classification results representing behavioral intentions through a fusion algorithm. This design can effectively avoid missed detections or misjudgments caused by a single model or single-layer analysis, thereby improving the overall accuracy and robustness of the detection system.
[0029] In this embodiment, a multi-layered collaborative detection architecture is constructed to achieve efficient and accurate detection of malicious text generated by a large-scale model. By employing a three-layer detection mechanism of "feature extraction," "semantic analysis," and "behavioral pattern recognition," combined with deep semantic understanding and adaptive learning optimization, the accuracy of malicious text detection is significantly improved. "Malicious text generation" refers to harmful, deceptive, or illegal text content generated using a large language model, including false information, harmful content, and offensive language. "Feature extraction" refers to extracting quantifiable basic feature information from the text generated by the large model, such as word frequency, grammatical structure, and sentiment, for subsequent detection and analysis. "Semantic analysis" refers to understanding and analyzing the deeper meaning of the text generated by the large model using natural language processing algorithms to identify its semantic features and potential intent. "Behavioral pattern recognition" is based on machine learning algorithms to identify and analyze behavioral feature patterns in the text content to determine whether the text is malicious.
[0030] Specifically, step S102, which involves extracting and quantizing the basic feature information of the text to be detected to obtain a basic feature vector, includes the following steps: Step A1: Extract basic feature information from the text to be detected; wherein, the basic feature information includes: lexical features, grammatical structure features, and text sentiment features.
[0031] Among them, lexical features may include: frequency of sensitive words, proportion of negative words, domain of professional terms, number of misspelled words, and total number of words in the text; grammatical structure features may include: total number of sentences, average sentence length, number of clauses, and frequency of punctuation; text sentiment features may include: number of positive / negative / neutral sentiment words and sentiment word concentration; in addition, basic feature information may also include: total number of words, semantic coherence features, symbol features, and format features, etc., without specific limitations.
[0032] Step A2: Based on the basic feature information, calculate the index value of each basic feature index according to the quantitative formula associated with each basic feature index; wherein, the basic feature index includes: keyword density, sentence complexity, and sentiment polarity.
[0033] The quantification formula can be a proportional formula, a difference formula, or a compound formula. The quantification formula is as follows: Keyword density = total number of keyword occurrences / total number of words in the text; Sentence complexity = number of clauses / total number of sentences; Sentiment polarity = (total number of positive sentiment words × preset positive coefficient - total number of negative sentiment words × preset negative coefficient) / total number of words in the text. The above is only an example of the quantification formula and does not limit the form of the quantification formula. For example, when calculating sentiment polarity, positive sentiment words, negative sentiment words, degree words (modifiers), and negation words are first identified from the text to be tested through a predefined sentiment dictionary. Then, the preliminary value of sentiment polarity is calculated based on the above quantification formula. Finally, the preliminary value is adjusted by degree words (modifiers) and negation words to obtain the final sentiment polarity.
[0034] In addition, basic feature indicators may also include: grammatical error rate (grammatical error rate = number of grammatical errors / total number of words), lexical standardization (lexical standardization = number of normal words / total number of words), semantic coherence (calculated through a preset model), etc., without specific limitations here.
[0035] Step A3: Combine the index values of all basic feature indicators to form the basic feature vector.
[0036] Steps A1 to A3 above define a standardized process for converting text from unstructured language into structured feature vectors. The features and formulas listed are all scalable examples. These steps provide a standardized, computable, and comprehensive representation of text surface features for subsequent analysis.
[0037] Furthermore, step S103, which involves obtaining semantic feature vectors and malicious type probability distribution results based on the text to be detected and the basic feature vectors using a pre-trained semantic analysis model, specifically includes the following steps: Step B1: Input the text to be detected and the basic feature vector into the semantic analysis model, and obtain the semantic feature vector representing the deep semantics of the text to be detected through the feature extraction module in the semantic analysis model.
[0038] The semantic analysis model adopts the BERT-base model, and the semantic feature vector is a 768-dimensional vector, with all dimensions forming a distributed semantic space.
[0039] Step B2: Using the classifier in the semantic analysis model, calculate the probability values of the text to be detected under the categories of false information, offensive language, and illegal content based on the semantic feature vector, so as to obtain the probability distribution results of the malicious type.
[0040] In this embodiment, by extracting semantic feature vectors, the model can understand the underlying intent and contextual meaning of the text, thereby effectively identifying malicious content that does not use obvious sensitive words but is expressed through semantic combination, metaphor, or complex logic, overcoming the inherent defects of traditional methods based on surface keyword matching. In addition, the probability distribution results of malicious types provide the system with a multi-dimensional and quantifiable risk assessment, which makes subsequent processing more flexible and significantly improves the precision and reliability of system decision-making.
[0041] Furthermore, step S104, which involves obtaining a malicious text classification result based on the text to be detected, the semantic feature vector, and the probability distribution result of the malicious type, using a pre-trained malicious behavior recognition model, specifically includes the following steps: Step C1: Input the text to be detected, the semantic feature vector, and the probability distribution result of the malicious type into the malicious behavior recognition model.
[0042] Among them, a malicious behavior recognition model is obtained by training a deep neural network or decision tree using historical sample data.
[0043] Step C2: Obtain the output of the malicious behavior recognition model. The output includes: a malicious text classification result that characterizes the malicious text type to which the text to be detected belongs, and at least one of the following: the confidence level corresponding to the malicious text classification result, the risk level of the text to be detected, and the malicious behavior pattern label involved in the text to be detected; wherein, each malicious text type corresponds to multiple malicious behavior pattern labels.
[0044] The malicious text classification results include: not malicious text, malicious text - false information, malicious text - offensive language, and malicious text - illegal content. Confidence is a quantifiable value for the credibility of the above classification results, ranging from 0 to 1, with higher values indicating more reliable judgments. Malicious behavior pattern labels are more granular and specific behavioral descriptions of malicious text types. For example, the behavior pattern labels under "false information" are: financial fraud, false advertising, and rumors; the behavior pattern labels under "offensive language" are: personal insults, group discrimination, and violent threats; and the behavior pattern labels under "illegal content" are: illegal and pornographic.
[0045] In addition, the risk level is calculated based on the malicious text classification results, confidence level, and malicious behavior pattern labels. It is used to quantify the degree of harm of malicious text (e.g., high risk, medium risk, low risk) to guide subsequent handling strategies (e.g., interception, alert, review).
[0046] Furthermore, step S105, which involves fusing the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text using a preset fusion algorithm to obtain the final detection result of the text to be detected, specifically includes the following steps: Step D1: Use a dynamic weight adjustment mechanism to determine the weight values of the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text.
[0047] For example, the weight values of the basic feature vectors are based on the comprehensive deviation of each quantitative indicator from the preset normal range to assess the abnormal intensity of the surface features (the more significant the abnormality, the higher the weight); the weight values of the malicious type probability distribution results are determined based on the information entropy of the probability distribution, the more concentrated the distribution (the lower the entropy value), the higher the weight value, indicating that the semantic analysis judgment is more certain; the weight values of the malicious text classification results are directly related to the confidence level of its own output, the higher the confidence level, the higher the weight value.
[0048] Step D2: Based on the determined weight values, calculate the final detection result using a weighted fusion algorithm; wherein, the final detection result includes: final classification result, comprehensive confidence level, comprehensive risk level, weight scores at each level, and explanation of the detection basis.
[0049] It should be noted that the classification system for the final classification results, overall confidence level, and overall risk level is consistent with the classification system in step S104, but it is a verification and confirmation of the previous results, and is more reliable.
[0050] In this embodiment, the pre-set fusion algorithm solves the problem of blind spots and biases that may exist in the judgment of a single level or a single model; the introduction of a dynamic weight fusion mechanism can flexibly adjust the degree of acceptance of evidence at different levels according to specific circumstances, thereby significantly improving the overall accuracy of the detection system in recognizing complex, hidden or new malicious texts.
[0051] Furthermore, after step S105, the method further includes: Step E1: Parse the final classification result of the text to be detected and the comprehensive confidence level corresponding to the final classification result from the final detection result; Step E2: When the overall confidence level is greater than the preset warning threshold, execute the warning response operation; wherein the warning response operation includes at least one of the following: performing content interception operation on the text to be detected, sending a real-time warning message to a designated management terminal, generating and recording a security event log corresponding to the final detection result.
[0052] In this embodiment, the intelligent detection results are directly linked with the automated security response, realizing closed-loop management from risk identification to risk handling, significantly improving the timeliness, proactivity, and manageability of security protection. Specifically, by analyzing the final classification result and comprehensive confidence level in the final detection result and using this as the basis for decision-making, when the confidence level exceeds a preset threshold, tiered and diverse early warning operations are automatically triggered. Furthermore, if the final detection result contains comprehensive risk level information, this mechanism can be expanded to trigger differentiated response measures based on different levels (such as high, medium, and low risk). (For example, high risk is immediately intercepted and an alarm is triggered by telephone, while medium risk is only recorded and notified by email), thereby achieving refined and intelligent security operations, enabling the defender to rationally allocate resources and transform passive defense into proactive control.
[0053] In addition, in practical applications, an asynchronous detection pipeline architecture can be constructed to send the text output by the large model application into the detection queue in real time, and the background parallel processing engine performs concurrent analysis on the tasks in the queue. This design decouples the detection process from the large model generation process in time, ensuring that even in high-concurrency, multi-user session scenarios, the system can complete malicious text detection tasks with high efficiency and stable throughput, thereby guaranteeing the real-time performance and responsiveness of the overall service and avoiding the impact of detection bottlenecks on user experience. Furthermore, existing content detection methods are relatively static and difficult to adapt to the rapid evolution of malicious content, with limited protection capabilities and an inability to form a complete security protection system, which seriously restricts the content security guarantee of large model applications. This system has online learning and self-optimization capabilities. By continuously collecting detection result feedback, importing user feature text, and upgrading the feature library, it continuously improves its ability to identify new malicious text patterns, realizing the self-evolution of the detection system and automatically adjusting and optimizing detection algorithm parameters based on detection result feedback.
[0054] like Figure 2As shown, in this embodiment, a three-layer detection mechanism architecture is constructed. The bottom layer, the feature extraction layer, is responsible for basic text feature analysis; the middle layer, the semantic analysis layer, performs deep semantic understanding; and the top layer, the behavior pattern recognition layer, comprehensively judges malicious behavior patterns, thus forming a complete detection pipeline. This embodiment constructs a three-layer progressive detection architecture of "basic feature extraction - deep semantic analysis - behavior pattern recognition," and forms an organic whole through targeted information transmission and functional complementarity between each layer. Specifically, the first layer, basic feature vector extraction, focuses on the surface attributes of the text (sensitive words, grammar, etc.), quickly filtering out obviously malicious text to reduce the computational load of subsequent deep models, while providing initial feature support for semantic analysis. The second layer, the semantic analysis model, breaks through the limitations of the surface layer, mines the hidden malicious intent in the text, and outputs a probability distribution, which not only makes up for the deficiency of single surface detection in missing "obscure malicious text," but also provides deep semantic basis for malicious behavior recognition. The third layer, the malicious behavior recognition model, integrates multi-dimensional information of text, semantic vectors, and probability distribution, outputs clear classification results, and effectively avoids misjudgments caused by semantic ambiguity in single semantic models. The three-layer detection is interconnected. Each layer provides accurate input for the next, and the next layer corrects the limitations of the previous layer. The advantages of the three layers are then integrated through a dynamic weight fusion algorithm, which significantly reduces the false alarm rate and false negative rate compared to existing single detection or simple combination detection.
[0055] Example 2 This invention provides a malicious text generation detection device for large model applications, such as... Figure 3 As shown, the device specifically includes the following components: The text acquisition module 301 is used to acquire the text to be detected generated by the large model application; Feature extraction module 302 is used to extract and quantize the basic feature information of the text to be detected to obtain a basic feature vector; The semantic analysis module 303 is used to obtain semantic feature vectors and malicious type probability distribution results based on the text to be detected and the basic feature vectors using a pre-trained semantic analysis model; wherein, the malicious type probability distribution results represent the probability values of the text to be detected under various malicious text types. The behavior recognition module 304 is used to obtain the malicious text classification result based on the text to be detected, the semantic feature vector and the probability distribution result of the malicious type using a pre-trained malicious behavior recognition model; The information fusion module 305 is used to fuse the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text using a preset fusion algorithm to obtain the final detection result of the text to be detected.
[0056] Specifically, the feature extraction module 302 is used for: Basic feature information is extracted from the text to be detected; wherein, the basic feature information includes: lexical features, grammatical structure features, and text sentiment features; Based on the aforementioned basic feature information, the index value of each basic feature index is calculated according to the quantitative formula associated with each basic feature index; wherein, the basic feature index includes: keyword density, sentence complexity, and sentiment polarity; The values of all basic feature indicators are combined to form the basic feature vector.
[0057] Furthermore, the semantic analysis module 303 is specifically used for: The text to be detected and the basic feature vector are input together into the semantic analysis model, and the semantic feature vector representing the deep semantics of the text to be detected is obtained through the feature extraction module in the semantic analysis model. The classifier in the semantic analysis model calculates the probability values of the text to be detected under the categories of false information, offensive language, and illegal content based on the semantic feature vector, so as to obtain the probability distribution results of the malicious type.
[0058] Furthermore, the behavior recognition module 304 is specifically used for: The text to be detected, the semantic feature vector, and the probability distribution result of the malicious type are input together into the malicious behavior recognition model; The output of the malicious behavior recognition model is obtained, and the output includes: a malicious text classification result that characterizes the malicious text type to which the text to be detected belongs, and at least one of the following: the confidence level corresponding to the malicious text classification result, the risk level of the text to be detected, and the malicious behavior pattern label involved in the text to be detected; wherein, each malicious text type corresponds to multiple malicious behavior pattern labels.
[0059] Furthermore, the device also includes a warning module, specifically used for: The final classification result of the text to be detected and the comprehensive confidence level corresponding to the final classification result are extracted from the final detection result. When the overall confidence level is greater than the preset warning threshold, a warning response operation is executed; wherein, the warning response operation includes at least one of the following: performing content interception operation on the text to be detected, sending a real-time warning message to a designated management terminal, and generating and recording a security event log corresponding to the final detection result.
[0060] Example 3 This embodiment also provides a computer device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc., capable of executing programs. Figure 4 As shown, the computer device 40 in this embodiment includes, but is not limited to, a memory 401 and a processor 402 that are communicatively connected to each other via a system bus. It should be noted that... Figure 4 Only a computer device 40 with components 401-402 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0061] In this embodiment, the memory 401 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 401 may be an internal storage unit of the computer device 40, such as the hard disk or memory of the computer device 40. In other embodiments, the memory 401 may also be an external storage device of the computer device 40, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 40. Of course, the memory 401 may include both the internal storage unit and its external storage device of the computer device 40. In this embodiment, the memory 401 is typically used to store the operating system and various application software installed on the computer device 40. In addition, the memory 401 may also be used to temporarily store various types of data that have been output or will be output.
[0062] In some embodiments, processor 402 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 402 is typically used to control the overall operation of computer device 40.
[0063] Specifically, in this embodiment, the processor 402 is used to execute the program stored in the memory 401 for a malicious text generation and detection method for large model applications. When the program for the malicious text generation and detection method for large model applications is executed, it performs the following steps: Obtain the text to be detected generated by a large model application; Extract the basic feature information of the text to be detected and quantize it to obtain the basic feature vector; Based on the text to be detected and the basic feature vector, a pre-trained semantic analysis model is used to obtain semantic feature vectors and malicious type probability distribution results; wherein, the malicious type probability distribution results represent the probability value of the text to be detected under various malicious text types; Based on the text to be detected, the semantic feature vector, and the probability distribution of the malicious type, a malicious text classification result is obtained using a pre-trained malicious behavior recognition model; The basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text are fused using a preset fusion algorithm to obtain the final detection result of the text to be detected.
[0064] For a detailed description of the above method steps, please refer to Example 1. This example will not be repeated here.
[0065] Example 4 This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program. When the computer program is executed by a processor, it implements the following method steps: Obtain the text to be detected generated by a large model application; Extract the basic feature information of the text to be detected and quantize it to obtain the basic feature vector; Based on the text to be detected and the basic feature vector, a pre-trained semantic analysis model is used to obtain semantic feature vectors and malicious type probability distribution results; wherein, the malicious type probability distribution results represent the probability value of the text to be detected under various malicious text types; Based on the text to be detected, the semantic feature vector, and the probability distribution of the malicious type, a malicious text classification result is obtained using a pre-trained malicious behavior recognition model; The basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text are fused using a preset fusion algorithm to obtain the final detection result of the text to be detected.
[0066] For a detailed description of the above method steps, please refer to the first embodiment. This embodiment will not repeat the details here.
[0067] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0068] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0070] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method for detecting malicious text generation in large-scale model applications, characterized in that, The method includes: Obtain the text to be detected generated by a large model application; Extract the basic feature information of the text to be detected and quantize it to obtain the basic feature vector; Based on the text to be detected and the basic feature vector, a pre-trained semantic analysis model is used to obtain semantic feature vectors and malicious type probability distribution results; wherein, the malicious type probability distribution results represent the probability value of the text to be detected under various malicious text types; Based on the text to be detected, the semantic feature vector, and the probability distribution of the malicious type, a malicious text classification result is obtained using a pre-trained malicious behavior recognition model; The basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text are fused using a preset fusion algorithm to obtain the final detection result of the text to be detected.
2. The malicious text generation detection method for large model applications according to claim 1, characterized in that, The step of extracting and quantizing the basic feature information of the text to be detected to obtain a basic feature vector specifically includes: Basic feature information is extracted from the text to be detected; wherein, the basic feature information includes: lexical features, grammatical structure features, and text sentiment features; Based on the aforementioned basic feature information, the index value of each basic feature index is calculated according to the quantitative formula associated with each basic feature index; wherein, the basic feature index includes: keyword density, sentence complexity, and sentiment polarity; The values of all basic feature indicators are combined to form the basic feature vector.
3. The malicious text generation detection method for large model applications according to claim 1, characterized in that, The step of obtaining semantic feature vectors and malicious type probability distribution results based on the text to be detected and the basic feature vectors using a pre-trained semantic analysis model specifically includes: The text to be detected and the basic feature vector are input together into the semantic analysis model, and the semantic feature vector representing the deep semantics of the text to be detected is obtained through the feature extraction module in the semantic analysis model. The classifier in the semantic analysis model calculates the probability values of the text to be detected under the categories of false information, offensive language, and illegal content based on the semantic feature vector, so as to obtain the probability distribution results of the malicious type.
4. The malicious text generation detection method for large model applications according to claim 1, characterized in that, The step of obtaining malicious text classification results based on the text to be detected, the semantic feature vector, and the probability distribution of the malicious type using a pre-trained malicious behavior recognition model specifically includes: The text to be detected, the semantic feature vector, and the probability distribution result of the malicious type are input together into the malicious behavior recognition model; The output of the malicious behavior recognition model is obtained, and the output includes: a malicious text classification result that characterizes the malicious text type to which the text to be detected belongs, and at least one of the following: the confidence level corresponding to the malicious text classification result, the risk level of the text to be detected, and the malicious behavior pattern label involved in the text to be detected; wherein, each malicious text type corresponds to multiple malicious behavior pattern labels.
5. The malicious text generation detection method for large model applications according to claim 1, characterized in that, After fusing the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text using a preset fusion algorithm to obtain the final detection result of the text to be detected, the method further includes: The final classification result of the text to be detected and the comprehensive confidence level corresponding to the final classification result are extracted from the final detection result. When the overall confidence level is greater than the preset warning threshold, a warning response operation is executed; wherein, the warning response operation includes at least one of the following: performing content interception operation on the text to be detected, sending a real-time warning message to a designated management terminal, and generating and recording a security event log corresponding to the final detection result.
6. A malicious text generation detection device for large-scale model applications, characterized in that, The device includes: The text acquisition module is used to acquire the text to be detected generated by the large model application; The feature extraction module is used to extract and quantize the basic feature information of the text to be detected, and obtain the basic feature vector. The semantic analysis module is used to obtain semantic feature vectors and malicious type probability distribution results based on the text to be detected and the basic feature vectors using a pre-trained semantic analysis model; wherein, the malicious type probability distribution results represent the probability values of the text to be detected under various malicious text types; The behavior recognition module is used to obtain the malicious text classification result based on the text to be detected, the semantic feature vector, and the probability distribution result of the malicious type using a pre-trained malicious behavior recognition model; The information fusion module is used to fuse the basic feature vector, the probability distribution result of the malicious type, and the classification result of the malicious text using a preset fusion algorithm to obtain the final detection result of the text to be detected.
7. The malicious text generation detection device for large model applications according to claim 6, characterized in that, The feature extraction module is specifically used for: Basic feature information is extracted from the text to be detected; wherein, the basic feature information includes: lexical features, grammatical structure features, and text sentiment features; Based on the aforementioned basic feature information, the index value of each basic feature index is calculated according to the quantitative formula associated with each basic feature index; wherein, the basic feature index includes: keyword density, sentence complexity, and sentiment polarity; The values of all basic feature indicators are combined to form the basic feature vector.
8. The malicious text generation detection device for large model applications according to claim 6, characterized in that, The semantic analysis module is specifically used for: The text to be detected and the basic feature vector are input together into the semantic analysis model, and the semantic feature vector representing the deep semantics of the text to be detected is obtained through the feature extraction module in the semantic analysis model. The classifier in the semantic analysis model calculates the probability values of the text to be detected under the categories of false information, offensive language, and illegal content based on the semantic feature vector, so as to obtain the probability distribution results of the malicious type.
9. A computer device, the computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 5.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.