A fraud-related short message identification model optimization method and system based on multiple agents

Abstract

The application discloses a kind of based on multi-agent's fraud-related short message identification model optimization method and system, it is related to communication network security and artificial intelligence technical field, the method is realized by multi-agent collaborative work: fraud strategy analysis agent carries out semantic analysis to multi-source short message data, extracts structured fraud strategy element and updates knowledge base;Sample generation agent generates fraud short message sample containing semantics, countermeasure variant automatically according to this, standardization training data set is constructed after being cleaned and multi-label labeled by data processing agent;Model training agent determines optimal model configuration by multiple parameter experiments, and training obtains production model and is deployed online;Collect online feedback, when meeting condition, automatically restart strategy extraction process, iteration optimization model.The application realizes the whole process automation of fraud-related short message identification model, greatly reduces artificial cost, shortens iteration cycle, adapts new fraud strategy by means of online feedback, improves model generalization, robustness and identification accuracy.

Description

Technical Field

[0001] This invention relates to the fields of communication network security and artificial intelligence technology, and in particular to an optimization method and system for identifying fraudulent text messages based on multi-agent technology. Background Technology

[0002] With the development of mobile communication services, SMS remains a crucial channel for fraudsters to commit scams. Fraudulent SMS messages typically impersonate banks, recruitment platforms, logistics companies, or government agencies, enticing users to click on links, provide personal information, or add contact details. In recent years, fraudulent SMS messages have exhibited significant characteristics of automation, variation, and rapid evolution. For example, they evade detection models by inserting symbols, using homophones, and rewriting semantics, posing a significant challenge to traditional SMS identification systems.

[0003] Currently, mainstream technologies for identifying fraudulent SMS messages primarily employ machine learning or deep learning models for text classification, such as TextCNN and BERT, to identify risks associated with SMS content. A typical process usually includes: manually collecting and organizing historical fraudulent SMS data, manually cleaning and labeling the data, constructing training and prediction datasets, training the text classification model, and deploying the model. While this approach achieved some success in early fraud detection, as fraud tactics continue to evolve, existing systems have gradually revealed significant shortcomings, such as the inability to proactively and continuously optimize the model, insufficient ability to identify new types of fraud and the inability to iterate quickly, and high manual costs. Summary of the Invention

[0004] The main objective of this invention is to provide an optimization method for a fraudulent SMS identification model based on multi-agent systems.

[0005] Another objective of this invention is to propose an optimization device for a fraudulent SMS identification model based on multi-agent systems.

[0006] The third objective of this invention is to provide an electronic device.

[0007] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes an optimization method for a fraudulent SMS identification model based on multi-agent systems, comprising:

[0009] S1. Collect multi-source SMS data, use the fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a multi-agent unified scheduling mechanism. S2, based on the structured fraud strategy elements in the updated fraud strategy knowledge base, automatically generates fraud SMS samples, while collecting difficult samples, and transmits all samples to the data processing agent through a unified scheduling mechanism, and cleans and labels the generated fraud SMS samples to build a standardized training dataset. S3, the model training agent performs experimental training based on a standardized training dataset, the model evaluation agent evaluates the performance of the model during the training process, the model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration, and trains and generates a production model on complete training data; S4. Deploy the production model to the SMS identification system and collect online feedback data. When the online feedback data meets the preset trigger conditions, automatically start a new round of structured fraud strategy element extraction process and iteratively update the production model.

[0010] Optionally, multi-source SMS data is collected, and a fraud strategy analysis agent is used to perform semantic parsing on the multi-source SMS data to extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a unified multi-agent scheduling mechanism, including: We collect real-world risk SMS data, user-reported SMS data, SMS data with low confidence from models, SMS data with inconsistent identification results from multiple models, and fraud case data obtained from web crawlers as multi-source SMS data. Semantic analysis is performed on the multi-source SMS data to extract fraud types, role impersonation, risk triggering events, behavior inducement methods, and fraud target behaviors as elements of structured fraud strategies. The extracted structured fraud strategy elements are used to construct fraud strategy templates and stored in a fraud strategy knowledge base to form a reusable fraud strategy structure model, thereby completing the update of the fraud strategy knowledge base. The updated fraud strategy knowledge base is input into the sample generation agent through the unified scheduling mechanism provided by the multi-agent collaborative scheduling module.

[0011] Optionally, based on the structured fraud strategy elements in the updated fraud strategy knowledge base, fraudulent SMS samples are automatically generated. Simultaneously, difficult samples are collected. All samples are transmitted to the data processing agent through a unified scheduling mechanism. The generated fraudulent SMS samples are then cleaned and multi-labeled to construct a standardized training dataset, including: Read the fraud strategy templates from the updated fraud strategy knowledge base and combine them with a pre-set keyword library to generate basic fraud SMS text. The basic fraudulent SMS text is subjected to character splitting, synonym replacement, and symbol insertion operations to generate various semantic variant samples; The semantic variant samples are perturbed based on an adversarial generation strategy to generate adversarial variant samples. The system automatically collects low-confidence SMS messages and difficult samples with conflicting multi-model predictions generated during the online identification process, and merges the basic fraudulent SMS texts, semantic variant samples, adversarial variant samples, and difficult samples to form a diverse fraudulent SMS sample library. The fraudulent text message sample library is transmitted to a data processing agent, where the samples are cleaned and labeled with multiple tags to construct a standardized training dataset.

[0012] Optionally, the model training agent performs experimental training based on a standardized training dataset, and the model evaluation agent evaluates the model's performance during training. The model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration, and trains a production model on complete training data, including: The generated fraudulent text message samples are cleaned by performing data deduplication, semantic error detection, and abnormal text filtering operations using a data processing intelligent agent. The cleaned fraudulent text message samples are automatically labeled with fraud type tags, fraud strategy tags, and risk level tags, with each text message sample being labeled with at least one of the above tags; The labeled fraudulent text message samples are divided into training set, validation set and test set to form a standardized training dataset containing a standardized data distribution. Based on a standardized training dataset, the model training agent performs multiple sets of experimental training with different model structures and training parameter combinations. The different model structures include BERT, RoBERTa, ModernBERT, TextCNN, or Transformer models. The model corresponding to each set of experimental parameters is trained quickly for a fixed duration using the validation set, and the accuracy, recall and F1 score of each experimental training result are calculated as performance indicators. The optimal model configuration is determined by comprehensively evaluating the experimental training results using a preset scoring function, and then a production model is generated by training on complete training data using the optimal model configuration.

[0013] Optionally, when online feedback data meets preset trigger conditions, a new round of structured fraud strategy element extraction process is automatically initiated, iteratively updating the production model, including: When the online feedback data meets the preset triggering conditions, a new round of processes is automatically started. In production mode, the determined optimal model configuration is called, and the selected model structure is formally trained using the complete training dataset. During the formal full-scale training process, a multi-model fusion strategy is adopted to weight and fuse the prediction results of multiple independently trained models to generate a new production model with higher recognition performance and robustness.

[0014] To achieve the above objectives, a second aspect of the present invention provides an optimization device for a multi-agent-based fraudulent SMS identification model, comprising: The extraction module is used to collect multi-source SMS data, use the fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a unified scheduling mechanism of multiple agents. The generation module is used to automatically generate fraudulent SMS samples based on the structured fraud strategy elements in the updated fraud strategy knowledge base. At the same time, it collects difficult samples, transmits all samples to the data processing agent through a unified scheduling mechanism, and cleans and labels the generated fraudulent SMS samples to build a standardized training dataset. The training module is used to train the model agent to perform experimental training based on a standardized training dataset. The model evaluation agent evaluates the performance of the model during the training process. The model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration and trains a production model on complete training data. The update module is used to deploy the production model to the SMS identification system and collect online feedback data. When the online feedback data meets the preset trigger conditions, a new round of structured fraud strategy element extraction process is automatically started to iteratively update the production model.

[0015] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0016] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory, in order to implement the multi-agent-based fraudulent SMS identification model optimization method as described in the first aspect embodiment.

[0017] To achieve the above objectives, a fourth aspect of this application proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-agent-based fraudulent SMS identification model optimization method as described in the first aspect embodiment.

[0018] The embodiments of the present invention have the following beneficial effects: This invention automates the entire process of fraudulent SMS identification models, from strategy discovery and data generation to training and optimization, significantly reducing manual costs and shortening model iteration cycles. Simultaneously, through an online feedback mechanism, it continuously adapts to new fraud strategies, effectively improving the model's generalization ability, robustness, and identification accuracy. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an optimization method for a multi-agent-based fraudulent SMS identification model provided in this embodiment of the invention; Figure 2 This invention provides a structural diagram of an optimized system for identifying fraudulent text messages based on a multi-agent model. Figure 3 This is a structural diagram of a multi-agent-based fraudulent SMS identification model optimization device provided in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The following describes, with reference to the accompanying drawings, an optimization method and system for identifying fraudulent text messages based on a multi-agent model according to embodiments of the present invention.

[0023] Example 1 This invention provides an optimization method for a multi-agent-based fraudulent SMS identification model, such as... Figure 1 As shown, the method includes the following steps: S1: Collect multi-source SMS data, use the fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a multi-agent unified scheduling mechanism.

[0024] In order to accurately capture the latest fraud methods and provide a basis for sample generation, this application builds a multi-agent collaborative system and constructs and updates a fraud strategy knowledge base through strategy parsing.

[0025] In this embodiment, a multi-agent system is first built as support, using AutoGen, Langchain, and other multi-agent frameworks as the underlying collaborative framework. This provides powerful task orchestration and tool invocation capabilities. Simultaneously, the fraudulent SMS identification task is comprehensively functionally decomposed, clearly defining core sub-tasks such as fraud strategy analysis, sample generation, data processing, and model training. A dedicated agent is defined for each sub-task, with corresponding role descriptions, system prompts, and callable toolsets configured. All agents are uniformly registered under a scheduling module named "MasterAgent," constructing a unified collaborative scheduling mechanism to achieve efficient collaborative operation of multiple agents. The fraud strategy analysis agent, responsible for the core work of this step, is defined as an "anti-fraud analyst," and its callable toolset includes semantic parsing tools, strategy extraction tools, and knowledge base query tools, all uniformly scheduled and managed by MasterAgent.

[0026] In this embodiment, the fraud strategy analysis agent collects multi-source SMS data as the basis for analysis. Specifically, this includes the latest user-reported SMS data pulled from the user reporting system, SMS data with low model confidence (fuzzy samples with model scores between 0.4 and 0.6) extracted from online logs, SMS data with conflicting prediction results from different versions of the model, and the latest fraud news and case descriptions obtained from a public fraud case database through web crawling. This ensures comprehensive data coverage and the ability to capture various fraud scenarios.

[0027] After the data collection is completed, the fraud strategy analysis agent performs precise semantic analysis and strategy extraction on these multi-source SMS samples. For example, for SMS samples, the agent calls semantic analysis tools to identify key entities in the SMS and structures them into standardized fraud strategy templates. These templates contain core elements such as strategy ID, fraud type, target platform, triggering event, panic trigger point, contact information, action instructions, and basic script templates.

[0028] In this embodiment, all structured fraud strategy templates are uniformly saved to the database to form an expandable fraud strategy knowledge base. At the same time, a knowledge base update mechanism is established. Whenever a new structured fraud strategy element is extracted, it will be updated to the knowledge base in real time, providing comprehensive and accurate strategy template support for subsequent sample generation tasks and ensuring that the generated samples can keep up with the latest fraud trends.

[0029] In this embodiment, a multi-agent collaborative system is built, multi-source SMS data is collected and structured fraud strategy elements are extracted, and an scalable fraud strategy knowledge base is constructed to provide accurate template support for subsequent sample generation.

[0030] S2 automatically generates fraudulent SMS samples based on the structured fraud strategy elements in the updated fraud strategy knowledge base, while collecting difficult samples. All samples are then transmitted to the data processing agent through a unified scheduling mechanism. The generated fraudulent SMS samples are cleaned and labeled with multiple tags to construct a standardized training dataset.

[0031] In this embodiment of the application, the core work of this step is completed collaboratively by the sample generation agent and the data processing agent. The sample generation agent is defined as a "sample designer" and can call tools such as generating script templates, generating text variants, and generating adversarial samples. The data processing agent is defined as a "data cleaner" and can call tools such as deduplication, quality inspection, and automatic annotation. Both work in an orderly manner under the scheduling of MasterAgent.

[0032] First, after the sample generation agent is awakened by MasterAgent, it obtains the latest structured fraud strategy elements from the fraud strategy knowledge base. It first calls the script generation template tool to convert the structured strategy templates in the knowledge base into basic scripts, and then calls the text variant generation tool to rewrite the basic scripts in a variety of ways using a large language model. This generates a large number of semantic variants that are similar in meaning but different in expression. The specific rewriting methods include keyword replacement, sentence transformation, variant rewriting (multi-form rewriting through large models and prompt word engineering), and adding noise (adding harmless and irrelevant words such as "[Friendly reminder]" or "Reply T to unsubscribe" at the beginning or end of the text message to avoid simple keyword filtering).

[0033] To further enhance the robustness of subsequent models, the sample generation agent also invokes a generative adversarial example tool to slightly perturb the generated samples, creating adversarial variants. Specific perturbation methods include synonym substitution, symbol perturbation, and misspelling injection. Simultaneously, the sample generation agent also obtains data from the difficult sample queue reported by the model evaluation agent, using these difficult samples as seeds to expand the sample pool through variant generation techniques, forming targeted augmentation samples that address the model's weaknesses and further enriching sample diversity. After sample generation is complete, the data processing agent standardizes all generated samples, first performing approximate deduplication and then employing a similarity calculation algorithm.

[0034] in, For text vectors, when similarity Duplicate samples are identified and removed; samples containing abnormal characters, garbled text, or semantically incoherent text, such as texts shorter than 5 or longer than 200 characters, are then filtered out to ensure sample quality.

[0035] Since the same text message may have multiple tag sources (including strategy-generated tags, model-predicted tags, and user-reported tags), to confirm the fraud type corresponding to the sample and ensure tag accuracy, this application introduces a tag confidence function to fuse and evaluate multi-source tags. The confidence calculation formula is as follows:

[0036] in, This indicates the number of data sources that support this label, and s represents the probability value predicted by the model. This indicates the degree of label discrepancy between different data sources. For the Sigmoid function, , , The weighting parameters are 0.4, 0.4 and 0.2 respectively in this embodiment.

[0037] In this embodiment of the application, the samples are screened and classified based on the label confidence level: when When, this label is included in the training set as a high-confidence label; when When this happens, the sample is marked as a difficult sample and fed back to the sample-generating agent for further enhancement processing. Simultaneously, to comprehensively represent the sample features, each text message is tagged with multiple labels, such as... Impersonating officials to commit fraud High panic, High. The processed samples are stored in databases such as MySQL and HDFS. The data processing agent randomly divides the data into training, validation, and test sets in a 7:2:1 ratio, ultimately constructing a standardized, high-quality training dataset to provide reliable data support for subsequent model training.

[0038] In this embodiment, by collecting multi-source SMS data and performing semantic parsing to extract structured strategy elements, and combining multi-agent collaborative operations, a standardized training dataset is generated, providing data support and a foundation for model training for fraudulent SMS identification.

[0039] S3, the model training agent performs multiple sets of experimental training with different model structures and training parameter combinations based on a standardized training dataset. The model evaluation agent evaluates the performance of the model during the experimental training process. The model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration. The optimal model configuration is then used to train and generate a production model on the complete training data.

[0040] To provide high-quality, standardized data for model training, this application utilizes multi-agent collaboration to generate diverse samples, which are then cleaned and labeled to construct a standardized training dataset. To select the optimal model through multi-dimensional experimental training and ensure its recognition performance, this application employs multi-agent collaboration to complete model experimental training, performance evaluation, and production-level training, determining the optimal model configuration and improving model stability and recognition accuracy.

[0041] In this embodiment of the application, this step is completed collaboratively by the model training agent, the model evaluation agent, and the multi-agent collaborative scheduling module (MasterAgent). The model training agent is defined as a "model trainer" and can call tools such as starting experiments, formal training, and selecting the optimal model. The model evaluation agent is defined as a "model quality inspector" and can call tools such as performance evaluation, robustness testing, and hard sample mining.

[0042] First, after the model training agent is triggered by the MasterAgent, it reads the standardized training set and validation set constructed from the database, starts experimental mode training, and launches dozens of parallel training experiments according to the preset search space. The experimental parameters cover multiple dimensions, including model structure (BERT, RoBERTa, ModernBERT, etc.), learning rate (2e-5, 3e-5, 5e-5), batch size (16, 32, 64), training steps (5000, 10000), etc.

[0043] In this embodiment, each experiment is trained for a preset number of steps. After training, the system calls a preset model scoring function:

[0044] in , , The system automatically calculates the scores of all experiments and initially selects the model configuration with the highest score as the candidate optimal model. Simultaneously, the experiment mode supports incremental experiments. When a new strategy is detected in the fraud strategy knowledge base, a short-step rapid experiment can be conducted based on the existing model to verify the recognition effect of the new strategy before deciding whether to start the full experimental process, thus improving training efficiency.

[0045] Once the candidate optimal model is determined, the model evaluation agent loads the model and performs a comprehensive evaluation on a reserved test set. This includes calculating core performance metrics such as accuracy, recall, F1 score, and false positive rate, as well as generating detailed classification reports and confusion matrices to visually present the model's performance. Subsequently, the model evaluation agent invokes a robustness testing tool, using generated adversarial examples (including synonym substitution, symbol perturbation, and misspelling injection samples) to construct a dedicated adversarial test set. By calculating the magnitude of performance degradation on the adversarial test set (e.g., the percentage decrease in F1 score), the robustness of the model is quantitatively evaluated.

[0046] Simultaneously, the agent invokes a hard sample mining tool to analyze the model's prediction results on the validation set, filtering out all incorrectly predicted samples and correctly predicted samples with a confidence level below 0.6. These samples are marked as "hard samples," and along with their current model's prediction logits, are fed back to the sample generation agent to provide direction for the next round of sample generation and model optimization. Afterward, the MasterAgent collects experimental scores submitted by the model training agent and robustness reports submitted by the model evaluation agent. Through built-in decision logic, it determines the optimal model configuration: if the candidate optimal model's test set F1 score is greater than 0.98 and its robustness decreases by less than 5%, the model is directly selected; otherwise, among the top three scoring models, the one with the best robustness is selected, and the optimal model's model file, configuration file, training parameters, and evaluation report are saved.

[0047] Once the optimal model configuration is determined, the model training agent receives instructions from the MasterAgent to start production mode, load the configuration information of the optimal model, and retrain the model using the complete standardized training dataset to ensure a thorough training process and improve model performance.

[0048] In this application embodiment, when a single model performs unstablely on a specific type of fraud, this application adopts a multi-model fusion strategy to improve recognition performance and robustness. Specifically, the best model that has just been trained is used as the main model, the model of the previous stable version is used as the auxiliary model, and a fast matching rule base (rule engine) based on regular expressions and keywords is used as a supplement. The final prediction result is determined by weighted voting, and the weights are determined through fusion experiments in the experimental mode, which further ensures the stability and recognition accuracy of the production model.

[0049] In this embodiment, multiple sets of experimental training are carried out through multi-agent collaboration, the optimal configuration is selected through comprehensive evaluation, and multi-model fusion is adopted to obtain a high-performance and robust production model.

[0050] S4. Deploy the production model to the SMS identification system and collect online feedback data. When the online feedback data meets the preset trigger conditions, automatically start a new round of structured fraud strategy element extraction process and iteratively update the production model.

[0051] To continuously optimize the production model and adapt to the latest fraud trends, this application deploys the model and triggers a closed-loop iteration process through online feedback to ensure the accuracy and robustness of identification.

[0052] In this embodiment of the application, this step is completed in collaboration between the online feedback and continuous optimization module (whose role is defined as "online monitoring sentinel" and can call tools such as monitoring online indicators, triggering condition judgment, and collecting feedback samples) and MasterAgent, and the core implementation is the deployment of the production model, online monitoring and closed-loop optimization.

[0053] First, this application encapsulates the trained production model and multi-model fusion strategy into a high-performance inference service, which is then deployed to a real-world SMS identification system to achieve real-time identification of online SMS messages and accurately identify various types of risky and fraudulent SMS messages.

[0054] During model deployment and operation, the system collects various online feedback data in real time, including the model's prediction results for each SMS message (uniformly written to Kafka), fraudulent SMS messages reported by users, and low-confidence samples predicted by the online model, ensuring that the feedback data comprehensively covers various scenarios in which the model operates. To achieve continuous model optimization, this application sets a unified trigger scoring function to determine whether a new round of optimization process needs to be initiated. The trigger scoring function is as follows:

[0055] in, The proportion of samples with low confidence. For the growth rate of user reports, This indicates the frequency of new fraud tactics emerging.

[0056] In this embodiment, when the calculated TriggerScore exceeds a preset threshold, the online feedback and continuous optimization module immediately sends a "retraining" event request to the MasterAgent. After receiving the request, the MasterAgent automatically starts a new round of closed-loop optimization process. This process completely replicates the core steps from S1 to S3 to achieve iterative updates of the production model.

[0057] Specifically, MasterAgent first activates the fraud strategy analysis agent to perform semantic analysis and strategy extraction on new types of fraudulent SMS messages collected online, updating the extracted new structured fraud strategy elements to the fraud strategy knowledge base. Subsequently, the sample generation agent generates diverse training samples covering both new and old strategies based on the updated knowledge base and historical strategies, while incorporating previously reported difficult samples for targeted enhancement. Next, the data processing agent cleans, deduplicates, performs label fusion evaluation, and divides the dataset for the newly generated samples, constructing a new standardized training dataset. Afterward, the model training agent and model evaluation agent sequentially conduct experimental mode training, model evaluation, optimal model selection, and production mode training to generate a new model version incorporating the knowledge of new strategies. Before the new model is generated and deployed, A / B testing is conducted with low traffic (e.g., 5% of online requests) to continuously monitor key indicators such as the false positive rate and recall rate of the new model. Once the gray-scale testing is passed (e.g., stable operation for 24 hours with key indicators superior to or equal to the old model), the traffic is gradually expanded until the old model is fully replaced, achieving rapid response to the latest fraud strategies.

[0058] In this embodiment of the application, the online feedback-closed-loop optimization mechanism ensures that the production model always keeps up with the latest fraud trends and continuously improves the accuracy and robustness of fraudulent text message identification. At the same time, this application will organize the execution log records and intermediate results in the entire process and store them in the backup database as required for future verification, review and further optimization.

[0059] In this embodiment of the application, by deploying the model and combining it with online feedback data to trigger closed-loop optimization, the model is iteratively updated, thereby improving the timeliness and robustness of fraudulent SMS identification.

[0060] Example 2 This invention relates to an optimization system for a multi-agent-based fraudulent SMS identification model, such as... Figure 2 As shown, the system achieves a complete automated process from fraud strategy discovery and automatic generation of training data to continuous model optimization through the collaborative work of multiple intelligent agents.

[0061] The overall system architecture includes the following modules: The system comprises a multi-agent collaborative scheduling module, a fraud strategy analysis agent, a sample generation agent, a data processing agent, a model training agent, a model evaluation agent, and an online feedback and continuous optimization module. These modules are unified through the multi-agent scheduling module for task scheduling and data flow, forming a closed-loop system for the automatic construction and continuous optimization of fraudulent SMS identification models.

[0062] The multi-agent collaborative scheduling module coordinates task execution and data flow among various agents. This module manages the task execution order of each agent, schedules data processing flows, triggers model training tasks, monitors model training status, and manages the training task queue. It automatically triggers a new model training process when a new fraud strategy is detected or model performance deteriorates. Through this scheduling module, collaborative work among multiple agents is achieved, forming an automated model building and optimization process.

[0063] A fraud strategy analysis agent is used to discover new fraud strategies from multi-source data. Data sources include: real-world risky SMS messages from the current network, user-reported SMS messages, SMS messages with low confidence levels in the model, SMS messages with inconsistent identification results from multiple models, and fraud case data obtained through web crawling. This agent extracts fraud strategy information from SMS content using semantic analysis methods and constructs a structured representation model of fraud strategies. A fraud strategy consists of several key elements, including: fraud type, role impersonation, risk triggering event, behavioral inducement method, and target behavior. This information is structured into fraud strategy templates and stored in a fraud strategy knowledge base, forming a reusable fraud strategy structure model for subsequent sample generation.

[0064] The sample generation agent automatically generates fraudulent SMS samples based on strategy information in the fraud strategy knowledge base to expand the fraudulent SMS sample library. The generation process includes: constructing a script template based on the fraud strategy structure, generating SMS text by combining a keyword library, generating multiple semantic variants and adversarial variants, and automatically collecting difficult samples with low confidence and multi-model prediction conflicts. Through the above mechanisms, a diverse fraudulent SMS sample library is formed.

[0065] The data processing agent is responsible for quality control of the generated data. Its main functions include: data deduplication, semantic error detection, abnormal text filtering, and automatic tagging. Each SMS message can be tagged with multiple tags, such as fraud type tags, fraud strategy tags, and risk level tags. The processed data is divided into training and testing sets. To improve the reliability of fraud type tags, a tag confidence calculation method can be used to fuse and evaluate multi-source tags.

[0066] The model training agent is used to automatically train a model for identifying fraudulent text messages. It supports various model architectures, including mainstream and common models such as BERT, RoBERTa, ModernBERT, TextCNN, and Transformer.

[0067] This module supports two training modes: Experimental mode: In experimental mode, the system automatically performs multiple training experiments, including different model structures, learning rates, batch sizes, and training steps. Each experiment is trained quickly within a short, fixed timeframe, and model performance is evaluated using a test set. Simultaneously, the system automatically evaluates the experimental results using a preset model scoring function, such as Score = α × F1 + β × Recall. γ×FalsePositiveRate, and automatically select the optimal model structure and parameter configuration based on the scoring results.

[0068] Production mode: In production mode, the system uses the optimal model configuration obtained from the experimental mode to formally train on the complete training data and generate a production model. In some scenarios, the system can also employ a multi-model fusion strategy to weightedly fuse the prediction results of multiple models, thereby improving recognition performance.

[0069] The model evaluation agent is used to evaluate the performance of the trained model. Evaluation metrics include accuracy, recall, F1 score, and false positive rate. It also performs robustness tests on the model, such as adversarial text recognition, synonym substitution recognition, and symbol perturbation recognition. Furthermore, the evaluation agent automatically identifies difficult samples in the model and feeds them back to the sample generation agent for the next round of training.

[0070] After the model is deployed to the SMS recognition system, the online feedback and continuous optimization module continuously collects online data, including model recognition results, user reports, and SMS messages with low confidence. The system uses a unified trigger scoring function to determine whether to trigger a new training process, which is calculated based on the proportion of low-confidence SMS messages, the growth rate of user reports, and the frequency of new fraud strategies. When the threshold is exceeded, the system automatically starts a new fraud strategy analysis process and updates the fraud strategy knowledge base, thereby triggering new sample generation and model training tasks, achieving a closed loop of continuous model optimization.

[0071] Example 3 This invention provides an optimization device for a multi-agent-based fraudulent SMS identification model, such as... Figure 3 As shown, the device includes: The extraction module 100 is used to collect multi-source SMS data, use the fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a multi-agent unified scheduling mechanism. The generation module 200 is used to automatically generate fraudulent SMS samples based on the structured fraud strategy elements in the updated fraud strategy knowledge base. At the same time, it collects difficult samples, transmits all samples to the data processing agent through a unified scheduling mechanism, and cleans and labels the generated fraudulent SMS samples to build a standardized training dataset. The training module 300 is used for model training agents to perform experimental training based on standardized training datasets, and for model evaluation agents to evaluate the performance of the model during the training process. The model training agents use preset scoring functions to comprehensively evaluate the experimental training results to determine the optimal model configuration and train and generate a production model on complete training data. The update module 400 is used to deploy the production model to the SMS identification system and collect online feedback data. When the online feedback data meets the preset trigger conditions, a new round of structured fraud strategy element extraction process is automatically started to iteratively update the production model.

[0072] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0073] Example 4 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0074] Example 5 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for optimizing a fraudulent SMS identification model based on multi-agent systems, characterized in that, include: S1. Collect multi-source SMS data, use the fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a multi-agent unified scheduling mechanism. S2, based on the structured fraud strategy elements in the updated fraud strategy knowledge base, automatically generates fraud SMS samples, while collecting difficult samples, and transmits all samples to the data processing agent through a unified scheduling mechanism, and cleans and labels the generated fraud SMS samples to build a standardized training dataset. S3, the model training agent performs experimental training based on a standardized training dataset, the model evaluation agent evaluates the performance of the model during the training process, the model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration, and trains and generates a production model on complete training data; S4. Deploy the production model to the SMS identification system and collect online feedback data. When the online feedback data meets the preset trigger conditions, automatically start a new round of structured fraud strategy element extraction process and iteratively update the production model.

2. The method according to claim 1, characterized in that, The process involves collecting multi-source SMS data, using a fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extracting structured fraud strategy elements, updating the fraud strategy knowledge base, and inputting the updated fraud strategy knowledge base into the sample generation agent through a unified multi-agent scheduling mechanism. This includes: We collect real-world risk SMS data, user-reported SMS data, SMS data with low confidence from models, SMS data with inconsistent identification results from multiple models, and fraud case data obtained from web crawlers as multi-source SMS data. Semantic analysis is performed on the multi-source SMS data to extract fraud types, role impersonation, risk triggering events, behavior inducement methods, and fraud target behaviors as elements of structured fraud strategies. The extracted structured fraud strategy elements are used to construct fraud strategy templates and stored in a fraud strategy knowledge base to form a reusable fraud strategy structure model, thereby completing the update of the fraud strategy knowledge base. The updated fraud strategy knowledge base is input into the sample generation agent through the unified scheduling mechanism provided by the multi-agent collaborative scheduling module.

3. The method according to claim 1, characterized in that, The process involves automatically generating fraudulent SMS samples based on structured fraud strategy elements in the updated fraud strategy knowledge base, while simultaneously collecting difficult samples. All samples are then transmitted to the data processing agent through a unified scheduling mechanism. The generated fraudulent SMS samples are then cleaned and multi-labeled to construct a standardized training dataset, including: Read the fraud strategy templates from the updated fraud strategy knowledge base and combine them with a pre-set keyword library to generate basic fraud SMS text. The basic fraudulent SMS text is subjected to character splitting, synonym replacement, and symbol insertion operations to generate various semantic variant samples; The semantic variant samples are perturbed based on an adversarial generation strategy to generate adversarial variant samples. The system automatically collects low-confidence SMS messages and difficult samples with conflicting multi-model predictions generated during the online identification process, and merges the basic fraudulent SMS texts, semantic variant samples, adversarial variant samples, and difficult samples to form a diverse fraudulent SMS sample library. The fraudulent text message sample library is transmitted to a data processing agent, where the samples are cleaned and labeled with multiple tags to construct a standardized training dataset.

4. The method according to claim 1, characterized in that, The model training agent performs experimental training based on a standardized training dataset. A model evaluation agent assesses the model's performance during training. The model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration. A production model is then trained on complete training data, including: The generated fraudulent text message samples are cleaned by performing data deduplication, semantic error detection, and abnormal text filtering operations using a data processing intelligent agent. The cleaned fraudulent text message samples are automatically labeled with fraud type tags, fraud strategy tags, and risk level tags, with each text message sample being labeled with at least one of the above tags; The labeled fraudulent text message samples are divided into training set, validation set and test set to form a standardized training dataset containing a standardized data distribution. Based on a standardized training dataset, the model training agent performs multiple sets of experimental training with different model structures and training parameter combinations. The different model structures include BERT, RoBERTa, ModernBERT, TextCNN, or Transformer models. The model corresponding to each set of experimental parameters is trained quickly for a fixed duration using the validation set, and the accuracy, recall and F1 score of each experimental training result are calculated as performance indicators. The optimal model configuration is determined by comprehensively evaluating the experimental training results using a preset scoring function, and then a production model is generated by training on complete training data using the optimal model configuration.

5. The method according to claim 1, characterized in that, When online feedback data meets preset trigger conditions, a new round of structured fraud strategy element extraction process is automatically initiated, iteratively updating the production model, including: When the online feedback data meets the preset triggering conditions, a new round of processes is automatically started. In production mode, the determined optimal model configuration is called, and the selected model structure is formally trained using the complete training dataset. During the formal full-scale training process, a multi-model fusion strategy is adopted to weight and fuse the prediction results of multiple independently trained models to generate a new production model with higher recognition performance and robustness.

6. A device for optimizing a fraudulent text message identification model based on multi-agent communication, characterized in that, include: The extraction module is used to collect multi-source SMS data, use the fraud strategy analysis agent to perform semantic parsing on the multi-source SMS data, extract structured fraud strategy elements, update the fraud strategy knowledge base, and input the updated fraud strategy knowledge base into the sample generation agent through a unified scheduling mechanism of multiple agents. The generation module is used to automatically generate fraudulent SMS samples based on the structured fraud strategy elements in the updated fraud strategy knowledge base. At the same time, it collects difficult samples, transmits all samples to the data processing agent through a unified scheduling mechanism, and cleans and labels the generated fraudulent SMS samples to build a standardized training dataset. The training module is used to train the model agent to perform experimental training based on a standardized training dataset. The model evaluation agent evaluates the performance of the model during the training process. The model training agent uses a preset scoring function to comprehensively evaluate the experimental training results to determine the optimal model configuration and trains a production model on complete training data. The update module is used to deploy the production model to the SMS identification system and collect online feedback data. When the online feedback data meets the preset trigger conditions, a new round of structured fraud strategy element extraction process is automatically started to iteratively update the production model.

7. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images