A text detection model low-resource distillation method and device based on cold start and reinforcement learning

Abstract

The application relates to a low-resource distillation method and device for a text detection model based on cold start and reinforcement learning. A chain labeling template containing multi-stage inference logic is constructed for a bad text detection task. A structured training dataset with inference budget constraints is generated based on a teacher model. A parameter-efficient fine-tuning paradigm is used to supervise the training of a student model, so that the student model has an inference path and output paradigm that meet preset conditions. An inter-group relative strategy optimization algorithm is called to optimize the strategy of the student model based on a pre-defined multi-dimensional reward function. The optimized student model is compressed and deployed to a resource-constrained end-side device, effectively solving the problem of inference chain breakage of the text detection model in a low-resource environment, ensuring the output format is standard, improving classification accuracy and inference efficiency, and reducing the consumption of computing resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a low-resource distillation method and apparatus for a text detection model based on cold start and reinforcement learning. Background Technology

[0002] In scenarios involving the security governance of malicious text content such as hate speech detection and cyberbullying identification, large language models (teacher models) possess excellent reasoning capabilities, but their massive parameter count (billions to tens of billions) makes them unsuitable for direct deployment on resource-constrained mobile apps, edge gateways, or embedded review terminals. Using lightweight models directly in low-resource scenarios presents the following challenges: Broken reasoning chains – small models struggle to grasp the multi-step reasoning logic of "keyword extraction → cause analysis → classification," resulting in chaotic output formats that business systems cannot directly parse; Poor robustness to boundary samples – small models are prone to over-inference or missed detections for hate speech boundary samples such as metaphors, irony, and dialect variations; Computational and energy constraints – long text reasoning consumes a large number of tokens, leading to device overheating, power consumption, and slow response; Unstable training – traditional reinforcement learning fine-tuning (such as PPO) relies on Critic networks, resulting in high memory consumption and difficulty in convergence, making it impossible to complete model iterations in a low-cost environment. Summary of the Invention

[0003] This application provides a low-resource distillation method and apparatus for text detection models based on cold start and reinforcement learning, in order to solve the problems of traditional reinforcement learning fine-tuning (such as PPO) relying on Critic networks, high memory consumption, difficulty in convergence, and inability to complete model iteration in a low-cost environment.

[0004] Firstly, this application provides a low-resource distillation method for text detection models based on cold start and reinforcement learning, including: For the malicious text detection task, a chain-style annotation template containing multi-stage inference logic is constructed. A structured training dataset with inference budget constraints is generated based on the teacher model. The student model is supervisedly trained using a parameter-efficient fine-tuning paradigm to enable the student model to have an inference path and output paradigm that meet preset conditions. The inter-group relative policy optimization algorithm is invoked to optimize the student model's policy based on a predefined multi-dimensional reward function, which includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. The optimized student model is then lightweighted, compressed, and deployed to resource-constrained edge devices.

[0005] In one possible implementation, keywords with discriminative value are extracted from the input text; based on the keywords, the original content of the input text, and preset classification criteria, an attribution analysis is generated to determine whether there is a malicious tendency; the category label with the highest matching degree with the attribution analysis is selected from a preset label set; and the keyword-attribution analysis-category label is used as a chain annotation template for the multi-stage reasoning logic of the malicious text detection task.

[0006] In one possible implementation, the maximum upper limit of reasoning steps for each reasoning sub-stage of the teacher model is determined; the content of single-step reasoning is encapsulated using hierarchical labels, and the independent output unit of each reasoning sub-stage is marked by start and end identifiers; the output results of the teacher model are encapsulated using a structured data format, and the output results are used as a structured training dataset with reasoning budget constraints. The output results carry keywords, attribution analysis, and category fields, wherein the total number of tokens in the basic reasoning chain in the structured training dataset is less than or equal to a first threshold, the total number of tokens in the extended reasoning chain is less than or equal to a second threshold, and the second threshold is greater than the first threshold.

[0007] In one possible implementation, in the attention layer of the student model, the query vector weight matrix and the value vector weight matrix are injected into the low-rank adaptation module, wherein the low-rank adaptation module is composed of a cascaded dimensionality reduction matrix and an increased dimensionality matrix; during training, the original weight matrix in the student model is frozen, and only the parameters of the low-rank adaptation module are updated with gradients; the student model is trained in a supervised manner using the cross-entropy loss function to complete the cold start training of the student model.

[0008] In one possible implementation, for the same input text, G candidate output sequences are generated by sampling using the current policy, where G is an integer greater than or equal to 2; reward scores are calculated for each of the G candidate output sequences using a multi-dimensional reward function; the reward results are standardized based on the mean and standard deviation of the reward scores to obtain the relative advantage estimate of each candidate output sequence; a shearing mechanism is used to constrain the policy update magnitude, and a KL divergence regularization term is introduced into the objective function to limit the distribution offset between the current policy and the reference policy, thereby optimizing the student model.

[0009] In one possible implementation, for the output format compliance reward, each candidate output sequence is verified to ensure it conforms to a preset structured format and contains all necessary fields, thereby obtaining a compliance reward score for each candidate output sequence. For the classification result accuracy reward, the overlap between the output keywords and standard keywords in each candidate output sequence is determined, as well as the matching degree between the output category and standard category in each candidate output sequence. Based on the overlap and matching degree, an accuracy reward score for each candidate output sequence is obtained. For the inference length reasonableness reward, the deviation between the output sequence length and the target length range in each candidate output sequence is determined, and a reasonableness reward score for each candidate output sequence is obtained based on the deviation.

[0010] In one possible implementation, the input text is preprocessed using a lightweight complexity evaluation model to determine the dynamic inference budget, and localized incremental updates are performed based on user feedback data collected from the edge device.

[0011] Secondly, this application provides a low-resource distillation apparatus for a text detection model based on cold start and reinforcement learning, comprising: The distillation module is used to construct chained annotation templates containing multi-stage reasoning logic for the bad text detection task. It generates a structured training dataset with reasoning budget constraints based on the teacher model and uses a parameter-efficient fine-tuning paradigm to conduct supervised training on the student model so that the student model has a reasoning path and output paradigm that meet the preset conditions. The optimization module is used to call the inter-group relative strategy optimization algorithm to optimize the student model based on a predefined multi-dimensional reward function. The multi-dimensional reward function includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. The deployment module is used to compress the optimized student model into a lightweight form and deploy it to resource-constrained edge devices.

[0012] Thirdly, this application provides a computer device, including: a processor and a memory, wherein the processor is configured to execute a low-resource distillation program for a text detection model based on cold start and reinforcement learning stored in the memory, so as to implement the low-resource distillation method for a text detection model based on cold start and reinforcement learning as described in any one of the first aspects above.

[0013] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the low-resource distillation method for text detection models based on cold start and reinforcement learning as described in any of the fourth aspects above.

[0014] This application provides a low-resource distillation method for text detection models based on cold start and reinforcement learning. It constructs a chain-like annotation template containing multi-stage inference logic for the malicious text detection task. A structured training dataset with inference budget constraints is generated based on the teacher model. A supervised training method using an efficient parameter fine-tuning paradigm is employed to ensure the student model possesses an inference path and output paradigm that meets preset conditions. An inter-group relative policy optimization algorithm is invoked to optimize the student model's policy based on a predefined multi-dimensional reward function. The optimized student model is then lightweighted, compressed, and deployed to resource-constrained edge devices. The chain-like annotation template and efficient parameter fine-tuning ensure the stability of the student model's inference path. The combination of the inter-group relative policy optimization algorithm and the multi-dimensional reward function improves output quality and efficiency, achieving efficient deployment in low-resource environments. This effectively solves the problem of inference chain breakage in text detection models under low-resource conditions, ensuring standardized output format, improving classification accuracy and inference efficiency, while reducing computational resource consumption. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 This application provides a flowchart illustrating a low-resource distillation method for text detection based on cold start and reinforcement learning. Figure 2 This application provides a flowchart illustrating the structured thought chain cold-start distillation process in a low-resource distillation method for text detection based on cold start and reinforcement learning. Figure 3 This application provides a flowchart illustrating the policy optimization process in a low-resource distillation method for text detection based on cold start and reinforcement learning. Figure 4 This application provides a schematic diagram of a low-resource distillation device for a text detection model based on cold start and reinforcement learning; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] Figure 1 This application provides a flowchart illustrating a low-resource distillation method for text detection based on cold start and reinforcement learning, as shown below. Figure 1 The method specifically includes: S11. Construct a chain-style annotation template containing multi-stage reasoning logic for the malicious text detection task. Generate a structured training dataset with reasoning budget constraints based on the teacher model. Use a parameter-efficient fine-tuning paradigm to conduct supervised training on the student model so that the student model has a reasoning path and output paradigm that meet the preset conditions.

[0022] In this embodiment of the application, the task of detecting harmful text refers to the task of identifying and classifying hate speech, cyberbullying, pornography, illegal content, and other content in text content that does not conform to social norms or laws and regulations.

[0023] Furthermore, a chain-like annotation template incorporating multi-stage reasoning logic is constructed for the malicious text detection task. In practical applications, the identification of malicious text often requires the model to perform multi-step logical reasoning. For example, firstly, sensitive words in the text are identified, then the specific meaning of these words in the context is analyzed, and finally, a final classification judgment is given based on the context. To achieve this complex reasoning process, human expert experience can be used to design a simple two-stage template. For example, the first stage requires the model to directly output keywords in the text that may contain malicious tendencies, and the second stage requires the model to directly provide the final classification label based on these keywords.

[0024] After obtaining the aforementioned chained annotation template, a powerful teacher model is used to generate a large amount of training data. For example, raw text can be input into the teacher model, which is then asked to reason according to a preset template and output corresponding keywords, attribution analysis, and classification results. During the generation process, no restrictions can be imposed on the length of the teacher model's inference, allowing it to freely generate inference steps of arbitrary length based on the complexity of the text.

[0025] After obtaining the structured training dataset, the student model needs to undergo initial training to learn the knowledge from the teacher model. For example, full parameter fine-tuning can be used to update the weight parameters of the student model.

[0026] Furthermore, only the output layer of the student model can be fine-tuned while the parameters of its backbone network are frozen. Since only a small number of parameters at the end of the model are updated, the student model may not be able to fully learn the deep knowledge of the teacher model in multi-stage reasoning logic, resulting in deviations between its reasoning path and output paradigm and the preset conditions. In subsequent steps, strategy optimization is used to adjust the student model to eliminate the deviations.

[0027] S12. Call the inter-group relative policy optimization algorithm to optimize the student model based on the predefined multi-dimensional reward function.

[0028] In this embodiment, after the student model completes supervised training, reinforcement learning can be introduced for optimization to further improve its performance and robustness. For example, a traditional policy gradient algorithm can be used to update the student model's policy based on a single reward signal (such as the final classification accuracy).

[0029] Furthermore, a reward function can be designed that includes multiple reward components, but these components do not have explicit weights or interaction mechanisms; they are simply added together to obtain the total reward. Such a multi-dimensional reward function should include at least: a reward for compliance with output format, a reward for accuracy of classification results, and a reward for reasonable inference length.

[0030] S13. Compress the optimized student model into a lightweight form and deploy it to a resource-constrained edge device.

[0031] Specifically, after the training and optimization processes described above, the student model is deployed to actual edge devices. For example, the trained student model can be directly deployed to a mobile terminal. However, even after distillation, the number of parameters and computational complexity of the student model may still exceed the resource limitations of the edge device, resulting in slow inference speed, high energy consumption, or failure to function properly. As an alternative approach, a simple model pruning technique can be used to randomly remove some unimportant connections or neurons from the model. This method can reduce the model size to some extent, enabling lightweight deployment.

[0032] This application provides a low-resource distillation method for text detection models based on cold start and reinforcement learning. It constructs a chain-like annotation template containing multi-stage inference logic for the malicious text detection task. A structured training dataset with inference budget constraints is generated based on the teacher model. A supervised training method using an efficient parameter fine-tuning paradigm is employed to ensure the student model possesses an inference path and output paradigm that meets preset conditions. An inter-group relative policy optimization algorithm is invoked to optimize the student model's policy based on a predefined multi-dimensional reward function. The optimized student model is then lightweighted, compressed, and deployed to resource-constrained edge devices. The chain-like annotation template and efficient parameter fine-tuning ensure the stability of the student model's inference path. The combination of the inter-group relative policy optimization algorithm and the multi-dimensional reward function improves output quality and efficiency, achieving efficient deployment in low-resource environments. This effectively solves the problem of inference chain breakage in text detection models under low-resource conditions, ensuring standardized output format, improving classification accuracy and inference efficiency, while reducing computational resource consumption.

[0033] In one optional embodiment of this application, the input text is preprocessed using a lightweight complexity evaluation model to determine the dynamic inference budget, and localized incremental updates are performed based on user feedback data collected from the edge device.

[0034] This application's solution, based on the aforementioned low-resource distillation method for text detection models using cold start and reinforcement learning, further optimizes the model's operational efficiency and long-term adaptability after deployment on edge devices. Specifically, the workflow of the student model is enhanced after deployment to resource-constrained edge devices.

[0035] First, upon receiving new input text, it undergoes preprocessing using a lightweight complexity evaluation model. This model quickly analyzes the features of the input text and determines a dynamic inference budget. This dynamic budget is then passed to the student model, guiding its inference process when performing malicious text detection tasks. For example, for text evaluated as low-complexity, the student model is allocated fewer inference steps or allowed to generate fewer tokens, thus avoiding unnecessary computational resource consumption; while for more complex text, a more sufficient inference budget is allocated to ensure inference accuracy. This mechanism allows the student model to flexibly adjust resource consumption based on the actual input text, solving the resource waste problem caused by the fixed inference budget in the original solution, significantly reducing the power consumption of the edge device and improving response speed. Simultaneously, during the student model's operation, the edge device continuously collects user feedback data. This feedback data reflects the model's performance in practical applications, such as user corrections or confirmations of detection results. After accumulating a certain amount of feedback data, the student model performs an incremental update locally on the edge device. This localized incremental update utilizes newly collected feedback data to fine-tune the model parameters, enabling it to learn new malicious text patterns or correct historical errors. Because the update process is completed locally, there is no need to upload raw user data to the cloud, effectively protecting user privacy. Furthermore, this continuous learning mechanism allows the student model to constantly adapt to changing business needs and new types of malicious text, solving the problem of fixed models after deployment and the inability to continuously optimize them, thus ensuring the model's detection performance and robustness in long-term use.

[0036] Through the aforementioned technical means, the proposed solution is organically combined with the fundamental distillation method. The fundamental method, through cold start and reinforcement learning, enables the student model to possess multi-stage inference logic and structured output capabilities in low-resource environments. Building upon this, a dynamic inference budget mechanism further optimizes resource allocation during the inference phase, allowing for more refined management of edge computing power while maintaining high accuracy. The localized incremental update mechanism ensures that the student model trained through initial distillation can continuously evolve, overcoming the challenge of performance degradation after model deployment. This combination enables the student model to not only possess high performance and low resource consumption during deployment but also maintain its effectiveness and adaptability throughout its entire lifecycle, thereby achieving efficient, continuous, and privacy-friendly malicious text detection on resource-constrained edge devices.

[0037] Figure 2 This application provides a flowchart illustrating the structured thought chain cold-start distillation process in a low-resource distillation method for text detection based on cold start and reinforcement learning, as shown in the diagram. Figure 2 The method specifically includes: S21. Extract keywords with discriminative value from the input text.

[0038] S22. Based on the keywords, the original content of the input text, and the preset classification criteria, generate an attribution analysis to determine whether there is a negative tendency.

[0039] S23. Select the category label with the highest matching degree with the attribution analysis from the preset label set.

[0040] S24. Use keyword-attribution analysis-category label as a chain annotation template for the multi-stage reasoning logic of the malicious text detection task.

[0041] In this embodiment, for the malicious text detection task, a chain-like annotation template containing multi-stage reasoning logic is pre-constructed. Specifically, the template is designed with three stages: the first stage extracts keywords with discriminative value from the input text; the second stage generates an attribution analysis based on the keywords, the original text content, and preset classification criteria to determine whether there is a malicious tendency; and the third stage selects the category label with the highest matching degree to the attribution analysis from a preset label set. This provides a clear and structured logical chain for subsequent model reasoning.

[0042] S25. Determine the maximum upper limit of reasoning steps for each reasoning sub-stage of the teacher model.

[0043] S26. Hierarchical labels are used to encapsulate the content of single-step reasoning, and the independent output unit of each reasoning sub-stage is marked by start and end identifiers.

[0044] S27. The output of the teacher model is encapsulated in a structured data format and used as a structured training dataset with inference budget constraints. The output carries keywords, attribution analysis, and category fields.

[0045] Furthermore, a teacher model with a large number of parameters was used to process a large amount of raw text data. During the processing, the teacher model strictly followed the aforementioned chained annotation template for multi-stage reasoning and generated structured outputs such as keywords, attribution analysis, and category labels.

[0046] Meanwhile, to accommodate the resource limitations of edge devices, an upper limit was imposed on the maximum number of inference steps in each inference sub-stage of the teacher model during the data generation phase. For example, keyword extraction was limited to a maximum of 8 steps, and causal analysis to a maximum of 3 steps. This ensured that the generated dataset not only contained high-quality structured inference information, but also that its inference length met the preset resource budget.

[0047] Wherein, the total number of tokens in the basic inference chain in the structured training dataset is less than or equal to a first threshold, the total number of tokens in the extended inference chain is less than or equal to a second threshold, and the second threshold is greater than the first threshold.

[0048] In one example: Input text 1 (simple): "You piece of trash, get out"; Complexity assessment: Text length = 10, no rhetorical devices, sentiment polarity = -0.85, classified as simple text. Invoking the basic inference chain: Keyword "trash" → Attribution "direct insult" → Category "hate speech - insult" Inference budget: 32 tokens (8 keywords + 20 attributions + 4 categories), below the first threshold of 80.

[0049] Input text 2 (complex): "Some people are just a natural burden, a waste of resources to live"; Complexity assessment: Text length = 16, metaphorical rhetoric detected ("redundant" mapping to a group of people), sentiment polarity = -0.62, classified as complex text. Invoking the extended inference chain: Rhetorical identification “metaphor: redundant refers to a specific group of people” → Context reconstruction “certain groups of people are described as worthless” → Keywords “redundant” “waste of resources” → Attribution “degrading a specific group of people through metaphor” → Category “hate speech - degrading category”; Inference budget: 128 tokens, below the second threshold of 160.

[0050] S28. In the attention layer of the student model, the query vector weight matrix and the value vector weight matrix are injected into the low-rank adaptation module, wherein the low-rank adaptation module is composed of a cascaded dimensionality reduction matrix and an dimensionality increase matrix.

[0051] S29. During training, the original weight matrix in the student model is frozen, and gradient updates are performed only on the parameters of the low-rank adaptation module.

[0052] S210. The student model is trained in a supervised manner using the cross-entropy loss function to complete the cold start training of the student model.

[0053] Furthermore, a parameter-efficient fine-tuning paradigm is employed to perform supervised training on the student model, enabling it to possess inference paths and output paradigms that conform to preset conditions. After obtaining the aforementioned structured training dataset, a lightweight student model is initialized.

[0054] During training, a parameter-efficient fine-tuning paradigm is employed. For example, only in the attention layer of the student model are the query vector weight matrix and value vector weight matrix injected into the low-rank adaptation module, and their parameters are updated, while the original weight matrix is ​​frozen. In this way, the student model can efficiently learn multi-stage inference logic and structured output format from the data generated by the teacher model, thus initially possessing the ability to detect bad text during the cold start phase, and forming an inference path and output paradigm that conforms to preset conditions.

[0055] In one example, a LoRA adapter is introduced into the Query (Q) and Value (V) weight matrices in the Attention substructure of the student model θ: W = W0 + ΔW = W0 + BA; Where W0 is the original weight matrix, ΔW is the weight matrix that needs to be adjusted in the student model, A is the increment of the query vector weight matrix, and B is the increment of the value vector weight matrix.

[0056] After the cold start, the LoRA incremental weights A and B are merged into the original weight matrix W0 and exported in a standard format (Core ML, TFLite, ONNX). The student model parameter size can be compressed to 1 / 2.67 of the teacher model (e.g., from 7B to 2.6B), and the merged model size is <500 MB (<250 MB after INT8 quantization), meeting the requirements for iOS / Android app submission. On-device inference engines (such as ML Kit, Snapdragon NPU SDK) can be loaded directly, with a first token latency of <200 ms, significantly accelerating model inference and reducing resource consumption.

[0057] Figure 3 This application provides a flowchart illustrating the policy optimization process in a low-resource distillation method for text detection based on cold start and reinforcement learning, as shown in the following diagram. Figure 3 The method specifically includes: S31. For the same input text, G candidate output sequences are generated by sampling using the current strategy, where G is an integer greater than or equal to 2.

[0058] In this application, the output capability of the student model is optimized and improved by using GRPO enhanced distillation and a lightweight reward function.

[0059] After the student model completes its cold-start training, a reinforcement learning phase is introduced to further improve its robustness and output quality on complex malicious text. For a piece of malicious text posted by user A that may contain metaphors or irony, the student model will sample and generate G candidate output sequences based on the current policy.

[0060] S32. Calculate the reward score for each of the G candidate output sequences using a multi-dimensional reward function.

[0061] Furthermore, the G candidate sequences are then fed into a predefined multi-dimensional reward function for evaluation. This reward function includes at least: an output format compliance reward, used to verify whether each candidate sequence conforms to the preset structured format and necessary fields; a classification result accuracy reward, used to evaluate the overlap between the output keywords and standard keywords and the matching degree between the output category and the standard category; and an inference length reasonableness reward, used to measure the deviation between the length of the output sequence and the target length range.

[0062] In one optional embodiment, for the output format compliance reward, each candidate output sequence is verified to ensure it conforms to a preset structured format and contains all necessary fields, thereby obtaining a compliance reward score for each candidate output sequence. For the classification result accuracy reward, the overlap between the output keywords and standard keywords in each candidate output sequence is determined, as well as the matching degree between the output category and standard category in each candidate output sequence. Based on the overlap and matching degree, an accuracy reward score for each candidate output sequence is obtained. For the inference length reasonableness reward, the deviation between the output sequence length and the target length range in each candidate output sequence is determined, and a reasonableness reward score for each candidate output sequence is obtained based on the deviation.

[0063] In one example, (1) Output format compliance reward: verify whether the output is a valid JSON and contains the specified field. If it meets the requirements, get 1.0; otherwise, get 0. (2) Classification result accuracy reward: use the weighted average of Jaccard similarity and F1-Score to evaluate the matching degree between the classification category and the standard answer. (3) Reasonable reasoning length reward: constrain computing power consumption. If the output length is within ±10% of the target length range, get 1.0; if it is within ±20%, get 0.5; otherwise, get 0.

[0064] S33. Standardize the reward results based on the mean and standard deviation of the reward scores to obtain the relative advantage estimate of each candidate output sequence.

[0065] S34. A shearing mechanism is used to constrain the policy update magnitude, and a KL divergence regularization term is introduced into the objective function to limit the distribution offset between the current policy and the reference policy, thereby optimizing the policy of the student model.

[0066] Specifically, by standardizing these reward scores, the relative advantage estimates of each candidate output sequence are obtained, which guide the policy update of the student model. At the same time, a shearing mechanism is used to constrain the policy update magnitude, and a KL divergence regularization term is introduced into the objective function to limit the distribution shift between the current policy and the reference policy, thereby optimizing the policy of the student model and ensuring the stability and effectiveness of training.

[0067] Furthermore, when the GRPO-trained student model is generated on the edge device, the output format compliance reward ensures that the output can be directly consumed by the JSON parser of the business system without regular expression cleaning; the inference length reasonableness reward enables the model to learn to generate short inference chains (<50 tokens) on simple samples and moderately expand them (<200 tokens) on complex samples, reducing token consumption by an average of 30%–50%, which is equivalent to reducing NPU runtime and heat generation. The student model achieves a classification accuracy of 96.49% of the teacher model, realizing near-cloud-level detection capabilities on low-resource edge devices.

[0068] Figure 4 This application provides a schematic diagram of a low-resource distillation device for a text detection model based on cold start and reinforcement learning, as shown in the diagram. Figure 4 Specifically, it includes: Distillation module 401 is used to construct a chain-style annotation template containing multi-stage reasoning logic for the bad text detection task. It generates a structured training dataset with reasoning budget constraints based on the teacher model and uses a parameter-efficient fine-tuning paradigm to conduct supervised training on the student model so that the student model has a reasoning path and output paradigm that meet the preset conditions. Optimization module 402 is used to call the inter-group relative strategy optimization algorithm to optimize the student model based on a predefined multi-dimensional reward function. The multi-dimensional reward function includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. Deployment module 403 is used to compress the optimized student model into a lightweight form and deploy it to resource-constrained edge devices.

[0069] This application provides a low-resource distillation device for text detection models based on cold start and reinforcement learning. The distillation module constructs a chain-like annotation template containing multi-stage inference logic for the malicious text detection task. A structured training dataset with inference budget constraints is generated based on the teacher model. A parameter-efficient fine-tuning paradigm is used for supervised training of the student model to ensure it possesses an inference path and output paradigm that meets preset conditions. The optimization module calls an inter-group relative policy optimization algorithm to optimize the student model's policy based on a predefined multi-dimensional reward function. The deployment module lightweight compresses the optimized student model and deploys it to resource-constrained edge devices. The chain-like annotation template and parameter-efficient fine-tuning ensure the stability of the student model's inference path. The combination of the inter-group relative policy optimization algorithm and the multi-dimensional reward function improves output quality and efficiency, achieving efficient deployment in low-resource environments. This effectively solves the problem of inference chain breakage in text detection models under low-resource conditions, ensuring standardized output format, improving classification accuracy and inference efficiency, while reducing computational resource consumption.

[0070] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 5 The computer device 500 shown includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. Various components in the mobile terminal 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general designated all buses as Bus System 505.

[0071] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0072] It is understood that the memory 502 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0073] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0074] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in application program 5022.

[0075] In this embodiment, by calling the program or instructions stored in memory 502, specifically the program or instructions stored in application program 5022, processor 501 executes the method steps provided in each method embodiment, including, for example: For the malicious text detection task, a chain-style annotation template containing multi-stage inference logic is constructed. A structured training dataset with inference budget constraints is generated based on the teacher model. The student model is supervisedly trained using a parameter-efficient fine-tuning paradigm to enable the student model to have an inference path and output paradigm that meet preset conditions. The inter-group relative policy optimization algorithm is invoked to optimize the student model's policy based on a predefined multi-dimensional reward function, which includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. The optimized student model is then lightweighted, compressed, and deployed to resource-constrained edge devices.

[0076] In one possible implementation, keywords with discriminative value are extracted from the input text; based on the keywords, the original content of the input text, and preset classification criteria, an attribution analysis is generated to determine whether there is a malicious tendency; the category label with the highest matching degree with the attribution analysis is selected from a preset label set; and the keyword-attribution analysis-category label is used as a chain annotation template for the multi-stage reasoning logic of the malicious text detection task.

[0077] In one possible implementation, the maximum upper limit of reasoning steps for each reasoning sub-stage of the teacher model is determined; the content of single-step reasoning is encapsulated using hierarchical labels, and the independent output unit of each reasoning sub-stage is marked by start and end identifiers; the output results of the teacher model are encapsulated using a structured data format, and the output results are used as a structured training dataset with reasoning budget constraints, the output results carrying keywords, attribution analysis, and category fields.

[0078] In one possible implementation, in the attention layer of the student model, the query vector weight matrix and the value vector weight matrix are injected into the low-rank adaptation module, wherein the low-rank adaptation module is composed of a cascaded dimensionality reduction matrix and an increased dimensionality matrix; during training, the original weight matrix in the student model is frozen, and only the parameters of the low-rank adaptation module are updated with gradients; the student model is trained in a supervised manner using the cross-entropy loss function to complete the cold start training of the student model.

[0079] In one possible implementation, for the same input text, G candidate output sequences are generated by sampling using the current policy, where G is an integer greater than or equal to 2; reward scores are calculated for each of the G candidate output sequences using a multi-dimensional reward function; the reward results are standardized based on the mean and standard deviation of the reward scores to obtain the relative advantage estimate of each candidate output sequence; a shearing mechanism is used to constrain the policy update magnitude, and a KL divergence regularization term is introduced into the objective function to limit the distribution offset between the current policy and the reference policy, thereby optimizing the student model.

[0080] In one possible implementation, for the output format compliance reward, each candidate output sequence is verified to ensure it conforms to a preset structured format and contains all necessary fields, thereby obtaining a compliance reward score for each candidate output sequence. For the classification result accuracy reward, the overlap between the output keywords and standard keywords in each candidate output sequence is determined, as well as the matching degree between the output category and standard category in each candidate output sequence. Based on the overlap and matching degree, an accuracy reward score for each candidate output sequence is obtained. For the inference length reasonableness reward, the deviation between the output sequence length and the target length range in each candidate output sequence is determined, and a reasonableness reward score for each candidate output sequence is obtained based on the deviation.

[0081] In one possible implementation, the input text is preprocessed using a lightweight complexity evaluation model to determine the dynamic inference budget, and localized incremental updates are performed based on user feedback data collected from the edge device.

[0082] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in processor 501. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0083] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0084] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0085] The computer device provided in this embodiment may be as follows: Figure 5 The computer device shown can perform, for example Figure 1-2 This paper describes all steps of a low-resource distillation method for text detection models based on cold start and reinforcement learning, thereby achieving... Figure 1-2 The technical performance of the low-resource distillation method for text detection models based on cold start and reinforcement learning is shown below. Please refer to [link / reference] for details. Figure 1-2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0086] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0087] When one or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned low-resource distillation method for text detection models based on cold start and reinforcement learning, which is executed on the low-resource distillation device side of the text detection model based on cold start and reinforcement learning.

[0088] The processor is used to execute a low-resource distillation program for a text detection model based on cold start and reinforcement learning stored in memory, to implement the following steps of a low-resource distillation method for a text detection model based on cold start and reinforcement learning executed on the device side: For the malicious text detection task, a chain-style annotation template containing multi-stage inference logic is constructed. A structured training dataset with inference budget constraints is generated based on the teacher model. The student model is supervisedly trained using a parameter-efficient fine-tuning paradigm to enable the student model to have an inference path and output paradigm that meet preset conditions. The inter-group relative policy optimization algorithm is invoked to optimize the student model's policy based on a predefined multi-dimensional reward function, which includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. The optimized student model is then lightweighted, compressed, and deployed to resource-constrained edge devices.

[0089] In one possible implementation, keywords with discriminative value are extracted from the input text; based on the keywords, the original content of the input text, and preset classification criteria, an attribution analysis is generated to determine whether there is a malicious tendency; the category label with the highest matching degree with the attribution analysis is selected from a preset label set; and the keyword-attribution analysis-category label is used as a chain annotation template for the multi-stage reasoning logic of the malicious text detection task.

[0090] In one possible implementation, the maximum upper limit of reasoning steps for each reasoning sub-stage of the teacher model is determined; the content of single-step reasoning is encapsulated using hierarchical labels, and the independent output unit of each reasoning sub-stage is marked by start and end identifiers; the output results of the teacher model are encapsulated using a structured data format, and the output results are used as a structured training dataset with reasoning budget constraints, the output results carrying keywords, attribution analysis, and category fields.

[0091] In one possible implementation, in the attention layer of the student model, the query vector weight matrix and the value vector weight matrix are injected into the low-rank adaptation module, wherein the low-rank adaptation module is composed of a cascaded dimensionality reduction matrix and an increased dimensionality matrix; during training, the original weight matrix in the student model is frozen, and only the parameters of the low-rank adaptation module are updated with gradients; the student model is trained in a supervised manner using the cross-entropy loss function to complete the cold start training of the student model.

[0092] In one possible implementation, for the same input text, G candidate output sequences are generated by sampling using the current policy, where G is an integer greater than or equal to 2; reward scores are calculated for each of the G candidate output sequences using a multi-dimensional reward function; the reward results are standardized based on the mean and standard deviation of the reward scores to obtain the relative advantage estimate of each candidate output sequence; a shearing mechanism is used to constrain the policy update magnitude, and a KL divergence regularization term is introduced into the objective function to limit the distribution offset between the current policy and the reference policy, thereby optimizing the student model.

[0093] In one possible implementation, for the output format compliance reward, each candidate output sequence is verified to ensure it conforms to a preset structured format and contains all necessary fields, thereby obtaining a compliance reward score for each candidate output sequence. For the classification result accuracy reward, the overlap between the output keywords and standard keywords in each candidate output sequence is determined, as well as the matching degree between the output category and standard category in each candidate output sequence. Based on the overlap and matching degree, an accuracy reward score for each candidate output sequence is obtained. For the inference length reasonableness reward, the deviation between the output sequence length and the target length range in each candidate output sequence is determined, and a reasonableness reward score for each candidate output sequence is obtained based on the deviation.

[0094] In one possible implementation, the input text is preprocessed using a lightweight complexity evaluation model to determine the dynamic inference budget, and localized incremental updates are performed based on user feedback data collected from the edge device.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0098] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A low-resource distillation method for text detection models based on cold start and reinforcement learning, characterized in that, The method includes: For the task of detecting malicious text, a chain-style annotation template containing multi-stage reasoning logic is constructed. A structured training dataset with reasoning budget constraints is generated based on the teacher model. The student model is supervised training using a parameter-efficient fine-tuning paradigm so that the student model has a reasoning path and output paradigm that meet the preset conditions. The inter-group relative strategy optimization algorithm is invoked to optimize the student model based on a predefined multi-dimensional reward function, which includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. The optimized student model is then lightweighted, compressed, and deployed to resource-constrained edge devices.

2. The method according to claim 1, characterized in that, The chained annotation template for malicious text detection tasks, which includes multi-stage reasoning logic, includes: Extract keywords with discriminative value from the input text; Based on the keywords, the original content of the input text, and the preset classification criteria, an attribution analysis is generated to determine whether there is a negative tendency. Select the category label with the highest matching degree to the attribution analysis from the preset label set; The keyword-attribution analysis-category label is used as a chain-style annotation template for the multi-stage reasoning logic of the malicious text detection task.

3. The method according to claim 1, characterized in that, The structured training dataset with inference budget constraints generated based on the teacher model includes: Determine the maximum upper limit of reasoning steps for each reasoning sub-stage of the teacher model; Hierarchical tags are used to encapsulate the content of single-step reasoning, and start and end identifiers are used to mark the independent output units of each reasoning sub-stage. The output of the teacher model is encapsulated in a structured data format and used as a structured training dataset with inference budget constraints. The output includes keywords, attribution analysis, and category fields. Wherein, the total number of tokens in the basic inference chain in the structured training dataset is less than or equal to a first threshold, the total number of tokens in the extended inference chain is less than or equal to a second threshold, and the second threshold is greater than the first threshold.

4. The method according to claim 1, characterized in that, The method of employing a parameter-efficient fine-tuning paradigm for supervised training of the student model includes: In the attention layer of the student model, the query vector weight matrix and the value vector weight matrix are injected into the low-rank adaptation module, wherein the low-rank adaptation module is composed of a cascaded dimensionality reduction matrix and an dimensionality increase matrix. During training, the original weight matrix in the student model is frozen, and gradient updates are performed only on the parameters of the low-rank adaptation module. The student model is trained in a supervised manner using the cross-entropy loss function to complete the cold start training of the student model.

5. The method according to claim 1, characterized in that, The aforementioned inter-group relative policy optimization algorithm optimizes the student model based on a predefined multi-dimensional reward function, including: For the same input text, G candidate output sequences are generated by sampling using the current policy, where G is an integer greater than or equal to 2; The reward score is calculated for each of the G candidate output sequences using a multi-dimensional reward function. The reward results are standardized based on the mean and standard deviation of the reward scores to obtain the relative advantage estimate of each candidate output sequence; A shearing mechanism is used to constrain the policy update magnitude, and a KL divergence regularization term is introduced into the objective function to limit the distribution offset between the current policy and the reference policy, thereby optimizing the policy of the student model.

6. The method according to claim 5, characterized in that, The step of calculating reward scores for each of the G candidate output sequences using a multi-dimensional reward function includes: For the output format compliance reward, each candidate output sequence is verified to see if it conforms to a preset structured format and whether it contains all the necessary fields, and a compliance reward score is obtained for each candidate output sequence. To determine the accuracy reward for the classification results, the overlap between the output keywords and standard keywords in each candidate output sequence is determined, as well as the matching degree between the output category and standard category in each candidate output sequence is determined. Based on the overlap and the matching degree, the accuracy reward score for each candidate output sequence is obtained. For the reasoning length reasonableness reward, the deviation between the output sequence length and the target length interval in each candidate output sequence is determined, and a reasonableness reward score for each candidate output sequence is obtained based on the deviation.

7. The method according to claim 1, characterized in that, The method further includes: The input text is preprocessed using a lightweight complexity evaluation model to determine the dynamic inference budget, and localized incremental updates are performed based on user feedback data collected from the edge device.

8. A low-resource distillation device for a text detection model based on cold start and reinforcement learning, characterized in that, The device includes: The distillation module is used to construct chained annotation templates containing multi-stage reasoning logic for the bad text detection task. It generates a structured training dataset with reasoning budget constraints based on the teacher model and uses a parameter-efficient fine-tuning paradigm to conduct supervised training on the student model so that the student model has a reasoning path and output paradigm that meet the preset conditions. The optimization module is used to call the inter-group relative strategy optimization algorithm to optimize the student model based on a predefined multi-dimensional reward function. The multi-dimensional reward function includes at least: output format compliance reward, classification result accuracy reward, and inference length reasonableness reward. The deployment module is used to compress the optimized student model into a lightweight form and deploy it to resource-constrained edge devices.

9. A computer device, characterized in that, include: A processor and a memory, the processor being configured to execute a low-resource distillation program for a text detection model based on cold start and reinforcement learning stored in the memory, to implement the low-resource distillation method for a text detection model based on cold start and reinforcement learning as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the low-resource distillation method for text detection models based on cold start and reinforcement learning as described in any one of claims 1-7.

Images