Model adjustment methods, information classification methods, and related equipment

By adjusting and pruning the sample information of the pre-trained model, the problems of excessive model size and number of parameters were solved, and the model inference speed was improved.

CN123117232APending Publication Date: 2026-11-17TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510628901.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, the size and number of parameters of the fine-tuned model are too large, resulting in slow model inference speed.

Method used

By acquiring sample information from the pre-trained information processing model and the target classification task, model adjustment and pruning are performed to obtain a basic information classification model and a pruned model adapted to the target task. Fine-tuning is then performed on the pruned model to reduce unnecessary parameters.

Benefits of technology

While maintaining model performance, the model size and number of parameters are significantly reduced, thereby improving the model inference speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117232A_ABST
    Figure CN123117232A_ABST
Patent Text Reader

Abstract

This application discloses a model adjustment method, an information classification method, and related equipment. The application acquires multiple sample information from a pre-trained information processing model and a target classification task; adjusts the pre-trained information processing model using the multiple sample information to obtain a basic information classification model corresponding to the target classification task; prunes the model to be processed to obtain a pruned model; when the model to be processed includes the basic information classification model, adjusts the pruned model using the multiple sample information to obtain a first information classification model; when the model to be processed includes the pre-trained information processing model, adjusts the pruned model using the basic information classification model and the multiple sample information to obtain a second information classification model. Therefore, this application can reduce the model size and number of parameters while ensuring model performance, and improve the model inference speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model adjustment method, an information classification method, and related equipment. Background Technology

[0002] Model fine-tuning refers to retraining a pre-trained model to adapt it to a specific task or dataset. Fine-tuning is typically performed on existing large-scale pre-trained models by performing minimal training on smaller, task-specific datasets to adjust some of the model's parameters, enabling it to perform better on the specific task.

[0003] In existing technologies, fine-tuned models often retain the complete structure and hundreds of millions of parameters of the original model. Therefore, although model fine-tuning can preserve the powerful capabilities of the pre-trained model, the excessively large model size and number of parameters after fine-tuning can lead to an overload on the model during inference and slow inference speed. Summary of the Invention

[0004] This application provides a model adjustment method, an information classification method, and related equipment, which can reduce the size and number of parameters of the model and improve the model inference speed while ensuring model performance.

[0005] This application provides a model adjustment method, comprising: acquiring multiple sample information of a pre-trained information processing model and a target classification task; adjusting the pre-trained information processing model using the multiple sample information to obtain a basic information classification model corresponding to the target classification task; pruning the model to be processed to obtain a pruned model, wherein the model to be processed includes either the pre-trained information processing model or the basic information classification model; when the model to be processed includes the basic information classification model, adjusting the pruned model using the multiple sample information to obtain a first information classification model, wherein the first information classification model is used to perform the target classification task; when the model to be processed includes the pre-trained information processing model, adjusting the pruned model using the basic information classification model and the multiple sample information to obtain a second information classification model, wherein the second information classification model is used to perform the target classification task.

[0006] This application provides an information classification method, including: acquiring information to be processed and a target information classification model, wherein the target information classification model is obtained by any model adjustment method or information classification method provided in this application to obtain a first information classification model or a second information classification model; and using the target information classification model to classify and predict the information to be processed to obtain the predicted information category corresponding to the target classification task.

[0007] This application embodiment also provides a model adjustment device, comprising: a training acquisition unit for acquiring multiple sample information of a pre-trained information processing model and a target classification task; a first adjustment unit for adjusting the pre-trained information processing model using the multiple sample information to obtain a basic information classification model corresponding to the target classification task; a pruning unit for pruning the model to be processed to obtain a pruned model, wherein the model to be processed includes either the pre-trained information processing model or the basic information classification model; a second adjustment unit for adjusting the pruned model using the multiple sample information when the model to be processed includes the basic information classification model to obtain a first information classification model, wherein the first information classification model is used to perform the target classification task; and a third adjustment unit for adjusting the pruned model using the basic information classification model and the multiple sample information when the model to be processed includes the pre-trained information processing model to obtain a second information classification model, wherein the second information classification model is used to perform the target classification task.

[0008] In some embodiments, the model adjustment device further includes a sample processing unit, which includes a first processing subunit and a second processing subunit, comprising: the first processing subunit for acquiring multiple initial sample information; and the second processing subunit for performing sample processing on the initial sample information to obtain sample information.

[0009] In some implementations, the initial sample information includes composite text, and the second processing subunit is specifically used to: determine the content to be processed from the initial sample information, wherein the content to be processed is non-text content in the composite text; and replace the content to be processed in the initial sample information with corresponding descriptive text to obtain the sample information.

[0010] In some implementations, the plurality of initial sample information includes initial negative sample information, and the sample processing unit further includes a third processing subunit, configured to: perform word segmentation on the initial negative sample information to obtain a word sequence; determine candidate words from the word sequence, wherein the candidate words are unknown words whose frequency of occurrence in the word sequence meets a preset frequency condition; determine candidate non-text content corresponding to the candidate words from the initial negative sample information; and determine target non-text content from the candidate non-text content, wherein the target non-text content is used to determine the content to be processed.

[0011] In some implementations, the initial sample information includes text information, and the second processing subunit is specifically used to: search for preset stop words in the text information of the initial sample information; remove the found preset stop words from the initial sample information to obtain the sample information.

[0012] In some implementations, the plurality of initial sample information includes initial negative sample information, and the plurality of sample information includes negative sample information. The second processing subunit is specifically used to: detect target feature information in the initial negative sample information using preset information features, wherein the target feature information is information corresponding to the preset information features, and the preset information features include one or more of preset keywords and preset character rules; and use the initial negative sample information in which the target information features are detected as the negative sample information.

[0013] In some embodiments, the pruning unit includes a first pruning subunit, a second pruning subunit, a third pruning subunit, and a fourth pruning subunit, comprising: the first pruning subunit for constructing a computational operation graph of the model to be processed; the second pruning subunit for grouping the weight parameters of the model to be processed based on the computational operation graph to obtain at least one weight parameter group; the third pruning subunit for determining the importance of the weight parameter group to the model to be processed; and the fourth pruning subunit for pruning the weight parameter group of the model to be processed according to the importance, to obtain the pruned model.

[0014] In some embodiments, the second adjustment unit is specifically used to: when the model to be processed includes the basic information classification model, obtain multiple target sample information from the multiple sample information according to a preset ratio; and adjust the pruned model using the multiple target sample information to obtain the first information classification model.

[0015] In some embodiments, the third adjustment unit is specifically used to: when the model to be processed includes the pre-trained information processing model, use the basic information classification model as the teacher model in knowledge distillation, and use the pruned model as the student model in knowledge distillation; use the multiple sample information and the teacher model to perform knowledge distillation on the student model to obtain the second information classification model.

[0016] This application also provides an information classification device, including: an information acquisition unit for acquiring information to be processed and a target information classification model, wherein the target information classification model is obtained by any model adjustment method or information classification method provided in this application to obtain a first information classification model or a second information classification model; and a prediction unit for using the target information classification model to classify and predict the information to be processed to obtain the predicted information category corresponding to the target classification task.

[0017] This application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute steps in any of the model adjustment methods or information classification methods provided in this application.

[0018] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the model adjustment methods or information classification methods provided in this application.

[0019] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the model adjustment methods or information classification methods provided in this application.

[0020] This application embodiment obtains multiple sample information of a pre-trained information processing model and a target classification task; adjusts the pre-trained information processing model using the multiple sample information to obtain a basic information classification model corresponding to the target classification task; prunes the model to be processed to obtain a pruned model, wherein the model to be processed includes either the pre-trained information processing model or the basic information classification model; when the model to be processed includes the basic information classification model, adjusts the pruned model using the multiple sample information to obtain a first information classification model, which is used to perform the target classification task; when the model to be processed includes the pre-trained information processing model, adjusts the pruned model using the basic information classification model and the multiple sample information to obtain a second information classification model, which is used to perform the target classification task.

[0021] In this application, a two-stage adjustment and pruning strategy based on the pre-trained information processing model is employed. This involves fine-tuning the pre-trained model once to adapt it to the target task, and reducing unnecessary parameters through pruning, followed by further fine-tuning on the pruned model. This significantly reduces the size and number of parameters of the final model while maintaining its performance, thereby alleviating the model load, optimizing computational efficiency, and significantly improving inference speed. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1a This is a schematic diagram of a scenario illustrating the information classification method provided in an embodiment of this application;

[0024] Figure 1b This is a flowchart illustrating the model adjustment method provided in an embodiment of this application;

[0025] Figure 1c This is a schematic diagram of the sample processing flow provided in the embodiments of this application;

[0026] Figure 1d This is a schematic diagram of composite text provided in the embodiments of this application;

[0027] Figure 1e This is a schematic diagram of the pruning process provided in the embodiments of this application;

[0028] Figure 1f This is a flowchart illustrating another model adjustment method provided in an embodiment of this application;

[0029] Figure 1g This is a flowchart illustrating another model adjustment method provided in an embodiment of this application;

[0030] Figure 1h This is a schematic diagram of the learning rate change during the training process provided in the embodiments of this application;

[0031] Figure 2 This is a flowchart illustrating the information classification method provided in an embodiment of this application;

[0032] Figure 3a This is a flowchart illustrating a model adjustment method provided in another embodiment of this application;

[0033] Figure 3b This is a schematic diagram of the precision-recall curve provided in the embodiments of this application;

[0034] Figure 4 This is a schematic flowchart of a model adjustment method provided in another embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of the model adjustment device provided in the embodiments of this application;

[0036] Figure 6 This is a schematic diagram of the structure of the information classification device provided in the embodiments of this application;

[0037] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] This application provides a model adjustment method, an information classification method, and related equipment.

[0040] Specifically, the model adjustment device or information classification device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0041] In some embodiments, the model adjustment device or information classification device may also be integrated into multiple electronic devices. For example, the model adjustment device or information classification device may be integrated into multiple servers, and the model adjustment method or information classification method of this application may be implemented by multiple servers.

[0042] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0043] For example, refer to Figure 1a Taking the joint execution of the information classification method by a terminal and a server as an example, the information classification system provided in this application includes a terminal and a server, etc.; the terminal and the server are connected through a network, such as through a wired or wireless network, etc., wherein the device can be integrated into the terminal.

[0044] The terminal can be used to: acquire information to be processed, send the information to the server for classification and prediction, and receive the predicted categories returned by the server. The terminal can include mobile phones, vehicle-mounted terminals, aircraft, tablets, laptops, or personal computers (PCs), etc. A client can also be set on the terminal, which can be an application client or a browser client, etc.

[0045] The server can be used to: acquire multiple sample information from a pre-trained information processing model and a target classification task; adjust the pre-trained information processing model using the multiple sample information to obtain a basic information classification model corresponding to the target classification task; prune the model to be processed to obtain a pruned model, where the model to be processed includes either the pre-trained information processing model or the basic information classification model; when the model to be processed includes the basic information classification model, adjust the pruned model using the multiple sample information to obtain a first information classification model, which is used to perform the target classification task; when the model to be processed includes the pre-trained information processing model, adjust the pruned model using the basic information classification model and multiple sample information to obtain a second information classification model, which is used to perform the target classification task.

[0046] The server can also be used to acquire the information to be processed and the target information classification model, which can be either a first information classification model or a second information classification model. The target information classification model is then used to classify and predict the information to be processed, resulting in the predicted information category corresponding to the target classification task. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0048] The following sections provide detailed descriptions. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments. It is understood that in the specific embodiments of this application, user-related data such as sample information, transaction information, and transaction remarks are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0049] In this embodiment, a model adjustment method is provided, such as Figure 1b As shown, the specific process of this model adjustment method can be as follows:

[0050] 110. Obtain multiple sample information for the pre-trained information processing model and the target classification task.

[0051] The pre-trained information processing model refers to a pre-trained information processing model, which is a general pre-trained model for information processing. In the embodiments of this application, the pre-trained information processing model can also be called a base model, that is, a basic pre-trained model. This model is usually pre-trained with a large amount of data to obtain general features and patterns extracted from the data. This model has general prediction capabilities and can then be fine-tuned to be applied to specific tasks or domains.

[0052] The target classification task refers to a classification task that requires fine-tuning (i.e., adjustment) of the pre-trained information processing model to adapt it to the target classification task. The target classification task can be a binary classification task or a multi-class classification task, specifically determined based on the application scenario or labels. The sample information refers to the samples used to fine-tune the pre-trained information processing model to adapt it to the target classification task. In this embodiment, the sample information carries the classification label corresponding to the target classification task, such as a 0 / 1 label. By utilizing the corresponding classification label, the model can learn how to predict the classification based on the input sample information through supervised learning during fine-tuning.

[0053] For example, in this embodiment of the application, the target classification task can be a classification task for a specific scene or a specific domain, and the sample information can be information for the corresponding specific scene or specific domain. The target classification task can be different for different specific scenes or domains. For example, for different domains, depending on the type of input data, the target classification task can be image classification, audio classification, video classification, or text classification. Furthermore, for different scenes, the target classification task can be sentiment classification, transaction classification, media information classification, language classification, and so on.

[0054] In some implementations, the pre-trained information processing model is a pre-trained model with multiple structures, such as a purely discriminative encoder, an encoder-decoder, a purely generative decoder, etc., which can be selected according to actual needs or application scenarios.

[0055] In some implementations, the pre-trained information processing model has classification capabilities, and its output layer is a classification layer (such as a fully connected layer). Alternatively, in some implementations, the pre-trained information processing model does not have classification capabilities; its output can be fed into a classification layer for fine-tuning to enable classification functionality. It should be noted that the specific classification process and principles of the information classification model in this application are related to the model structure. For details, please refer to the classification process of related models; this application does not impose any limitations.

[0056] In some implementations, the sample information is text information, which can be composite text (i.e., text containing both textual and non-textual content) or single text (i.e., text containing only textual content). The pre-trained information processing model is a pre-trained language model, such as an autoregressive language model, a masked modeling language model, a discriminative language model (such as a large language model), etc. Specifically, it can be a Generative Pre-trained Transformer (GTP), a Bidirectional Transformer (BERT) model, a Roberta model (RoBERTa), DeepSeek, etc. This application does not limit the specific language model used by the pre-trained information processing model; it can be selected according to actual needs or application scenarios.

[0057] In some implementations, the target classification task can be a classification task within a transaction scenario, such as a transaction information classification task. This transaction information classification task can also be a specific classification task, such as a risk classification task for transaction types like commodity transactions, service transactions, virtual item transactions, or virtual resource transactions.

[0058] For example, in some implementations, the target classification task is a risk classification task of virtual resource transactions in an instant messaging application. Before obtaining the pre-trained information processing model and multiple sample information of the target classification task, the method further includes: obtaining transaction note information of virtual resource transactions from the instant messaging application, wherein the transaction note information is used to determine the sample information.

[0059] Virtual resources refer to resources that exist in digital form within instant messaging applications. For example, virtual resources may include, but are not limited to, one or more of the following: electronic balances (such as spare change, balance, and other digital funds), virtual items (such as game items, skins, and equipment), points, membership privileges, and virtual gifts.

[0060] Virtual resource transactions refer to the act of transferring virtual resources through instant messaging applications. For example, virtual resource transactions may include, but are not limited to, one or more of the following: money transfers, payments, and sending red envelopes (digital cash gifts).

[0061] In this context, transaction remarks refer to descriptive information added by the transaction initiator in a virtual resource transaction. For example, transaction remarks may include, but are not limited to, transaction objectives, transaction descriptions, transaction purposes, or other custom-defined additional information. It is understood that through transaction remarks, the transaction initiator can convey their intent, purpose, or expectations for the virtual resource transaction, thus reflecting the intent behind the transaction.

[0062] For example, instant messaging applications can support virtual resource transactions such as payments and / or sending red envelopes between two or more users. This allows for the extraction of transaction notes from the application's historical records of these transactions. A single transaction note can serve as / be used to determine a sample, such as an initial sample.

[0063] In some implementations, the sample information may include, in addition to transaction remarks, other relevant transaction information, such as one or more of the following: transaction amount, transaction time, and attribute information of the transaction initiator and the transaction recipient.

[0064] In some implementations, the sample information includes only transaction notes. By adjusting the pre-trained model, the adjusted model can achieve transaction risk classification by only requiring a small amount of simple transaction notes as input. This reduces the amount of data processing and the difficulty of data acquisition in the risk classification scenario, improves classification efficiency, and reduces resource consumption.

[0065] In some implementations, the multiple sample information includes positive sample information and negative sample information. In different classification scenarios, the positive and negative sample information correspond to different specific categories. For example, taking transaction risk classification as an example, the target classification task can be a binary classification task, with the positive and negative sample information representing the high-risk and low-risk categories, respectively.

[0066] In some implementations, negative sample information can be obtained from multiple data sources, including but not limited to general label tables and labeling information from different scenarios. For example, taking the risk classification task of virtual resource transactions in instant messaging applications as an example, negative sample information can come from general label tables, risk control strategy group labeled samples, and risk accounts actually managed by the instant messaging application, etc. This approach can filter out much more negative sample information; statistically, the overall volume of negative sample information is more than 10 times that of the training data of existing models. Thus, in risk control tasks with an imbalanced distribution of positive and negative samples, data diversity is greatly enriched.

[0067] In some implementations, positive sample information can be randomly drawn from a data source such as a general label table. For example, in a risk classification task for virtual resource transactions in an instant messaging application, positive sample information is the information corresponding to white-labeled users randomly drawn from the label table.

[0068] In some implementations, a corresponding number of positive and negative sample information can be selected according to a preset positive-negative sample ratio to obtain multiple sample information for fine-tuning. This preset positive-negative sample ratio can be set according to the application scenario or actual needs. For example, the ratio of negative to positive sample information in the multiple sample information may be 1:5.

[0069] In this embodiment, the raw data of sample information obtained from the data source can be further processed in one or more ways to obtain sample information that is easier to understand and process, thereby improving the fine-tuning effect of the model. For example, in some implementations, before obtaining multiple sample information of the pre-trained information processing model and the target classification task, the following steps are also included:

[0070] Obtain information from multiple initial samples;

[0071] The initial sample information is processed to obtain the sample information.

[0072] The initial sample information is the initial information used to determine the sample information. In this embodiment, the initial sample information can be the original data of the sample (i.e., the original sample information) or information obtained by processing the original sample information. One initial sample information can be used to determine one sample information.

[0073] For example, the information used in a target classification task (such as sample information) may contain various forms of information. For instance, sample information might be a composite of text and other content. This composite information, composed of multiple forms of content, increases the difficulty for the model to process.

[0074] In some implementations, sample processing includes one or more of the following: replacing non-text content with corresponding descriptive text, removing preset stop words, and using preset feature information for negative sample screening.

[0075] It should be noted that the processes in this application embodiment, such as replacing non-text content with corresponding descriptive text, removing preset stop words, and using preset feature information for negative sample screening, can be used individually for sample processing of the initial sample information, or any two or all of them can be combined for sample processing of the initial sample information. When used in combination, the order of execution is not limited and can be adjusted according to actual needs or application scenarios. For example, such as... Figure 1c The sample processing flow shown can first remove preset stop words from the initial sample information, then replace non-text content with corresponding descriptive text, and finally use preset feature information to perform negative sample screening.

[0076] The following describes the implementation methods of the processing in this application, such as replacing non-text content with corresponding descriptive text, removing preset stop words, and using preset feature information for negative sample screening.

[0077] For example, in some implementations, non-textual content in the sample information can be replaced with corresponding text so that the sample information can convey all the information it contains through text, facilitating model understanding and processing, thereby improving the effectiveness of model fine-tuning and enhancing the model's generalization ability. Specifically, the initial sample information includes composite text. Sample processing is performed on the initial sample information to obtain sample information, including:

[0078] From the initial sample information, determine the content to be processed, which is the non-text content in the composite text;

[0079] Replace the content to be processed in the initial sample information with the corresponding descriptive text to obtain the sample information.

[0080] In this embodiment, the initial sample information refers to sample information that has not undergone non-text content replacement. For example, the initial sample information can be the original sample information, or data after further processing of the original sample information (such as removing preset stop words and / or using preset feature information for negative sample screening).

[0081] Composite text refers to text that includes both textual and non-textual content. It is text created by combining multiple forms of expression; it not only contains textual information but also incorporates other forms of expression, such as images, videos, audio, charts, and animations. In some implementations, composite text includes information containing both text and emojis; that is, composite text is text content with emojis embedded within it.

[0082] Non-text content refers to content in composite text that is not in text form, such as images, videos, audio, charts, animations, etc.

[0083] The descriptive text refers to the text used to describe the content to be processed. This descriptive text can convey the information or core information that the content to be processed is intended to express. The descriptive text can be one or more of keywords or descriptive statements, and can be specifically set according to the application scenario or actual needs. In some implementations, the descriptive text may include one or more of content text and / or type text used to describe the content to be processed.

[0084] For example, taking the risk classification task of virtual resource transactions in an instant messaging application as an example, the transaction remarks information (i.e., the initial sample information) of any virtual resource transaction can be as follows: Figure 1dThe composite text shown includes both text and emojis (i.e., non-text content). Emojis can be identified / detected from this composite text, and the corresponding emojis can be replaced with relevant keywords. For example, the emoji of a house in the composite text can be replaced with the keyword "house emoji," and the replaced information can be used as sample information. Alternatively, the replaced information can be further processed by removing preset stop words and / or using preset feature information for negative sample filtering to obtain the corresponding sample information.

[0085] In this embodiment, the house emoji is replaced with the keyword "house emoji" in the form of content + type, instead of the keyword "house" in the form of content alone, so as to tell the model during training that the text belongs to the text corresponding to the emoji.

[0086] In some implementations, non-text content detection tools can be used to match the initial sample information to identify non-text content, which can then be used as the content to be processed. These detection tools may include, but are not limited to, one or more of regular expressions, non-text content libraries, dictionaries, or lists.

[0087] In some implementations, in certain scenarios, the non-text content in the composite text can be preset content. For example, taking the risk classification task of virtual resource transactions in an instant messaging application as an example, the transaction initiator can use emoticons in the transaction remarks information, which are selected from multiple predefined emoticons in the instant messaging application. Therefore, a mapping relationship between each non-text content (such as emoticons) and the corresponding descriptive text can be pre-constructed so that after detecting the content to be processed in the initial sample information, the corresponding descriptive text can be obtained based on the corresponding mapping relationship for replacement, thereby improving the replacement efficiency.

[0088] In some implementations, word segmentation can be used to identify non-textual content in negative sample information. On one hand, word segmentation can accurately identify non-textual content. On the other hand, using negative sample information as the basis for mining the non-textual content to be replaced ensures that the mined non-textual content is strongly correlated with the negative samples. Since negative sample information often contains anomalous behaviors and patterns, these anomalous features can effectively help the model identify abnormal signals that differ from normal situations, thereby enhancing the model's sensitivity to anomalous information. Especially in risk classification tasks, negative samples often better characterize potential risk features; therefore, mining non-textual content based on negative samples can help the model better identify risks.

[0089] Furthermore, since frequently occurring non-textual content is often highly correlated with the features of negative samples, it needs to be given special attention. By using frequently occurring non-textual content in negative samples as the non-textual content to be replaced, the fine-tuned model's ability to identify negative samples can be enhanced, improving the model's accuracy in distinguishing between positive and negative samples, thereby improving the model's classification accuracy.

[0090] Specifically, the initial sample information includes initial negative sample information. Before processing the initial sample information to obtain the final sample information, the following steps are also included:

[0091] The initial negative sample information is segmented into words to obtain a word sequence;

[0092] The candidate words are determined from the word sequence. The candidate words are unknown words whose frequency in the word sequence meets the preset frequency condition.

[0093] Determine the non-text content corresponding to the candidate words from the initial negative sample information;

[0094] From the candidate non-text content, the target non-text content is determined, and the target non-text content is used to determine the content to be processed.

[0095] Among them, the initial negative sample information is the initial information used to determine the negative sample information. One initial negative sample information can be used to determine a sample information.

[0096] Word segmentation refers to the process of splitting input (such as initial sample information) into individual words or phrases to form a word sequence. In this embodiment, word segmentation can be performed using one or more of the following methods: vocabulary-based segmentation, statistical segmentation, and deep learning-based segmentation.

[0097] Unknown words refer to words that cannot be identified during word segmentation. For example, non-text content such as images, videos, audio, charts, and animations are not in text form and therefore cannot be segmented.

[0098] The preset frequency condition refers to the condition used to determine whether the frequency of a word in a word sequence reaches or exceeds a preset minimum value. This preset frequency condition can be expressed as a minimum frequency, a word frequency threshold, or other forms, etc. Optionally, the preset frequency condition includes the occurrence frequency of an unknown word in the word sequence exceeding a first word frequency threshold, and the target non-text content being candidate non-text content whose occurrence frequency exceeds a second word frequency threshold among all candidate non-text content, or candidate non-text content whose occurrence frequency order is higher than a preset order.

[0099] For example, taking the risk classification task of virtual resource transactions in instant messaging applications as an example, through sample analysis and word segmentation experiments on transaction notes, it was found that the tokenizer would identify emoticons such as house emoticons and ghost face emoticons as "UNK," which represents "unknown" (i.e., unknown words). This is because the tokenizer cannot recognize emoticons, which is why existing models perform poorly in recognizing and classifying transaction notes containing a large number of specific emoticons.

[0100] In this embodiment, a word segmenter is used to identify the feature of the emoticon as "UNK" in order to mine the unknown words corresponding to the emoticon, thereby mining high-frequency non-text content. Specifically, for multiple transaction notes (i.e., multiple initial sample information) for this task, the English characters and numbers of the transaction notes that are negative samples can be extracted and input into the word segmenter. After processing by the word segmenter, the token sequence (i.e., word sequence) corresponding to each negative sample is obtained. All token sequences (i.e., candidate word sequences) with the "UNK" token (i.e., unknown word) appearing more than 3 times are selected. Based on the position of "UNK" (candidate word) in these sequences, the characters at the corresponding positions of the original transaction notes before word segmentation (i.e., candidate non-text content, i.e., candidate emoticons) are calculated. These characters are counted by word frequency and sorted from high to low. Candidate emoticons with a higher frequency than the preset order are selected as target characters (i.e., target non-text content, i.e., target emoticons). For example, you can select the top-ranked characters (i.e., emojis) as target characters.

[0101] In some implementations, any non-textual content in the initial sample information can be used as the content to be processed, and directly replaced with the corresponding descriptive text. Specifically, determining the content to be processed from the initial sample information includes:

[0102] Identify the target non-text content from the initial sample information;

[0103] The identified non-text content is treated as the content to be processed.

[0104] For example, for each initial positive sample and / or initial negative sample, the initial sample information can be matched with pre-mined target emojis to identify content identical to the target emojis, i.e., the corresponding emojis. These identified emojis are then used as content to be processed. Taking the risk classification task of virtual resource transactions in instant messaging applications as an example, after replacing non-textual content in the transaction notes, the adjusted model significantly improves the classification and recognition performance of transaction notes for high-risk transactions with a large amount of non-textual content.

[0105] Exemplarily, in some implementations, preset stop words in the sample information may be removed to filter out words irrelevant to the target classification task in the sample information, so that the training convergence speed is faster. Specifically, the initial sample information includes text information, and performing sample processing on the initial sample information to obtain sample information includes:

[0106] Search for preset stop words in the text information of the initial sample information;

[0107] Remove the found preset stop words from the initial sample information to obtain the sample information.

[0108] The preset stop words refer to words irrelevant to the target classification task. In practical applications, the preset stop words may be set according to actual requirements or application scenarios.

[0109] In the embodiments of the present application, the initial sample information at this time refers to sample information from which the preset stop words have not been removed. For example, the initial sample information may be original sample information, or data obtained by further processing the original sample information (such as replacing non-text content with the corresponding descriptive text and / or performing negative sample screening processing using preset feature information).

[0110] For example, taking the risk classification task of virtual resource transactions in an instant messaging application as an example, stop words irrelevant to risk classification such as "de", "happy", "then", "vacation", and "Dragon Boat Festival" may be preset. These preset stop words may be used to perform stop word identification (search) and removal on the transaction note information (i.e., initial sample information) of the task, so as to use the information after removal as sample information, or further perform processing of replacing non-text content with the corresponding descriptive text and / or negative sample screening processing using preset feature information on the information after removal, to obtain corresponding sample information. In this way, by removing information useless for the transaction information risk classification objective from the sample information, the model pays more attention to risk words, thereby making the training convergence speed faster.

[0111] Exemplarily, in some implementations, preset information features may be used to screen negative sample information from initial negative sample information, so as to improve the quality of negative samples and enhance the generalization ability and classification accuracy of the model. Specifically, the plurality of initial sample information includes initial negative sample information, the plurality of sample information includes negative sample information, and performing sample processing on the initial sample information to obtain sample information includes:

[0112] Using the preset information features to detect target feature information in the initial negative sample information, wherein the target feature information is information corresponding to the preset information features, and the preset information features include one or more of preset keywords and preset character rules;

[0113] The initial negative sample information in which the target information feature is detected is taken as negative sample information.

[0114] Among them, the preset information features refer to the information features used to identify target information features in sample information. The preset information features are related to the information of the negative category (i.e., the category corresponding to the negative sample). For example, the preset information features may include one or more of the preset keywords, preset character rules, etc., which are related to the negative category. The preset character rules refer to the rules of character patterns, and the preset character rules can be regular expression rules.

[0115] Among them, target feature information refers to information that conforms to preset information features.

[0116] In this embodiment, the preset information features can be set according to the application scenario or actual needs. For example, in some implementations, the preset information features can be associated with the target classification task so that the sample information is more relevant to the target classification task, thereby improving the classification effect and accuracy.

[0117] In this embodiment of the application, the initial negative sample information refers to the information of negative samples that have not been screened using preset feature information. For example, the initial negative sample information can be the original negative sample information, or the data after further processing of the original negative sample information (such as replacing non-text content with corresponding descriptive text and / or removing preset stop words).

[0118] For example, taking the risk classification task of virtual resource transactions in instant messaging applications as an example, high-frequency risk keywords can be used as a candidate word list (preset keyword vocabulary), and regular expression rules (i.e., preset character rules) can be specified in advance. For instance, some high-risk information (i.e., initial negative sample information) will exhibit a long string of numbers interspersed with percentage signs %. Therefore, if the initial negative sample information contains keywords from these candidate word lists or conforms to the regular expression rules, it is selected as sample information. Alternatively, the selected information can be further processed by replacing non-text content with corresponding descriptive text and / or removing preset stop words to obtain the corresponding sample information. Otherwise, the initial negative sample information is discarded.

[0119] The quantity, quality, and diversity of negative sample information are crucial to a model's ability to handle target classification tasks. In some implementations, text preprocessing and feature extraction methods (such as using jieba segmentation and TF-IDF tools) can be employed to process and extract features from the initial negative sample information to identify predefined keywords. For instance, tools like jieba segmentation and TF-IDF can be used to preprocess and extract features from the initial negative sample information to explore new sensitive word discovery techniques, identify more specific high-frequency risk words, and improve the quantity, quality, and diversity of the screened negative sample information.

[0120] 120. Adjust the pre-trained information processing model using information from multiple samples to obtain the basic information classification model corresponding to the target classification task.

[0121] The basic information classification model is an information classification model obtained by fine-tuning the pre-trained information processing model. In this embodiment, the basic information classification model has all the parameters (i.e., all parameters) of the pre-trained information processing model.

[0122] For example, a pre-trained information processing model can be trained using information from multiple samples in a target classification task. This pre-trained model can then be fine-tuned to improve the base information classification model's ability to perform the target classification task effectively. It is understood that the fine-tuned base information classification model possesses the capability for that target classification task.

[0123] For example, a pre-trained information processing model (such as the BERT model) can be selected as the base model, and a fully connected layer can be added after the output of this model to obtain the information processing model to be adjusted. Multiple sample information can be used as input for this fine-tuning stage. The model to be adjusted is iteratively updated through a training process that includes classification prediction, loss calculation based on prediction results, and backpropagation to update model parameters, until a preset termination condition is met, resulting in the fine-tuned model, i.e., the basic information classification model. The preset termination condition may include, but is not limited to, reaching a preset maximum number of training epochs, loss function convergence, and overfitting.

[0124] In some implementations, the weight parameters of a predetermined number of layers (e.g., the first 12 layers) of the pre-trained information processing model can be frozen to freeze the low-level feature extraction capability of the pre-trained information processing model, thereby accelerating training, reducing computational overhead, and avoiding overfitting.

[0125] 130. Prune the model to be processed to obtain the pruned model. The model to be processed includes either a pre-trained information processing model or a basic information classification model.

[0126] Pruning refers to the process of reducing model complexity by removing unimportant parameters or network structures. Pruning can improve computational efficiency, reduce memory usage, and potentially improve generalization ability. The pruned model refers to the model obtained after pruning.

[0127] For example, a pruned model (hereinafter referred to as the first pruned model) can be obtained by pruning a basic information classification model. Alternatively, a pruned model (hereinafter referred to as the second pruned model) can be obtained by pruning a pre-trained information processing model. It should be noted that during the pruning process, redundant structures in the model are removed. Therefore, compared with the original model, the pruned model has lower complexity and scale, but still retains the capabilities of the original model. For example, the pruned model obtained by pruning the pre-trained information processing model has the general prediction capability of the pre-trained information processing model, and the pruned model obtained by pruning the basic information classification model has the target classification task capability of the pre-basic information classification model.

[0128] In some implementations, model quantization techniques can be used to compress the pruned model, using the compressed model as the pruned model. This further reduces the model size using model quantization. For example, by converting the float32 format parameters in the original pruned model to float16, the size can be halved. Alternatively, int8 quantization techniques such as post-training dynamic quantization and post-training static quantization can be introduced to reduce the size of the pruned model.

[0129] In some implementations, by constructing weight parameter groups for computation graph pairs and then comparing the importance of each group to prune and remove unimportant weight parameter groups, and by analyzing the dependencies between weights based on the computation graph, it is possible to more accurately determine which weight groups are redundant or unimportant, rather than blindly pruning individual weights. This improves the efficiency of the pruning process and avoids unnecessary impacts on network structure and performance. Specifically, the model to be processed is pruned to obtain the pruned model, including:

[0130] Construct a computational operation graph for the model to be processed;

[0131] Based on the computational operation graph, the weight parameters of the model to be processed are grouped to obtain at least one weight parameter group.

[0132] Determine the importance of the weight parameter set to the model to be processed;

[0133] Based on their importance, the weight parameter groups of the model to be processed are pruned to obtain the pruned model.

[0134] The computational operation graph, also known as a computational graph, is a graph structure representing the relationships between computational operations (such as matrix multiplication and activation functions) of the model being processed. Each node in the computational graph represents an operation in the model, and each edge represents data flow or dependency. The model's weight parameters can be the inputs to the corresponding operations (i.e., nodes).

[0135] For example, such as Figure 1e The pruning process shown transforms the computation of the original model into a computational operation graph. This graph structure provides a clear understanding of the relationships between the various parts of the model. Within the computational graph, nodes can be grouped based on their dependencies. For example, weight parameters can be grouped according to the inputs, outputs, and relationships between operations in the computational operation graph, resulting in at least one weight parameter group. For instance, the model might include Layer 1 (input A, output B) and Layer 2 (input B, output C). In Layer 1, weights influence the operation's outcome; B is directly generated from the weights of Layer 1. Layer 2's input is B, and its weights are closely related to the output of Layer 1. Therefore, the weights of Layer 1 and Layer 2 can be grouped together because they directly depend on the output of the previous layer.

[0136] Next, the importance (i.e., degree of importance) of each weight parameter group to the model is calculated. If one or more weight parameter groups have a relatively small impact on the model, then one or more weight parameter groups can be pruned while keeping the other weight parameter groups unchanged to reduce the complexity of the model.

[0137] In some implementations, for each set of weight parameters, its importance to the model can be calculated based on a preset index function. For example, the preset index function may include, but is not limited to, one or more of the following functions: L2 norm (or weight norm), which calculates the L2 norm of the weight matrix; a smaller norm may indicate that the set has less impact on the model output; gradient information, which calculates the magnitude of the gradient through backpropagation; parameters with smaller gradients may change less during training, indicating that they have less impact on the model; and changes in the loss function, which compares the changes in the loss function after removing a certain set of parameters to determine the impact of that set on model performance.

[0138] In some implementations, the weight parameter groups to be pruned can be determined based on their importance and a preset pruning ratio, and then the pruning weight parameter groups can be pruned to obtain the pruned model. For example, the weight parameter groups can be sorted according to their importance, and according to the preset pruning ratio, the weight parameter groups with low importance can be gradually removed starting from the weight parameter group with the lowest importance in the sorting, until the required pruning ratio is reached.

[0139] 140. When the model to be processed includes a basic information classification model, the pruned model is adjusted using multiple sample information to obtain the first information classification model.

[0140] Among them, the first information classification model is a model obtained by pruning and adjusting the basic information classification model.

[0141] For example, such as Figure 1fThe model adjustment method shown in this application embodiment uses a pre-trained information processing model as a base model. First, the base model is adjusted once to obtain a basic information classification model with target classification task capability. Then, the basic information classification model is pruned to obtain a first pruned model. Finally, the first pruned model is adjusted a second time to obtain a first information classification model with target classification task capability and simplified structure.

[0142] Therefore, this embodiment of the application performs a fine-tuning of the pre-trained model to adapt it to the classification target task. Then, it prunes the model to reduce unnecessary parameters after the first adjustment, and performs fine-tuning again on the pruned model. This significantly reduces the size and number of parameters of the final model while ensuring model performance, thereby reducing the model load, optimizing the model's computational efficiency, and significantly improving the model's inference speed.

[0143] In some implementations, since the basic information classification model already possesses the capability for the target classification task, its pruned model also possesses a certain capability for the target classification task. Therefore, during the secondary model adjustment, a smaller number of samples can be used for training to save training time and resources. Specifically, when the model to be processed includes the basic information classification model, the pruned model is adjusted using multiple sample information to obtain the first information classification model, including:

[0144] When the model to be processed includes a basic information classification model, multiple target sample information is obtained from multiple sample information according to a preset ratio.

[0145] The pruned model is adjusted using information from multiple target samples to obtain the first information classification model.

[0146] The preset ratio can be set according to the application scenario or actual needs. The target sample information refers to the sample information used to adjust the pruned model. This target sample information is selected from multiple sample information sets according to the preset ratio.

[0147] For example, 20% or 40% of the sample information used in a single adjustment can be selected as the target sample information. This pruned model can then be adjusted a second time to obtain a first information classification model with the ability to perform the target classification task and a simplified structure.

[0148] For example, in this secondary adjustment, multiple sample information can be used as input. The model is iteratively updated through a process involving classification prediction using the first pruned model, loss calculation based on the prediction results and the corresponding classification labels for the target classification task, and backpropagation based on the loss to update model parameters, until a preset termination condition is met, resulting in the fine-tuned model, i.e., the first information classification model. The preset termination condition may include, but is not limited to, reaching a preset maximum number of training epochs, loss function convergence, and overfitting. The loss can be calculated using a loss function, for example, a combination of one or more of the following: mean squared error loss function, cross-entropy loss function, and perceptual loss function.

[0149] In some implementations, the multiple target sample information includes positive sample information and negative sample information. Optionally, the ratio of negative sample information to positive sample information in the multiple target sample information and the multiple sample information is the same, such as 1:5, so as to ensure that the sample categories are consistent in the two adjustments and improve the stability of the model.

[0150] 150. When the model to be processed includes a pre-trained information processing model, the pruned model is adjusted using the basic information classification model and multiple sample information to obtain the second information classification model.

[0151] The second information classification model is a model obtained by adjusting the pre-trained information processing model after pruning.

[0152] For example, such as Figure 1g The model adjustment method shown in this application embodiment uses a pre-trained information processing model as a base model. First, the base model is adjusted once to obtain a basic information classification model with target classification task capability. Then, the base model is pruned to obtain a second pruned model with simplified structure. Finally, the basic information classification model is used to adjust the second pruned model to obtain a second information classification model with target classification task capability and simplified structure.

[0153] Therefore, this embodiment of the application performs a fine-tuning on the pre-trained model to adapt it to the classification target task, reduces unnecessary parameters in the pre-trained model through pruning, and then uses the fine-tuned model to perform further fine-tuning on the pruned model. This significantly reduces the size and number of parameters of the final model while maintaining model performance, thereby reducing model load, optimizing computational efficiency, and significantly improving model inference speed.

[0154] In some implementations, knowledge distillation training can be introduced to enable the second-pruned model to learn the knowledge of the basic information classification model. This helps the second-pruned model (a pruned model with reduced parameters) to learn how to classify better by mimicking the behavior of the basic information classification model and acquiring the learned effective features or knowledge, thereby enhancing its ability to perform specific target classification tasks.

[0155] After introducing knowledge distillation training, another issue arises: how to generate the student model. The traditional approach is to directly use lightweight models with randomly initialized parameters, such as Transformer or TextCNN, as the student model. However, this approach has drawbacks: the student model does not inherit the weights and deep architecture of the teacher model; it simply mechanically learns the teacher's labels. Often, these weights and architecture contain a large amount of pre-trained knowledge, resulting in poor learning outcomes. Another method is to simply remove some layers from the teacher model, such as using only the first six layers as the student model. This approach disrupts the deep structure of the teacher model, and removing subsequent layers may also result in the loss of the teacher model's high-level feature abstraction capabilities, often leading to unsatisfactory results.

[0156] To address this issue, this application proposes a simple and efficient student model generation scheme based on pruning. First, the student model (the model after the second pruning) is directly generated by pruning the pre-trained model with all parameters (i.e., the pre-trained information processing model). This retains the high-importance pre-trained weight parameters of the teacher model and maintains its deep structure. Next, the pre-trained language model with all parameters is fine-tuned, and teacher labels are generated on the training set using the fine-tuned basic information classification model. Finally, the student model is trained using a combination of teacher labels and original labels (i.e., the classification labels corresponding to the target classification task carried by the sample information). Experimental results show that, due to the addition of teacher label knowledge, the distilled student model achieves 96% of the recall rate at 99.9% precision.

[0157] For example, when the model to be processed includes a pre-trained information processing model, the pruned model is adjusted using the basic information classification model and information from multiple samples to obtain a second information classification model, including:

[0158] When the model to be processed includes a pre-trained information processing model, the basic information classification model is used as the teacher model in knowledge distillation, and the pruned model is used as the student model in knowledge distillation.

[0159] By using multiple sample information and the teacher model, knowledge distillation is performed on the student model to obtain the second information classification model.

[0160] Knowledge distillation is a machine learning technique primarily used to transfer knowledge from a complex, computationally intensive "large model" (i.e., the teacher model) to a smaller, more computationally efficient model (i.e., the student model). This method typically improves the performance of the smaller model by training it to mimic the behavior of the larger model.

[0161] In knowledge distillation, the teacher model and the student model are two types of models. The teacher model is typically a larger, more complex model trained on a large amount of data. Its purpose is usually to provide an accurate reference standard or to guide the training of the student model. The student model is typically a smaller, simpler model. It is usually trained through guidance from the teacher model or by knowledge distillation, thereby learning the knowledge and predictive capabilities of the teacher model.

[0162] For example, a pre-trained language model with all parameters (i.e., a pre-trained information processing model) can be fine-tuned to obtain a teacher model with all parameters (i.e., a basic information classification model). The teacher model t is then used to generate teacher labels (also called soft labels) P on the training set (i.e., multiple sample information). P is a probability distribution representing the teacher model's predicted probability distribution for sample classification. The teacher model is then used to perform knowledge distillation on the student model (i.e., the second-pruned model) on the training set to train a lightweight student model (i.e., a second information classification model).

[0163] For example, in this knowledge distillation, multiple sample information can be used as input. The student model performs classification prediction, calculates loss based on the prediction results, teacher labels, and original labels (also called hard labels, i.e., the classification labels corresponding to the target classification task carried by the sample information), and iteratively updates the student model through backpropagation based on the loss. This iterative training process continues until a preset termination condition is met, resulting in a fine-tuned model, i.e., the second information classification model. The preset termination condition may include, but is not limited to, reaching a preset maximum number of training epochs, loss function convergence, and overfitting. The loss can be calculated using a loss function, such as a combination of one or more of the mean squared error loss function, cross-entropy loss function, and perceptual loss function.

[0164] Specifically, in this knowledge distillation, the student model learns the original label on the one hand and calculates the loss based on the original label based on the difference between the class distribution predicted by the student model and the original label; on the other hand, it fits the teacher label P and calculates the loss based on the teacher label based on the difference between the class distribution predicted by the student model and the teacher label.

[0165] Cross-entropy loss or other loss functions can be used to measure the difference between the class distribution predicted by the student model and the true labels, and to calculate the loss based on the original labels. Furthermore, assuming the predicted probability distribution of the student model f is Q, the parameters are θ, and the input sample information is x, then the learning of the teacher labels (i.e., the loss based on the teacher labels) can be mathematically expressed as the loss function θ that minimizes the KL divergence between the teacher label P and the predicted probability distribution Q of the student model. * :

[0166] θ * =argmin θ KL_div(P||Q(θ)),

[0167] in,

[0168] Where KL_div(P||Q) represents the KL divergence, Q(i; θ) represents the probability distribution predicted by the student model f. f(x,i; θ) represents the logit probability that the student model f predicts that the sample information x is of class i when the input data is x. P(i) represents the probability distribution of the teacher label P. t(x,i) represents the logit probability that the teacher model t predicts that the sample information x is of class i.

[0169] In some implementations, during model tuning, including adjusting the pre-trained information processing model and / or the pruned model, a strategy combining learning rate warmup and cosine decay can be used to iteratively update model parameters to make the learning process smoother and the training process more stable. This strategy involves starting the learning rate from a small value (usually 0) and gradually increasing it linearly until a set maximum learning rate is reached. After reaching the maximum learning rate, the learning rate gradually decays according to a cosine function during subsequent training. For example, ... Figure 1h The diagram illustrates the learning rate changes during the training process. Assuming a learning rate lr = 0.0001 and 1000 training iterations, as shown by the solid lines in the diagram, the learning rate will slowly increase linearly from 0 to its maximum value lr in the first 10% of iterations, i.e., the first 100 iterations, and then slowly decay back to 0 with a cosine function over half a cycle.

[0170] To ensure the accuracy of the first and / or second information classification models in classifying the target task, commonly used metrics such as AUC score, F1 score, or average precision score may not be used as evaluation metrics. In some implementations, the evaluation metric for the first and / or second information classification models can be recall at 99.9% precision. A higher recall at this precision indicates better predictive performance for the target classification task. For example, under this evaluation metric, the recall of the existing model (pre-trained information processing model) is 14%, while the recall of the information classification model after fine-tuning and pruning in this embodiment is 48%. Obviously, the embodiment of this application has a better classification ability for the target task.

[0171] The model adjustment scheme provided in this application can be applied to various model adjustment scenarios. For example, taking transaction information classification as an example, multiple sample information of a pre-trained information processing model and the target classification task are obtained; the pre-trained information processing model is adjusted using the multiple sample information to obtain the basic information classification model corresponding to the target classification task; the model to be processed is pruned to obtain a pruned model, where the model to be processed includes either the pre-trained information processing model or the basic information classification model; when the model to be processed includes the basic information classification model, the pruned model is adjusted using the multiple sample information to obtain a first information classification model, which is used to perform the target classification task; when the model to be processed includes the pre-trained information processing model, the pruned model is adjusted using the basic information classification model and the multiple sample information to obtain a second information classification model, which is used to perform the target classification task.

[0172] As can be seen from the above, the embodiments of this application can utilize a two-stage adjustment and pruning strategy based on the pre-trained information processing model. This allows the model to adapt to the target task through a single fine-tuning of the pre-trained model. Furthermore, unnecessary parameters are reduced through pruning, and further fine-tuning is performed on the pruned model. In this way, while maintaining model performance, the size and number of parameters of the final model are significantly reduced, thereby alleviating the model load, optimizing the model's computational efficiency, and significantly improving the model's inference speed.

[0173] In this embodiment, an information classification method is provided, such as... Figure 2 As shown, the specific process of this information classification method can be as follows:

[0174] 210. Obtain a classification model for the information to be processed and the target information.

[0175] The information to be processed refers to the information processed by the target information classification model.

[0176] For example, in this embodiment, the target information classification model is obtained by any of the model training methods provided in this embodiment, resulting in either a first information classification model or a second information classification model. The information to be processed may come from user input and / or data from the system or other external sources.

[0177] In some implementations, the information to be processed is text information, which can be composite text (i.e., text containing both text content and non-text content) or single text (i.e., text containing only text content). In some implementations, composite text is information containing both text and emojis, that is, composite text is text content with emojis embedded within it.

[0178] In some implementations, when the target classification task is a risk classification task for virtual resource transactions in an instant messaging application, the information to be processed can be transaction remarks information of the virtual resource transactions. For example, transaction remarks information of virtual resource transactions can be obtained from the instant messaging application. For instance, an instant messaging application may support virtual resource transactions such as payments and / or sending red envelopes between two or more users, so the transaction remarks information of the virtual resource transactions such as payments and / or sending red envelopes that occur in the instant messaging application can be obtained as the information to be processed.

[0179] In some implementations, the information to be processed may include, in addition to transaction remarks, other relevant transaction information, such as one or more of the following: transaction amount, transaction time, transaction initiator and transaction recipient attribute information.

[0180] In some implementations, the information to be processed includes only transaction notes, so that the target information classification model only needs to obtain a small amount of simple transaction notes as input to achieve transaction risk classification, thereby reducing the amount of data processing and the difficulty of data acquisition in this risk classification scenario, improving classification efficiency and reducing resource consumption.

[0181] 220. Use the target information classification model to classify and predict the information to be processed, and obtain the predicted information category corresponding to the target classification task.

[0182] Here, the predicted information category refers to the category predicted by the target information classification model based on the information to be processed. The predicted information category can be different for different target classification tasks.

[0183] For example, during the inference phase, a target information classification model can be used to classify and predict the information to be processed. The information classification method of this application embodiment can be applied to various information classification scenarios. For example, taking a classification task in a transaction scenario, when a transaction occurs, the transaction system can obtain the transaction information (such as transaction remarks) and call the target information classification model in the transaction system to process the transaction information to obtain the predicted information category of the transaction information, such as high risk or low risk, so as to manage and control the risk of high-risk transactions according to the predicted information category of the transaction information.

[0184] It should be noted that, in the embodiments of this application, the specific classification process and principle of the target information classification model are related to the model structure. For details, please refer to the classification process of the relevant model. This embodiment of the application does not impose any restrictions.

[0185] In some implementations, when the information to be processed is composite text, the non-text content in the information to be processed can be replaced with corresponding descriptive text to obtain the replaced information. The replaced information can then be classified and predicted using a target information classification model to obtain the predicted information category corresponding to the target classification task. Specifically, classifying and predicting the information to be processed using a target information classification model to obtain the predicted information category corresponding to the target classification task includes: identifying target non-text content from the information to be processed; using the identified target non-text content as the content to be processed; replacing the content to be processed in the information to be processed with corresponding descriptive text to obtain the replaced information; and using a target information classification model to classify and predict the replaced information to obtain the predicted information category corresponding to the target classification task.

[0186] For example, taking the risk classification task of virtual resource transactions in instant messaging applications as an example, the transaction remarks information (i.e., the initial sample information) of any virtual resource transaction can be composite text, which includes text and emoticons (i.e., non-text content). Emoticons can be identified / detected from this composite text, and the corresponding emoticons can be replaced with corresponding keywords. For example, the emoticon of a house in the composite text can be replaced with the corresponding keyword "house emoticon", and the replaced information can be used as input to the target information classification model.

[0187] In some implementations, non-text content detection tools can be used to match the information to be processed, identifying non-text content as the content to be processed. These detection tools may include, but are not limited to, one or more of regular expressions, non-text content libraries, dictionaries, or lists.

[0188] The information classification scheme provided in this application can be applied to various information classification scenarios. For example, taking transaction information classification as an example, a classification model for the information to be processed and the target information is obtained; the target information classification model is used to classify and predict the information to be processed, thereby obtaining the predicted information category corresponding to the target classification task.

[0189] As can be seen from the above, the embodiments of this application can utilize a two-stage adjustment and pruning strategy based on the pre-trained information processing model. This allows the model to adapt to the target task through a single fine-tuning of the pre-trained model. Furthermore, unnecessary parameters are reduced through pruning, and further fine-tuning is performed on the pruned model. In this way, while maintaining model performance, the size and number of parameters of the final model are significantly reduced, thereby alleviating the model load, optimizing the model's computational efficiency, and significantly improving the model's inference speed.

[0190] The method described in the above embodiments will be further described in detail below.

[0191] In this embodiment, the method of this application embodiment will be described in detail using transaction information classification as an example.

[0192] like Figure 3a As shown, the specific process of a model adjustment method is as follows:

[0193] 310. Obtain multiple sample information for the pre-trained information processing model and the target classification task.

[0194] For example, taking the classification of transaction information for virtual resource transactions in instant messaging applications as an example, a pre-trained language model (such as Roberta) can be used as the base model, i.e., a pre-trained information processing model. Transaction notes (i.e., initial sample information) from multiple historical virtual resource transactions are then obtained from the instant messaging application's database. In existing instant messaging applications, transaction notes may be composite text containing both text and emoticons.

[0195] Analysis of the model's inference results revealed that existing transaction information classification models perform poorly in recognizing high-risk transaction information containing a large number of emojis (i.e., emoticons). The model scores low for high-risk transaction information with many emojis, and a low score implies that the model considers the transaction information risk-free. This is clearly unreasonable. Improving the model's recognition performance for this type of information containing a large number of specific emojis is a key aspect of enhancing the capabilities of transaction risk information classification models.

[0196] In this embodiment, the acquired transaction remarks are processed to obtain multiple sample information. For example, preset stop words are removed from the transaction remarks, emojis are replaced with corresponding descriptive text, and finally, negative sample filtering is performed using preset feature information to obtain multiple sample information.

[0197] Therefore, this application's embodiments reconstruct sample information from a wider range of sources by combining multiple sample processing procedures. Simultaneously, by combining business experience and data analysis, several types of emojis that frequently appear in high-risk transaction information were identified, and data cleaning and replacement were performed on transaction information containing these high-frequency emojis. Furthermore, more model training techniques were employed, significantly improving the model's business performance, especially its recognition accuracy for information containing specific high-frequency emojis.

[0198] 320. Adjust the pre-trained information processing model using information from multiple samples to obtain the basic information classification model corresponding to the target classification task.

[0199] For example, during the fine-tuning phase of training the full-parameter model, the pre-trained information processing model can be trained using the information from multiple processed samples to adjust (fine-tune) the pre-trained information processing model, so that the obtained basic information classification model can better perform the target classification task.

[0200] For example, a pre-trained information processing model (such as the BERT model) can be selected as the base model, and a fully connected layer can be added after the output of this model to obtain the information processing model to be adjusted. Multiple sample information can be used as input for this fine-tuning stage. The model to be adjusted is iteratively updated through a training process that includes classification prediction, loss calculation based on prediction results, and backpropagation to update model parameters, until a preset termination condition is met, resulting in the fine-tuned model, i.e., the basic information classification model. This basic information classification model has all the parameters of the base model (i.e., all parameters) and is capable of classifying transaction information.

[0201] 330. Prune the basic information classification model to obtain the pruned model.

[0202] For example, a pruned model (i.e., the first pruned model) can be obtained by pruning a basic information classification model. For instance, DepGraph can be used as a pruning tool to prune the basic information classification model. The main idea of ​​DepGraph is: in the neural network architecture, a computation graph is constructed to group all weight parameters belonging to the same pruning strategy into a group. Then, some indicator functions (such as the norm of the parameters within the group) are used to determine the importance of these groups of parameters. Based on a set pruning ratio, weight parameter groups with low importance are removed.

[0203] Therefore, in this embodiment, the base model is first fine-tuned on the full training data, and then DepGraph is used to prune the fine-tuned model. After multiple experimental comparisons, 75% of the linear mapping layer parameters and half of the attention head modules in the multi-head attention mechanism were finally pruned.

[0204] 340. Adjust the pruned model using information from multiple samples to obtain the first information classification model, which is used to perform the target classification task.

[0205] For example, in this embodiment, a pre-trained information processing model is used as the base model. First, the base model is adjusted once to obtain a basic information classification model capable of target classification tasks. Then, the basic information classification model is pruned to obtain a first pruned model. Finally, the first pruned model is adjusted a second time to obtain a first information classification model with target classification task capabilities and a simplified structure. This first information classification model has the capability to classify transaction information, and its structure is simpler than that of the base model.

[0206] For example, when the model to be processed includes a basic information classification model, adjusting the pruned model using multiple sample information to obtain a first information classification model includes: when the model to be processed includes a basic information classification model, obtaining multiple target sample information from multiple sample information according to a preset ratio; adjusting the pruned model using multiple target sample information to obtain a first information classification model.

[0207] For example, 20% or 40% of the sample information used in a single adjustment can be selected as the target sample information. This pruned model can then be adjusted a second time to obtain a first information classification model with the ability to perform the target classification task and a simplified structure.

[0208] In this embodiment, the first information classification model can be used to classify transaction information for virtual resource transactions in instant messaging applications. For example, when a virtual resource transaction occurs in an instant messaging application, the transaction remarks information can be obtained. If the transaction remarks information is composite text including emoticons, the emoticons in the transaction remarks information can be replaced with the corresponding descriptive text to obtain the replaced information. The first information classification model is then called in the transaction system to process the replaced information and obtain the predicted information category of the transaction information, such as high risk or low risk, so as to manage and control the risk of high-risk transactions based on the predicted information category of the transaction information.

[0209] like Figure 3bThe precision-recall curve diagram shown in the figure illustrates that, while the precision-recall curve represented by the solid line in the figure shows good performance at low to medium recall rates when only the first pruned model is used for transaction information classification, the precision in the high recall region still lags significantly behind the original model (the recall rate is only 40% of the original model at 95% precision), making it unsuitable for direct application to online transaction information risk classification and identification. Therefore, this application proposes a second fine-tuning of the first-stage pruned model. Experiments show that after the second fine-tuning on different proportions of the training set, the performance of the pruned model is significantly restored. Finally, as shown by the precision-recall curve represented by the dashed line in the figure, the second-stage pruned model (i.e., the first information classification model) achieves a recall rate of 80% to 85% of the fully parameter-tuned model (i.e., the basic information classification model) at a precision of 99.9%. Therefore, the first information classification model not only has a simplified structure but also possesses better capabilities for transaction information classification tasks.

[0210] As can be seen from the above, the embodiments of this application can efficiently compress the model size and significantly improve the inference speed through fine-tuning model pruning and secondary fine-tuning, while maintaining the model's business performance close to that of the original model with hundreds of millions of parameters. The first information classification model obtained by the embodiments of this application has better target classification task capabilities and a wider range of applications compared to the second information classification model.

[0211] The method described in the above embodiments will be further described in detail below.

[0212] In this embodiment, the method of this application embodiment will be described in detail using transaction information classification as an example.

[0213] like Figure 4 As shown, the specific process of a model adjustment method is as follows:

[0214] 410. Obtain multiple sample information for the pre-trained information processing model and the target classification task.

[0215] For example, taking the classification of transaction information for virtual resource transactions in instant messaging applications as an example, a pre-trained language model (such as Roberta) can be used as the base model, i.e., a pre-trained information processing model. Transaction notes (i.e., initial sample information) from multiple historical virtual resource transactions are then obtained from the instant messaging application's database. In existing instant messaging applications, transaction notes may be composite text containing both text and emoticons.

[0216] Multiple transaction notes can be processed to obtain multiple sample information. For example, preset stop words can be removed from the transaction notes, emojis can be replaced with corresponding descriptive text, and finally, negative sample filtering can be performed using preset feature information to obtain multiple sample information.

[0217] 420. Adjust the pre-trained information processing model using information from multiple samples to obtain the basic information classification model corresponding to the target classification task.

[0218] For example, during the fine-tuning phase of training the full-parameter model, the pre-trained information processing model can be trained using the information from multiple processed samples to adjust (fine-tune) the pre-trained information processing model, so that the obtained basic information classification model can better perform the target classification task.

[0219] For example, a pre-trained information processing model (such as the BERT model) can be selected as the base model, and a fully connected layer can be added after the output of this model to obtain the information processing model to be adjusted. Multiple sample information can be used as input for this fine-tuning stage. The model to be adjusted is iteratively updated through a training process that includes classification prediction, loss calculation based on prediction results, and backpropagation to update model parameters, until a preset termination condition is met, resulting in the fine-tuned model, i.e., the basic information classification model. This basic information classification model has all the parameters of the base model (i.e., all parameters) and is capable of classifying transaction information.

[0220] 430. Prune the pre-trained information processing model to obtain the pruned model.

[0221] For example, a pruned model (i.e., a second pruned model) can be obtained by pruning a pre-trained information processing model. For instance, DepGraph can be used as a pruning tool to prune the pre-trained information processing model.

[0222] 440. When the model to be processed includes a pre-trained information processing model, the pruned model is adjusted using the basic information classification model and multiple sample information to obtain a second information classification model, which is used to perform the target classification task.

[0223] For example, in this embodiment, a pre-trained information processing model is used as a base model. First, the base model is adjusted to obtain a basic information classification model capable of target classification tasks. Then, the base model is pruned to obtain a structurally simplified second pruned model. Finally, the basic information classification model is used to adjust the second pruned model to obtain a second information classification model with target classification task capabilities and a simplified structure. This second information classification model has the capability for transaction information classification tasks, and its structure is simpler than that of the base model.

[0224] For example, when the model to be processed includes a pre-trained information processing model, the pruned model is adjusted using the basic information classification model and multiple sample information to obtain a second information classification model, including: when the model to be processed includes a pre-trained information processing model, the basic information classification model is used as the teacher model in knowledge distillation, and the pruned model is used as the student model in knowledge distillation; knowledge distillation is performed on the student model using multiple sample information and the teacher model to obtain the second information classification model.

[0225] For example, a pre-trained language model with all parameters (i.e., a pre-trained information processing model) can be fine-tuned to obtain a teacher model with all parameters (i.e., a basic information classification model). The teacher model t is then used to generate teacher labels (also called soft labels) P on the training set (i.e., multiple sample information). P is a probability distribution representing the teacher model's predicted probability distribution for sample classification. The teacher model is then used to perform knowledge distillation on the student model (i.e., the second-pruned model) on the training set to train a lightweight student model (i.e., a second information classification model).

[0226] In this embodiment, the second information classification model can be used to classify transaction information for virtual resource transactions in instant messaging applications. For example, when a virtual resource transaction occurs in an instant messaging application, the transaction remarks information can be obtained. If the transaction remarks information is composite text including emoticons, the emoticons in the transaction remarks information can be replaced with the corresponding descriptive text to obtain the replaced information. The second information classification model is then called in the transaction system to process the replaced information and obtain the predicted information category of the transaction information, such as high risk or low risk, so as to manage and control the risk of high-risk transactions based on the predicted information category of the transaction information.

[0227] Existing transaction information risk classification models are deployed on clusters with CPUs as the core computing units. However, a typical BERT model has approximately 100 million weight parameters and a relatively deep network, typically between 24 and 30 layers. This excessive number of model parameters and complexity results in slow inference speeds for the original BERT model, often requiring 1.5 to 2 hours to complete the inference process for day T transaction information data. Furthermore, the large model file size leads to high cluster load and network communication costs.

[0228] Experimental verification shows that the adjusted first information classification model and second classification model in this application have a 3.2-fold increase in business performance (recall rate under a 99.9% precision limit) compared to the base model. The inference speed of the compressed model (i.e., the first information classification model and the second classification model) in the cluster is accelerated by 3.4 to 4 times, the overall file size of the compressed model in the cluster is reduced to 5% of the original, and the business performance of the compressed model can reach 96% of that of the full-parameter information classification model.

[0229] As can be seen from the above, the embodiments of this application, through pre-trained model pruning and knowledge distillation training, can efficiently compress the model size, significantly improve inference speed, and maintain the model's business performance close to that of the original model with hundreds of millions of parameters. The second information classification model obtained in this application embodiment has better target classification task capabilities and a higher upper limit compared to the first information classification model. Its applicability is limited to scenarios where the task data is well-defined and the teacher model performs well.

[0230] Furthermore, traditional student model generation methods, which directly use a randomly initialized text classification model such as TextCNN or Transformer as the student model, suffer from inconsistent performance in many scenarios because they completely fail to inherit the weight parameters and structure of the teacher model. In contrast, this application's embodiment organically combines the latest pruning techniques and knowledge distillation training. The student model is generated by directly pruning the teacher model, thus inheriting its weights and depth structure. This approach results in a better performance for the adjusted second information classification model.

[0231] To better implement the above methods, this application also provides a model adjustment device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0232] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the model adjustment device specifically integrated into the server as an example.

[0233] For example, such as Figure 5 As shown, the model adjustment device may include a training acquisition unit 510, a first adjustment unit 520, a pruning unit 530, a second adjustment unit 540, and a third adjustment unit 550, as follows:

[0234] (I) Training Acquisition Unit 510

[0235] Used to obtain multiple sample information for pre-trained information processing models and target classification tasks.

[0236] In some embodiments, the model adjustment device further includes a sample processing unit, which includes a first processing subunit and a second processing subunit, comprising: a first processing subunit for acquiring multiple initial sample information; and a second processing subunit for processing the initial sample information to obtain sample information.

[0237] In some implementations, the initial sample information includes composite text, and the second processing subunit is specifically used to: determine the content to be processed from the initial sample information, wherein the content to be processed is non-text content in the composite text; and replace the content to be processed in the initial sample information with the corresponding descriptive text to obtain sample information.

[0238] In some implementations, the initial sample information includes initial negative sample information, and the sample processing unit further includes a third processing subunit, used for: performing word segmentation on the initial negative sample information to obtain a word sequence; determining candidate words from the word sequence, wherein the candidate words are unknown words whose frequency of occurrence in the word sequence meets a preset frequency condition; determining candidate non-text content corresponding to the candidate words from the initial negative sample information; and determining target non-text content from the candidate non-text content, wherein the target non-text content is used to determine the content to be processed.

[0239] In some implementations, the initial sample information includes text information, and the second processing subunit is specifically used to: search for preset stop words in the text information of the initial sample information; and remove the found preset stop words from the initial sample information to obtain sample information.

[0240] In some implementations, multiple initial sample information includes initial negative sample information, and multiple sample information includes negative sample information. The second processing subunit is specifically used to: detect target feature information in the initial negative sample information using preset information features, wherein the target feature information is information corresponding to the preset information features, and the preset information features include one or more of preset keywords and preset character rules; and use the initial negative sample information in which the target information features are detected as negative sample information.

[0241] (II) First Adjustment Unit 520

[0242] This is used to adjust the pre-trained information processing model using information from multiple samples, so as to obtain the basic information classification model corresponding to the target classification task.

[0243] (III) Pruning Unit 530

[0244] This is used to prune the model to be processed, resulting in a pruned model. The model to be processed includes either a pre-trained information processing model or a basic information classification model.

[0245] In some embodiments, the pruning unit includes a first pruning subunit, a second pruning subunit, a third pruning subunit, and a fourth pruning subunit, comprising: a first pruning subunit for constructing a computational operation graph of the model to be processed; a second pruning subunit for grouping the weight parameters of the model to be processed based on the computational operation graph to obtain at least one weight parameter group; a third pruning subunit for determining the importance of the weight parameter group to the model to be processed; and a fourth pruning subunit for pruning the weight parameter group of the model to be processed according to the importance, to obtain a pruned model.

[0246] (iv) Second Adjustment Unit 540

[0247] When the model to be processed includes a basic information classification model, the pruned model is adjusted using information from multiple samples to obtain a first information classification model, which is then used to perform the target classification task.

[0248] In some implementations, the second adjustment unit is specifically used to: when the model to be processed includes a basic information classification model, obtain multiple target sample information from multiple sample information according to a preset ratio; and adjust the pruned model using the multiple target sample information to obtain a first information classification model.

[0249] (V) Third Adjustment Unit 550

[0250] When the model to be processed includes a pre-trained information processing model, the pruned model is adjusted using the basic information classification model and multiple sample information to obtain a second information classification model, which is then used to perform the target classification task.

[0251] In some implementations, the third adjustment unit is specifically used to: when the model to be processed includes a pre-trained information processing model, use the basic information classification model as the teacher model in knowledge distillation, and use the pruned model as the student model in knowledge distillation; use multiple sample information and the teacher model to perform knowledge distillation on the student model to obtain a second information classification model.

[0252] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0253] As can be seen from the above, the model adjustment device in this embodiment includes a training acquisition unit, a first adjustment unit, a pruning unit, a second adjustment unit, and a third adjustment unit. The training acquisition unit is used to acquire multiple sample information of a pre-trained information processing model and the target classification task; the first adjustment unit is used to adjust the pre-trained information processing model using the multiple sample information to obtain a basic information classification model corresponding to the target classification task; the pruning unit is used to prune the model to be processed to obtain a pruned model, where the model to be processed includes either the pre-trained information processing model or the basic information classification model; the second adjustment unit is used, when the model to be processed includes the basic information classification model, to adjust the pruned model using the multiple sample information to obtain a first information classification model, which is used to perform the target classification task; the third adjustment unit is used, when the model to be processed includes the pre-trained information processing model, to adjust the pruned model using the basic information classification model and the multiple sample information to obtain a second information classification model, which is used to perform the target classification task.

[0254] Therefore, the embodiments of this application can utilize a two-stage adjustment and pruning strategy based on the pre-trained information processing model. This involves fine-tuning the pre-trained model once to adapt it to the target task. Furthermore, unnecessary parameters are reduced through pruning, and then fine-tuning is performed again on the pruned model. In this way, while maintaining model performance, the final model's size and number of parameters are significantly reduced, thereby alleviating the model's load, optimizing computational efficiency, and significantly improving inference speed.

[0255] To better implement the above methods, this application also provides an information classification device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0256] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the information classification device specifically integrated into the server as an example.

[0257] For example, such as Figure 6 As shown, the information classification device may include an information acquisition unit 610 and a prediction unit 620, as follows:

[0258] (I) Information Acquisition Unit 610

[0259] This is used to obtain a classification model for information to be processed and a target information. The target information classification model is obtained by any model adjustment method or information classification method provided in the embodiments of this application to obtain a first information classification model or a second information classification model.

[0260] (II) Prediction Unit 620

[0261] It is used to classify and predict the information to be processed using a target information classification model, and obtain the predicted information category corresponding to the target classification task.

[0262] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0263] As can be seen from the above, the information classification device of this embodiment includes an information acquisition unit and a prediction unit. The information acquisition unit is used to acquire the information to be processed and a target information classification model. The target information classification model is obtained by any model adjustment method or information classification method provided in this application embodiment to obtain a first information classification model or a second information classification model. The prediction unit is used to use the target information classification model to classify and predict the information to be processed, thereby obtaining the predicted information category corresponding to the target classification task.

[0264] Therefore, the embodiments of this application can utilize a two-stage adjustment and pruning strategy based on the pre-trained information processing model. This involves fine-tuning the pre-trained model once to adapt it to the target task. Furthermore, unnecessary parameters are reduced through pruning, and then fine-tuning is performed again on the pruned model. In this way, while maintaining model performance, the final model's size and number of parameters are significantly reduced, thereby alleviating the model's load, optimizing computational efficiency, and significantly improving inference speed.

[0265] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0266] In some embodiments, the model adjustment device or information classification device may also be integrated into multiple electronic devices. For example, the model adjustment device or information classification device may be integrated into multiple servers, and the model adjustment method or information classification method of this application may be implemented by multiple servers.

[0267] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 7 As shown, it illustrates a schematic diagram of the server structure involved in an embodiment of this application. Specifically:

[0268] The server may include components such as a processor 710 with one or more processing cores, a memory 720 with one or more computer-readable storage media, a power supply 730, an input module 740, and a communication module 750. Those skilled in the art will understand that... Figure 4 The server structure shown does not constitute a limitation on the server and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0269] The processor 710 is the control center of the server, connecting various parts of the server via various interfaces and lines. It performs various server functions and processes data by running or executing software programs and / or modules stored in the memory 720, and by accessing data stored in the memory 720. In some embodiments, the processor 710 may include one or more processing cores; in some embodiments, the processor 710 may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 710.

[0270] The memory 720 can be used to store software programs and modules. The processor 710 executes various functional applications and data processing by running the software programs and modules stored in the memory 720. The memory 720 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 720 may also include a memory controller to provide the processor 710 with access to the memory 720.

[0271] The server also includes a power supply 730 that supplies power to the various components. In some embodiments, the power supply 730 can be logically connected to the processor 710 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 730 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0272] The server may also include an input module 740, which can be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0273] The server may also include a communication module 750. In some embodiments, the communication module 750 may include a wireless module, through which the server can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 750 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0274] Although not shown, the server may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 710 in the server loads the executable files corresponding to the processes of one or more applications into the memory 720 according to the following instructions, and the processor 710 runs the applications stored in the memory 720 to realize various functions, as follows:

[0275] The pre-trained information processing model is adjusted to obtain the basic information classification model corresponding to the target classification task. The model to be processed is pruned to obtain the pruned model, which includes either the pre-trained information processing model or the basic information classification model. When the model to be processed includes the basic information classification model, the pruned model is adjusted using multiple sample information to obtain the first information classification model, which is used to perform the target classification task. When the model to be processed includes the pre-trained information processing model, the pruned model is adjusted using the basic information classification model and multiple sample information to obtain the second information classification model, which is used to perform the target classification task.

[0276] Alternatively, obtain the information to be processed and the target information classification model. The target information classification model is obtained by any model adjustment method or information classification method provided in the embodiments of this application to obtain a first information classification model or a second information classification model. Use the target information classification model to classify and predict the information to be processed to obtain the predicted information category corresponding to the target classification task.

[0277] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0278] As can be seen from the above, the embodiments of this application can utilize a two-stage adjustment and pruning strategy based on the pre-trained information processing model. This allows the model to be fine-tuned once to adapt to the target task. Furthermore, unnecessary parameters are reduced through pruning, and further fine-tuning is performed on the pruned model. In this way, while ensuring model performance, the size and number of parameters of the final model are significantly reduced, thereby reducing the model load, optimizing the model's computational efficiency, and significantly improving the model's inference speed.

[0279] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0280] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the model adjustment methods or information classification methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0281] The pre-trained information processing model is adjusted to obtain the basic information classification model corresponding to the target classification task. The model to be processed is pruned to obtain the pruned model, which includes either the pre-trained information processing model or the basic information classification model. When the model to be processed includes the basic information classification model, the pruned model is adjusted using multiple sample information to obtain the first information classification model, which is used to perform the target classification task. When the model to be processed includes the pre-trained information processing model, the pruned model is adjusted using the basic information classification model and multiple sample information to obtain the second information classification model, which is used to perform the target classification task.

[0282] Alternatively, obtain the information to be processed and the target information classification model. The target information classification model is obtained by any model adjustment method or information classification method provided in the embodiments of this application to obtain a first information classification model or a second information classification model. Use the target information classification model to classify and predict the information to be processed to obtain the predicted information category corresponding to the target classification task.

[0283] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0284] According to one aspect of this application, a computer program product or computer program is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of the methods provided in the various optional implementations of the above embodiments. The computer program / instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0285] Since the instructions stored in the storage medium can execute the steps of any of the model adjustment methods or information classification methods provided in the embodiments of this application, the beneficial effects that any of the model adjustment methods or information classification methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0286] The above provides a detailed description of a model adjustment method, an information classification method, and related equipment provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model adjustment method, characterized in that, include: Obtain multiple sample information for pre-trained information processing models and target classification tasks; The pre-trained information processing model is adjusted using the information from the multiple samples to obtain the basic information classification model corresponding to the target classification task. The model to be processed is pruned to obtain a pruned model, wherein the model to be processed includes either the pre-trained information processing model or the basic information classification model. When the model to be processed includes the basic information classification model, the pruned model is adjusted using the multiple sample information to obtain a first information classification model, which is used to perform the target classification task. When the model to be processed includes the pre-trained information processing model, the pruned model is adjusted using the basic information classification model and the multiple sample information to obtain a second information classification model, which is used to perform the target classification task.

2. The model adjustment method as described in claim 1, characterized in that, When the model to be processed includes the basic information classification model, the pruned model is adjusted using the multiple sample information to obtain the first information classification model, including: When the model to be processed includes the basic information classification model, multiple target sample information is obtained from the multiple sample information according to a preset ratio; The pruned model is adjusted using the information from the multiple target samples to obtain the first information classification model.

3. The model adjustment method as described in claim 1, characterized in that, When the model to be processed includes the pre-trained information processing model, the pruned model is adjusted using the basic information classification model and the multiple sample information to obtain a second information classification model, including: When the model to be processed includes the pre-trained information processing model, the basic information classification model is used as the teacher model in knowledge distillation, and the pruned model is used as the student model in knowledge distillation. The student model is subjected to knowledge distillation using the multiple sample information and the teacher model to obtain the second information classification model.

4. The model adjustment method as described in claim 1, characterized in that, The process of pruning the model to be processed to obtain the pruned model includes: Construct the computational operation graph of the model to be processed; Based on the computational operation graph, the weight parameters of the model to be processed are grouped to obtain at least one weight parameter group. Determine the importance of the weight parameter set to the model to be processed; Based on the importance level, the weight parameter group of the model to be processed is pruned to obtain the pruned model.

5. The model adjustment method according to any one of claims 1 to 4, characterized in that, Before acquiring multiple sample information from the pre-trained information processing model and the target classification task, the process also includes: Obtain information from multiple initial samples; The initial sample information is processed to obtain sample information.

6. The model adjustment method as described in claim 5, characterized in that, The initial sample information includes composite text, and the sample processing of the initial sample information to obtain sample information includes: The content to be processed is determined from the initial sample information, wherein the content to be processed is the non-text content in the composite text; The content to be processed in the initial sample information is replaced with the corresponding descriptive text to obtain the sample information.

7. The model adjustment method as described in claim 5, characterized in that, The plurality of initial sample information includes initial negative sample information. Before performing sample processing on the initial sample information to obtain sample information, the method further includes: The initial negative sample information is segmented into words to obtain a word sequence; Determine candidate words from the word sequence, wherein the candidate words are unknown words whose frequency of occurrence in the word sequence meets a preset frequency condition; Determine the candidate non-text content corresponding to the candidate word from the initial negative sample information; From the candidate non-text content, a target non-text content is determined, and the target non-text content is used to determine the content to be processed.

8. The model adjustment method as described in claim 5, characterized in that, The initial sample information includes text information, and the sample processing of the initial sample information to obtain sample information includes: Search for preset stop words in the text information of the initial sample information; The sample information is obtained by removing the preset stop words found from the initial sample information.

9. The model adjustment method as described in claim 5, characterized in that, The plurality of initial sample information includes initial negative sample information, and the plurality of sample information includes negative sample information. The step of processing the initial sample information to obtain sample information includes: Using preset information features, target feature information in the initial negative sample information is detected. The target feature information is information corresponding to the preset information features. The preset information features include one or more of preset keywords and preset character rules. The initial negative sample information that detects the target information feature is used as the negative sample information.

10. An information classification method, characterized in that, include: A classification model for information to be processed and target information is obtained, wherein the target information classification model is obtained by the method described in any one of claims 1 to 9, which yields a first information classification model or a second information classification model. The target information classification model is used to classify and predict the information to be processed, thereby obtaining the predicted information category corresponding to the target classification task.

11. A model adjustment device, characterized in that, include: The training acquisition unit is used to acquire multiple sample information for the pre-trained information processing model and the target classification task; The first adjustment unit is used to adjust the pre-trained information processing model using the multiple sample information to obtain the basic information classification model corresponding to the target classification task. The pruning unit is used to prune the model to be processed to obtain the pruned model. The model to be processed includes either the pre-trained information processing model or the basic information classification model. The second adjustment unit is used to adjust the pruned model using the multiple sample information when the model to be processed includes the basic information classification model, to obtain a first information classification model, and the first information classification model is used to perform the target classification task. The third adjustment unit is used to adjust the pruned model using the basic information classification model and the multiple sample information when the model to be processed includes the pre-trained information processing model, to obtain a second information classification model, which is used to perform the target classification task.

12. An information classification device, characterized in that, include: An information acquisition unit is used to acquire information to be processed and a target information classification model, wherein the target information classification model is obtained by the method described in any one of claims 1 to 9 to obtain a first information classification model or a second information classification model; The prediction unit is used to classify and predict the information to be processed using the target information classification model to obtain the predicted information category corresponding to the target classification task.

13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the model adjustment method or information classification method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the model adjustment method or information classification method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the model adjustment method or information classification method according to any one of claims 1 to 10.