Category-level recall control method and device, electronic equipment and storage medium

Abstract

This disclosure provides a category-level recall control method, device, electronic device, and storage medium, involving natural language processing and deep learning technologies in the field of artificial intelligence. It can be applied to multi-category classification scenarios in high-risk businesses such as content security review, financial risk control, and public opinion monitoring. The specific implementation scheme is as follows: Input samples are classified based on a pre-set multi-category classification model to obtain calibration scores for each category on the input samples; based on pre-configured target recall rates for each category, and combined with a pre-built mapping table of recall rates and thresholds, a standard threshold corresponding to the target recall rate of each category is determined; the category with the highest calibration score among the multiple categories is selected as the candidate predicted category for the input sample; based on the candidate predicted category, a standard threshold for the candidate predicted category is determined from the standard thresholds corresponding to the target recall rates of each category; based on the calibration score of the candidate predicted category and the corresponding standard threshold, it is determined whether to recall the input sample.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, particularly to natural language processing and deep learning technologies in the field of artificial intelligence. Specifically, it relates to category-level recall control methods, devices, electronic equipment, and storage media, which can be applied to multi-category classification scenarios in high-risk businesses such as content security review, financial risk control, and public opinion monitoring. Background Technology

[0002] For large-scale, multi-category classification scenarios (such as multi-category text classification scenarios), especially for high-risk businesses such as content security review, financial risk control, and public opinion monitoring, there is currently a lack of effective means to accurately control the recall rate of specific categories. Summary of the Invention

[0003] This disclosure provides category-level recall control methods, apparatus, electronic devices, and storage media that can solve the problem of how to accurately control the recall rate of a specific category.

[0004] In a first aspect, embodiments of this disclosure provide a category-level recall control method, including: The input samples are classified based on a pre-defined multi-class classification model to obtain calibration scores for each class on the input samples. Based on the pre-configured target recall rates for each category, and combined with a pre-built mapping table between recall rate and threshold, a standard threshold corresponding to the target recall rate for each category is determined. The category with the highest calibration classification among the multiple categories is taken as the candidate predicted category of the input sample; Based on the candidate prediction category, the standard threshold of the candidate prediction category is determined from the standard threshold corresponding to the target recall rate of each category; Based on the calibration score of the candidate prediction category and the standard threshold of the candidate prediction category, it is determined whether to recall the input sample.

[0005] Secondly, embodiments of this disclosure provide a category-level recall control device, comprising: The classification module is used to classify the input sample based on a preset multi-class classification model and obtain the calibration score of each class on the input sample; The first determining module is used to determine the standard threshold corresponding to the target recall rate of each category based on the pre-configured target recall rate of each category and in combination with the pre-built mapping table of recall rate and threshold. The second determining module is used to select the category with the highest calibration classification among the multiple categories as the candidate predicted category of the input sample; The third determining module is used to determine the standard threshold of the candidate prediction category from the standard thresholds corresponding to the target recall rates of each category based on the candidate prediction category; The recall determination module is used to determine whether to recall the input sample based on the calibration score of the candidate prediction category and the standard threshold of the candidate prediction category.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0007] Fourthly, embodiments of this disclosure provide an intelligent agent, including: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and execute the method described in the first aspect by calling the large model to obtain output information; An output module is used to output the output information obtained by the processing module.

[0008] Fifthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in the first aspect above.

[0009] In a sixth aspect, embodiments of this disclosure provide a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0010] According to the technical solution disclosed herein, precise recall of samples of a specific category can be achieved. Furthermore, based on a given category-level target recall rate and the standard thresholds corresponding to each target recall rate, precise control of the recall rate for a specific category can be achieved, enabling differentiated recall control. Through the configuration interface for the category-level target recall rate and the threshold mapping table, a multi-class classification system can possess "configurable, measurable, and closed-loop" category-level recall control capabilities.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of the category-level recall control method provided in the embodiments of this disclosure; Figure 2 This is a flowchart of the training method for the pre-trained model provided in the embodiments of this disclosure; Figure 3 This is a flowchart of the category-level recall control method provided in the embodiments of this disclosure; Figure 4 This is a block diagram of a category-level recall control device provided in an embodiment of this disclosure; Figure 5 This is a block diagram of the intelligent agent provided in the embodiments of this disclosure; Figure 6 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] This disclosure relates to the fields of artificial intelligence technologies such as natural language processing and deep learning.

[0015] Artificial Intelligence (AI) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0016] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. It is a discipline that uses computer technology to analyze, understand, and process natural language, treating computers as a powerful tool for language research. With computer support, it conducts quantitative research on language information and provides language descriptions that can be used by both humans and computers.

[0017] Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound.

[0018] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0020] The following description, with reference to the accompanying drawings, outlines a category-level recall control method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0021] It should be noted that the executing entity of the category-level recall control method in this embodiment of the present disclosure can be a category-level recall control device. This device can be implemented by software and / or hardware and can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0022] It is worth noting that the category-level recall control method of this disclosure embodiment can be implemented by an autonomous agent based on a large model, which can precisely control the recall of a specific category based on the given target recall rate of each category.

[0023] Figure 1 This is a flowchart of a category-level recall control method provided in an embodiment of this disclosure. For example... Figure 1 As shown, this category-level recall control method may include, but is not limited to, the following steps.

[0024] In step 101, the input samples are classified based on a preset multi-class classification model to obtain the calibration scores of each class on the input samples.

[0025] In embodiments of this disclosure, the multi-class classification model can be a multi-class text classification model used to complete a multi-class text classification task. In this task scenario, the input sample can be an input text sample. For example, the input text sample can be subjected to multi-class text classification to predict its final category. Alternatively, in embodiments of this disclosure, the multi-class classification model can be a multi-class image classification model used to complete a multi-class image classification task. In this task scenario, the input sample can be an input image sample. For example, the input image sample can be subjected to multi-class image classification to predict its final category.

[0026] In some embodiments, the above-mentioned multi-class classification model can be a model obtained by adapting a small number of trainable parameters to a specific downstream classification task while keeping the main parameters of the pre-trained model (such as a pre-trained language model) basically frozen.

[0027] In one alternative implementation, a trainable low-rank factorization matrix can be added to the key linear layers of a pre-trained model (such as a pre-trained language model); a unified scoring space based on the conditional probability of the label sequence can be constructed, and the pre-trained model can be trained by combining cross-entropy loss, supervised contrastive loss, and boundary loss to adjust the parameters of the trainable low-rank factorization matrix. The trained pre-trained model can then be used as the aforementioned multi-class classification model.

[0028] In some embodiments, the calibration score may refer to the score obtained by calibrating the original score (or logits) output from the last layer of a multi-class classification model. For example, the standard score for each class may refer to the score obtained by calibrating the original score (or logits) of each class output from the last layer of a multi-class classification model using the temperature parameters of each class.

[0029] In step 102, based on the pre-configured target recall rates for each category and combined with a pre-built mapping table of recall rates and thresholds, a standard threshold corresponding to the target recall rate for each category is determined.

[0030] In the embodiments of this disclosure, the target recall rate for each of the above categories can be determined according to actual business needs. For example, for large-scale multi-category text classification scenarios, especially for high-risk businesses such as content security review, financial risk control, and public opinion monitoring, where there is a need for "precise control of the recall rate of a specific category", the target recall rate for that specific category can be pre-configured. For example, a target recall rate (such as a recall target of 99.9%) can be set separately for a certain high-risk category, and independent target recall rates can also be set for other categories.

[0031] In the embodiments of this disclosure, given the target recall rates for each category, a standard threshold corresponding to the target recall rate of each category can be determined based on the pre-configured target recall rates for each category and a pre-built mapping table of recall rates and thresholds. For example, for each category, if a threshold corresponding to the target recall rate of that category exists in the mapping table, that threshold can be directly used as the standard threshold for that category. If a threshold corresponding to the target recall rate of that category does not exist in the mapping table, the standard threshold corresponding to the target recall rate of that category can be determined by interpolation based on the mapping table.

[0032] In some embodiments, the above-mentioned recall-threshold mapping table can be constructed based on empirical values, such as relying on domain knowledge, historical data, or human judgment to design and populate the recall-threshold mapping table.

[0033] In some embodiments, the aforementioned recall-threshold mapping table can be constructed based on the optimal threshold adjustment strategy parameters of the multi-class threshold under the category-level recall objective, and the category-level recall objective itself. In one possible implementation, multi-class threshold tuning can be modeled as a Markov Decision Process (MDP). The optimal threshold adjustment strategy parameters of the multi-class threshold under the category-level recall objective are learned through trial and error using a reward function. Based on the category-level recall objective and the optimal threshold adjustment strategy parameters of the multi-class threshold under the category-level recall objective, the mapping data between recall and threshold is determined, and a recall-threshold mapping table is constructed based on the mapping data.

[0034] For example, multi-class thresholds can be optimized based on reinforcement learning, and a mapping table between recall and thresholds can be constructed based on the optimized multi-class thresholds. Specifically, during the threshold optimization process, the system state at each iteration can be represented as a vector s. t This state may include, but is not limited to, the following information: the threshold vectors for the current category. τ (t) =( τ 1 (t) ,…, τ K (t) In the validation set Above, the recall rates for each category after applying the current threshold: Precision for each category under the same threshold configuration: Optionally, this state may also include distribution statistics, such as the mean and variance of the score distributions for each category, and the overall macro / micro F1 score. The state vector s constructed from the above information... t This describes the overall performance of the system under the current threshold configuration.

[0035] Define the reinforcement learning process at time 10:00 t action a t The adjustment amount to the current threshold vector can be modeled as a K-dimensional continuous vector: This formula represents the new round's threshold = the current round's threshold + the change in this adjustment. Where, the first... t The threshold vector at the next iteration contains K numbers; a t The suggestion given by this round of reinforcement learning regarding "how to adjust the threshold" is also a K-dimensional vector. By imposing appropriate boundary constraints on the action (e.g., limiting the threshold to a legal range), the updated threshold vector can be ensured. τ (t+1) It has always remained within a reasonable range.

[0036] In order to learn a threshold strategy that satisfies both recall and precision, this disclosure designs a reward function of the following form to measure the performance of the current state and action:

[0037] in: This is an indicator function that provides a fixed reward when the recall rate of a certain category reaches or exceeds the target. Describes the deviation between the current recall rate and the target recall rate, with a coefficient. Control the intensity of the penalty for deviation; The coefficient reflects the false alarm rate of the current category. This design balances the weights between recall and precision. Through this approach, reinforcement learning tends to optimize by making the recall rates for each class approach or even slightly exceed the target. While meeting recall constraints, we should strive to improve accuracy and reduce false alarms.

[0038] Based on the above definitions of state, action, and reward function, the multi-class threshold optimization problem can be modeled as a Markov decision process (MDP). During the training phase: (1) Initialize the threshold vector τ (0) (For example, starting from a globally unified threshold or heuristic configuration); (2) Under the current threshold configuration, utilize the validation set calculate , Indicators, construct state s t ; (3) Based on the current strategy Sampling action a t Update the threshold vector τ (t+1) ; (4) Re-evaluate the validation set performance using the new threshold to obtain the next state s. t+1 and corresponding rewards ; (5) Repeat the above process, collect the state-action-reward-next state sequence, and update the policy parameters using reinforcement learning algorithms such as policy gradient and proximal policy optimization (PPO) to maximize the long-term cumulative reward.

[0039] The optimal strategy is obtained through multiple rounds of iteration. It can automatically provide near-optimal threshold adjustment schemes in high-dimensional threshold spaces, ensuring that the threshold vector converges under the constraints of this strategy. On the validation set, the following conditions must be met simultaneously: the recall rate of each category should be as close as possible to the preset target. Overall accuracy and false positive levels remain within acceptable business ranges. After training, the policy output or convergence threshold configuration results can be solidified into "recall-threshold" mapping data, which facilitates the construction of a mapping table between recall and threshold based on the mapping data.

[0040] Optionally, in some embodiments, the above-mentioned recall-threshold mapping table can be expanded in the following way: for each category, a set of calibration scores corresponding to each category is collected on the validation set using a multi-class classification model; within the score range of each category, a set of discrete threshold points are selected, and for each threshold point, the corresponding recall rate is calculated on the validation set; each threshold point and the corresponding recall rate are expanded to the recall-threshold mapping table.

[0041] For example, for each category c In the validation set Using the model that has already undergone temperature calibration, we collect a set of calibration scores (i.e., post-calibration scores) corresponding to this category: ,here This indicates that under the current model and temperature parameters, the sample... x Category c The final score is used for statistical analysis when scanning thresholds in subsequent categories. c Within the range of scores, a set of discrete threshold points can be selected. This can be generated using methods such as uniform partitioning or adaptive partitioning based on quantiles. For each threshold... On the validation set, the corresponding recall and precision are calculated, where recall represents the percentage of true positives and negatives when all true positives are true. c The formula for determining what percentage of the samples have scores not lower than the threshold is: Precision indicates the percentage of scores that are not lower than the threshold and are predicted as category. c What percentage of the sample truly belongs to [the group / group]? c The sample, the formula is: Through the above calculations, for each threshold point... Each of these can yield a set of triples: Record the triple as a category. c The mapped data points form an approximately monotonic "threshold-recall" curve, along with corresponding precision information. This threshold-recall mapping data can be expanded to the aforementioned recall-threshold mapping table, thereby enriching the mapping table.

[0042] Optionally, the above mapping table between recall and threshold can support piecewise linear interpolation to obtain a standard threshold for the category. For example, for any given target recall... In category c Find the data points that meet the "recall-threshold" criteria. Two nearest neighbors and Among them, indicators l , u This represents two adjacent sampling positions on the recall-threshold mapping table (also called the recall-threshold curve). Subsequently, piecewise linear interpolation is used to estimate the threshold corresponding to the target recall rate:

[0043] This formula means that given two "(recall, threshold)" points, we want a recall rate that falls somewhere in between. The threshold corresponding to time Then treat it as drawing a straight line between these two points, and then "find the median value" proportionally. This interpolation method, while ensuring simple and efficient computation, can provide a high-precision threshold estimate between discrete sampling points. Even if the target recall does not fall exactly on a pre-sampled point, it can quickly obtain the corresponding approximate threshold.

[0044] Optionally, all categories can be selected. c The mapping data for values ​​= 1, ..., K, along with interpolation rules, are summarized to construct a multidimensional recall-threshold mapping table. During the online deployment phase, the system receives pre-configured recall rates for each category of targets. Then, interpolation queries can be performed independently for each category based on the mapping table to obtain the corresponding threshold. And the threshold set for all categories This serves as the decision threshold vector for the current model, used for real-time inference and recall control. Through the aforementioned mapping table construction and interpolation strategy, this disclosure enables a fast, stable, and low-cost reverse lookup capability from "target recall rate" to "specific threshold configuration," providing an engineering-feasible infrastructure for fine-grained control of online multi-class recall rates.

[0045] It should be noted that in some embodiments, the execution order of steps 101 and 102 can be interchanged or they can be executed simultaneously.

[0046] In step 103, the class with the highest calibration score among the multiple classes is selected as the candidate predicted class for the input sample.

[0047] In the embodiments of this disclosure, after classifying the input sample using a multi-class classification model and obtaining the calibration score of each class on the input sample, the class with the highest calibration score among the multiple classes can be used as the candidate predicted class of the input sample.

[0048] For example, suppose the multi-class classification model corresponds to class 1, class 2 and class 3. When the input sample is classified using the multi-class classification model, the calibration score of class 1 on the input sample is A, the calibration score of class 2 on the input sample is B, and the calibration score of class 3 on the input sample is C, where A>B>C. The class 1 corresponding to the highest calibration score A can be used as the candidate predicted class of the input sample.

[0049] It should be noted that in some embodiments, the execution order of steps 102 and 103 can be interchanged or they can be executed simultaneously.

[0050] In step 104, a standard threshold for the candidate prediction category is determined from the standard thresholds corresponding to the target recall rates of each category.

[0051] For example, since the candidate prediction category is one of the categories mentioned above, the standard threshold corresponding to the candidate prediction category can be determined from the standard threshold corresponding to the target recall rate of each category.

[0052] In step 105, based on the calibration score of the candidate predicted category and the standard threshold of the candidate predicted category, it is determined whether to recall the input sample.

[0053] In the embodiments of this disclosure, the calibration score of a candidate prediction category can be compared with a standard threshold for that category. If the calibration score is greater than or equal to the standard threshold, the model's confidence in that category is considered high enough, and the input sample can be recalled. The prediction result can then be accepted, and the candidate prediction category can be output as the final prediction category for the input sample. If the calibration score is less than the standard threshold, the model's confidence in that category is considered insufficient. In this case, a definitive category cannot be given directly. Optionally, the input sample can be marked as a "low-confidence" sample and further processed according to business strategies. For example, it can be transferred to a manual review queue, submitted to a candidate model or multi-model integration for re-evaluation, or marked as an observation sample for subsequent model iterations.

[0054] Optionally, this disclosure can support online monitoring and dynamic adjustment of recall behavior for each category. For example, it can statistically analyze or periodically evaluate the actual recall rate, precision, F1 score, and other metrics of each category on real data, and compare them with the target recall rate. Alternatively, a comparison can be made using preset strategies. When the actual recall rate for certain categories deviates from the target value, or the overall false positive rate exceeds the acceptable range for the business, the business can update the target recall rate, risk level, or priority through the configuration center. Upon receiving the new configuration, the system can immediately recalculate the thresholds for each category using mapping tables and interpolation strategies. And apply the new threshold configuration without restarting the service, so that the recall control strategy can take effect quickly.

[0055] In the above embodiments, a multi-class classification model can be used to classify input samples, obtaining calibration scores for each class on the input sample. Based on these calibration scores, candidate predicted classes for the input sample are determined. According to pre-configured target recall rates for each class, combined with a pre-built recall-threshold mapping table, the corresponding standard threshold for the candidate predicted class is determined. A classification decision is made based on the standard score and standard threshold of the candidate predicted class to determine whether to recall the input sample. This enables precise recall of samples of a specific class. Furthermore, based on a given class-level target recall rate and the corresponding standard threshold, precise control of the recall rate for a specific class can be achieved, enabling differentiated recall control. Through the configuration interface for class-level target recall rates and the threshold mapping table, the multi-class classification system can possess "configurable, measurable, and closed-loop" class-level recall control capabilities.

[0056] It should be noted that the aforementioned multi-class classification model can be obtained by adapting a small number of trainable parameters to a specific downstream classification task while keeping the main parameters of the pre-trained model (such as a pre-trained language model) essentially frozen. Optionally, in some embodiments, Figure 2 This is a flowchart of the training method for the pre-trained model provided in this embodiment of the disclosure. For example... Figure 2 As shown, the training method for this pre-trained model may include, but is not limited to, the following steps.

[0057] In step 201, a trainable low-rank decomposition matrix is ​​added to the key linear layers in the pre-trained model.

[0058] For example, taking a multi-class text classification task as an example, the multi-class classification model can be a multi-class text classification model. The input text space can be defined. tag space , where K is the number of classes. The training data is denoted as: , representing N samples, each a tuple of "text + label". For each category Construct the corresponding label text or label token sequence label k For example, the category "sports news" corresponds to the English word "sports news," which is used for subsequent perplexity scoring. Define the classification function. f θ , represents inputting a text x Output a vector of length K (each dimension corresponds to a category score or logit), denoted by the formula: ,in, θ These are the trainable parameters for the model, and the output is the raw scores or logits vectors for each category.

[0059] Introduce a set of category-level recall targets, denoted by the following formula: , representing for each category c The business side can provide a "desired recall rate", for example: Category 1: Category 2: In this way, a business requirement such as "recalling classes back to 99%" can be transformed into a mathematically defined target variable. This allows it to enter the optimization process.

[0060] In embodiments of this disclosure, a trainable low-rank decomposition matrix can be added to the key linear layers of the pre-trained model; for example, a low-rank adapter LoRA can be added to the key linear layers of the pre-trained model. The original weights are then integrated. Not given directly W Instead of adding a large matrix of the same size to 0, it forces: , where ΔW= BA , In other words, use first. A Compress the input from k dimensions to r dimensions (a very thin bottleneck); then use B Pull the r-dimensional dimension back to the d-dimensional dimension; thus, ΔW is actually a small modification with a very low rank.

[0061] During training W 0. Completely frozen, no updates; training low-rank factorization matrix. A, B These two sets of small matrices, along with the necessary classification head parameters, reduce the number of trainable parameters from hundreds of millions to millions or hundreds of thousands. However, because many such LoRA modules are attached to each layer, the overall expressive power remains strong, making it suitable for downstream tasks such as multi-class, multi-label text classification.

[0062] In step 202, the training samples are input into the pre-trained model to obtain the scores of each category of the output of the last layer in the pre-trained model.

[0063] In embodiments of this disclosure, training samples can be classified using a pre-trained model to obtain scores for each category output by the last layer of the pre-trained model. In these embodiments, the category scores can be constructed based on the log-likelihood of the label sequences.

[0064] In the embodiments of this disclosure, a unified scoring space based on the conditional probability of labeled text can be constructed. This facilitates joint optimization using cross-entropy, supervised contrastive loss, and boundary loss, increasing the interval between positive and negative sample scores and simplifying subsequent threshold control. In other words, the model's judgment that "the text belongs to a certain category" can be transformed into a log-likelihood score of the corresponding label token sequence, making scores between different categories comparable. Simultaneously, through contrastive learning and large-margin constraints, the score distribution becomes more "segmentable," thereby achieving stable recall control within the same scoring space.

[0065] For example, for any input text x And for category k, firstly, a corresponding label token sequence is predefined for category k. k In the parameters of a pre-trained model (such as a pre-trained language model) θ Order The model represents the performance of a given text. x This relates to the conditional probability of generating the tag token t under its context. Therefore, the confidence level of category k can be defined as the log-likelihood of the tag sequence, as shown in the formula: ,in, s k ( x The larger the value, the more the model tends to interpret the text as... xIt belongs to category k; and its negative value can be regarded as the perplexity of the label, which is used to measure the "uncertainty" of the model for that category.

[0066] In step 203, the multi-class cross-entropy loss based on the class scores is determined according to the scores corresponding to the true class labels of the training samples and the scores of each class.

[0067] For example, for supervised labeled sample pairs ( x , y Based on the above scores, a standard multi-class cross-entropy loss can be directly constructed, and the formula can be expressed as follows:

[0068] in, x For input samples; y The true class label (in 1…K). This loss ensures the model's basic predictive power, giving the correct class a higher activation value on the score.

[0069] In the embodiments of this disclosure, the multi-class cross-entropy loss based on the class scores can be calculated using the above-described formula for multi-class cross-entropy loss, based on the scores corresponding to the true class labels of the training samples and the scores of each class.

[0070] In step 204, the supervised contrast loss is determined based on the class score similarity of samples of the same class and the class score similarity of samples of different classes in the training samples.

[0071] It should be noted that, in order to further improve the separability between different categories, a supervised contrastive learning objective is constructed within a training batch. Let: pairs of samples of the same class be... The sets of negative samples of different categories are sim() is a similarity function (such as cosine similarity based on fractional vectors). If the temperature parameter is used, then the supervised comparison loss can be defined as:

[0072] This loss can bring the score representations of samples of the same class closer together and push the score representations of samples of different classes further apart, so that the model can form a clearer cluster structure in the score space.

[0073] In the embodiments of this disclosure, the supervised contrast loss can be calculated using the above-described formula for calculating supervised contrast loss, based on the class score similarity of samples of the same class and the class score similarity of samples of different classes in the training samples.

[0074] In step 205, the boundary loss is determined by using large-margin constraints between categories based on the scores corresponding to the true category labels of the training samples and the scores of each category.

[0075] To ensure a clear score gap between the correct class and the highest negative class, a margin constraint loss can be introduced, which can be expressed as follows:

[0076] in, m >0 represents a predefined minimum interval between categories; this loss forces the correct class score to be at least higher than the nearest negative class score. m This enhances the separability of the distribution boundary.

[0077] In the embodiments of this disclosure, the boundary loss can be calculated using the above-described margin constraint loss calculation formula based on the scores corresponding to the true category labels of the training samples and the scores of each category.

[0078] It should be noted that in some embodiments, the execution order of steps 203, 204, and 205 can be interchanged or they can be executed simultaneously.

[0079] In step 206, the joint loss is determined based on the multi-class cross-entropy loss, supervised contrast loss, and boundary loss.

[0080] In embodiments of this disclosure, the joint loss can be obtained by weighted summation of the multi-class cross-entropy loss, supervised contrast loss, and boundary loss. For example, the formula for the joint loss can be expressed as follows: ,in, λ , β ≥0 is a hyperparameter that balances the contributions of different loss terms.

[0081] In step 207, the pre-trained model is trained according to the joint loss.

[0082] In the embodiments of this disclosure, by minimizing the joint loss, the pre-trained model can be trained within the LoRA parameter-efficient fine-tuning framework, resulting in a classification model with clear score boundaries, stable sample spacing, and well-suited for "threshold-centric" recall control strategies.

[0083] In the above embodiments, while keeping the main parameters of the pre-trained model basically frozen, a small number of trainable parameters can be used to adapt to specific downstream classification tasks, thereby reducing computational and storage costs. By finely adjusting the key linear transformations through low-rank incremental update matrices, the model can obtain the ability to discriminate the current task (e.g., text classification including multiple categories) without changing the overall structure.

[0084] It should be noted that, to mitigate the differences in confidence distribution patterns between different categories and to improve the comparability of calibrated scores across categories, providing a reliable basis for setting thresholds and recall targets by category, score calibration can be performed based on category-specific temperatures. For example, by learning a temperature parameter separately for each category, the original logits can be mapped to a probability distribution that better matches the true accuracy. This results in more stable recall and precision performance across different categories at the same score threshold, reducing systemic bias caused by inherent imbalances or differences in difficulty between categories. Optionally, Figure 3 This is a flowchart of a category-level recall control method provided in an embodiment of this disclosure. For example... Figure 2 As shown, this category-level recall control method may include, but is not limited to, the following steps.

[0085] In step 301, the input sample is classified based on a preset multi-class classification model to obtain the scores of each category output by the last layer of the multi-class classification model.

[0086] In the embodiments of this disclosure, for an input sample, a pre-trained multi-class classification model can be used to classify the input sample to obtain the scores of each category output by the last layer of the multi-class classification model, which facilitates the subsequent calibration of the scores of the corresponding categories based on the temperature parameters of each category.

[0087] In step 302, the scores of each category are calibrated based on the temperature parameters of each category to obtain the calibration score of each category on the input sample.

[0088] In embodiments of this disclosure, for each category's score, the score of that category can be calibrated using the category's temperature parameter. For example, the ratio of the category's score to the category's temperature parameter can be used as the calibration score of that category on the input sample.

[0089] It should be noted that, in some embodiments, the temperature parameters for each category described above can be constructed based on empirical values, such as relying on domain knowledge, historical data, or manual judgment to design and populate the temperature parameters for each category. Alternatively, the temperature parameters for each category described above can be pre-learned. In one possible implementation, the temperature parameters for each category described above are pre-learned in the following way: classifying validation samples based on a multi-class classification model to obtain scores for each category output by the last layer of the multi-class classification model; calibrating the scores for each category using a set of learnable temperature parameters to obtain calibration scores for each category, wherein the set of temperature parameters includes the temperature parameters for each category; and jointly optimizing each temperature parameter in the set of temperature parameters by minimizing the negative log-likelihood loss based on the calibration scores.

[0090] For example, for any input sample xLet the original score of the multi-class classification model after fine-tuning with the aforementioned trainable low-rank factorization matrix (such as LoRA) and perplexity contrastive learning be the output of the last layer: .in, This represents the model's relative preference for class k before softmax and calibration. At this stage, the logits of different classes may exhibit systematic differences in numerical scale and distribution. Directly using them for comparison and setting a uniform threshold can easily lead to inconsistencies between recall and precision performance for each class. To characterize the inherent uncertainty differences between different classes and to normalize and correct them, this disclosure provides a method for each class... c Introduce an independent temperature parameter T c >0, based on this, the original logits are temperature-scaled, and the standardized score (i.e., the standard score) is defined as follows: , when T k When the score is greater than 1, the overall score for that category is "compressed," and the probability distribution after Softmax tends to flatten, which is used to reduce the originally excessively high confidence level of that category; when T k When the score is less than 1, the score for that category is "amplified," which helps to increase its relative weight in the overall distribution.

[0091] In independent validation sets The present disclosure can jointly learn the set of temperature parameters by minimizing the negative log-likelihood loss based on the calibration score. For example, for any sample The calibrated scores can be used first. Calculate the softmax probability, which represents the probability given a temperature parameter. Then, the model considers the sample x Belongs to the real category y The probability of is given by the following formula:

[0092] Subsequently, the negative log-likelihood was used as the calibration target, representing the value on the validation set. Find a set of temperature parameters. Let the real labels y probability To make it as large as possible, the formula can be expressed as follows:

[0093] In practice, the expectation of the above formula can be approximated by summing and averaging a finite number of samples on the validation set; during optimization, other model parameters (including...) are kept constant. θ(e.g., low-rank decomposition matrix, etc.) are fixed, only for It performs updates, thereby learning specific temperature parameters for each category with minimal additional overhead.

[0094] It is worth noting that once the temperature parameters have converged, a set of optimal temperature parameters can be obtained. At this point, for any input sample x All use As the calibrated base score (i.e., calibration score) for category k, where This represents the temperature parameter for category k. Optionally, this standard score space can be used in stages such as statistical analysis of the recall-threshold curve for each category, optimization of the multi-category joint threshold and construction of the interpolation mapping table, querying the threshold based on the target recall and making classification decisions. This is done on the basis of the score. By performing category-specific temperature correction at the score level, the comparability and stability of different categories at the decision threshold can be significantly improved, making it possible to achieve "custom recall by category" under a unified framework.

[0095] In step 303, based on the pre-configured target recall rates for each category and combined with a pre-built mapping table of recall rates and thresholds, a standard threshold corresponding to the target recall rate for each category is determined.

[0096] In the embodiments of this disclosure, step 303 can be implemented in any of the ways described in the various embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0097] In step 304, the class with the highest calibration score among the multiple classes is selected as the candidate predicted class for the input sample.

[0098] In the embodiments of this disclosure, step 304 can be implemented in any of the ways described in the various embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0099] In step 305, a standard threshold for the candidate prediction category is determined from the standard thresholds corresponding to the target recall rates of each category.

[0100] In the embodiments of this disclosure, step 305 can be implemented in any of the ways described in the various embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0101] In step 306, based on the calibration score of the candidate predicted category and the standard threshold of the candidate predicted category, it is determined whether to recall the input sample.

[0102] In the embodiments of this disclosure, step 306 can be implemented in any of the ways described in the various embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0103] In the above embodiments, by setting a separate temperature parameter for each category, the scores of the corresponding categories are calibrated using the temperature parameters of each category, so that the original logits are mapped to a probability distribution that better matches the true accuracy. This makes the recall and precision performance of different categories more stable under the same score threshold, reducing the system bias caused by inherent imbalances or differences in difficulty between categories.

[0104] It is worth noting that the category-level recall control method provided in this disclosure is applicable to all multi-category classification scenarios (such as multi-category text classification scenarios) where "high-risk categories must have high recall and different categories have inconsistent recall requirements," and can be embedded into existing systems as an independent recall control module.

[0105] For example, in a content security and compliance review platform, the method disclosed herein can be used as the backend control layer of a text review model: in the offline stage, the review model is fine-tuned using LoRA and a "recall-threshold" mapping table is constructed for each category; after going online, the system first outputs the calibration score for each category for each text, and then automatically looks up the corresponding threshold and performs classification decisions based on the target recall rate set by the operators for the high-risk (or high-risk) label. For low-confidence samples, manual review is triggered to achieve fine control of "almost no high-risk categories are missed and low-risk categories are not excessively disturbed".

[0106] For example, in financial risk control and anti-fraud scenarios, the methods disclosed herein can be deployed in modules such as transaction risk control, customer service quality inspection, and complaint text analysis. Higher recall targets are set for high-risk tags, while maintaining a balanced strategy for ordinary business categories, thereby achieving differentiated recall control based on the differences in tag importance under a unified model.

[0107] For example, in the scenario of public opinion monitoring and intelligent customer service, the method disclosed herein can support the dynamic adjustment of the recall rate of each category of targets (such as brand crisis recall) according to the event stage or operation strategy. It takes effect immediately through online configuration interface and threshold mapping table, so that the multi-category classification system has the category-level recall control capability of "configurable, measurable and closed-loop".

[0108] In summary, the method disclosed herein can provide differentiated and configurable category-based recall control capabilities across existing product lines such as content security, financial risk control, public opinion monitoring, and intelligent customer service, significantly enhancing product competitiveness. Compared to the industry-standard "single global threshold + simple temperature scaling" approach, this disclosure allows for setting target recall rates separately for each risk category and achieving stable and controllable target attainment through calibration, reinforcement learning, and mapping tables.

[0109] Figure 4 This is a block diagram of a category-level recall control device provided in an embodiment of this disclosure. For example... Figure 4 As shown, the category-level recall control device may include: a classification module 401, a first determination module 402, a second determination module 403, a third determination module 404, and a recall judgment module 405.

[0110] The classification module 401 is used to classify the input sample based on a preset multi-class classification model to obtain the calibration score of each class on the input sample.

[0111] The first determining module 402 is used to determine the standard threshold corresponding to the target recall rate of each category based on the pre-configured target recall rate of each category and in combination with the pre-built mapping table of recall rate and threshold.

[0112] The second determination module 403 is used to select the class with the highest calibration classification among multiple classes as the candidate predicted class of the input sample.

[0113] The third determining module 404 is used to determine the standard threshold of the candidate prediction category from the standard threshold corresponding to the target recall rate of each category based on the candidate prediction category.

[0114] The recall decision module 405 is used to determine whether to recall input samples based on the calibration score of the candidate predicted category and the standard threshold of the candidate predicted category.

[0115] In some embodiments, the above-mentioned multi-class classification model is pre-trained in the following manner: a trainable low-rank factorization matrix is ​​added to the key linear layer in the pre-trained model; a unified scoring space based on the conditional probability of the label sequence is constructed, and the pre-trained model is trained by combining cross-entropy loss, supervised contrastive loss and boundary loss to adjust the parameters of the trainable low-rank factorization matrix, and the trained pre-trained model is used as a multi-class classification model.

[0116] In some embodiments, the above-described construction of a unified scoring space based on the conditional probability of the label sequence, and the training of the pre-trained model using a combination of cross-entropy loss, supervised contrastive loss, and boundary loss, includes: inputting training samples into the pre-trained model to obtain the scores of each category output from the last layer of the pre-trained model; constructing the category scores based on the log-likelihood of the label sequence; determining a multi-class cross-entropy loss based on the scores corresponding to the true category labels of the training samples and the scores of each category; determining a supervised contrastive loss based on the similarity of category scores between samples of the same class and samples of different classes in the training samples; determining a boundary loss using a large-margin constraint between categories based on the scores corresponding to the true category labels of the training samples and the scores of each category; determining a joint loss based on the multi-class cross-entropy loss, supervised contrastive loss, and boundary loss; and training the pre-trained model based on the joint loss.

[0117] In some embodiments, the classification module 401 is used to: classify the input sample based on a preset multi-class classification model to obtain the scores of each category output by the last layer in the multi-class classification model; and calibrate the scores of each category based on the temperature parameters of each category to obtain the calibration scores of each category on the input sample.

[0118] In some embodiments, the temperature parameters of each category are pre-learned in the following manner: the validation samples are classified based on a multi-class classification model to obtain the scores of each category output by the last layer in the multi-class classification model; the scores of each category are calibrated using a set of learnable temperature parameters to obtain calibration scores for each category, wherein the set of temperature parameters includes the temperature parameters of each category; and each temperature parameter in the set of temperature parameters is jointly optimized by minimizing the negative log-likelihood loss based on the calibration scores.

[0119] In some embodiments, the above-mentioned recall-threshold mapping table is pre-constructed in the following manner: multi-class threshold parameter tuning is modeled as a Markov decision process (MDP), and the optimal threshold adjustment strategy parameters of the multi-class threshold under the category-level recall objective are learned through trial and error using a reward function; based on the category-level recall objective and the optimal threshold adjustment strategy parameters of the multi-class threshold under the category-level recall objective, the mapping data of recall and threshold is determined, and the mapping table of recall and threshold is constructed according to the mapping data.

[0120] In some embodiments, the above-mentioned recall-threshold mapping table is expanded in the following way: for each category, a set of calibration scores corresponding to each category is collected on the validation set using a multi-class classification model; within the score range of each category, a set of discrete threshold points are selected, and for each threshold point, the corresponding recall is calculated on the validation set; each threshold point and the corresponding recall are expanded to the recall-threshold mapping table.

[0121] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0122] Figure 5 This is a block diagram of the intelligent agent provided in the embodiments of this disclosure. For example... Figure 5 As shown, the intelligent agent may include an input module 501, a processing module 502, and an output module 503. The input module 501 receives input information; the processing module 502 determines a target task based on the input information received by the input module, determines a large model based on the target task, and executes the category-level recall control method in the above method embodiment by calling the large model to obtain output information; the output module 503 outputs the output information obtained by the processing module. For example, the input information may be the text to be processed, the target task may be a multi-category text classification task, and the output information may be the predicted classification result of the text to be processed.

[0123] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0124] like Figure 6 The diagram shown is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0125] like Figure 6 As shown, the electronic device includes one or more processors 601, a memory 602, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take the 601 processor as an example.

[0126] The memory 602 is the non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the category-level recall control method provided in this disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the category-level recall control method provided in this disclosure.

[0127] Memory 602, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the category-level recall control method in the embodiments of this disclosure (e.g., appendix). Figure 4 The classification module 401, the first determination module 402, the second determination module 403, the third determination module 404, and the recall judgment module 405 are shown. The processor 601 executes various functional applications and data processing of the server by running non-transient software programs, instructions, and modules stored in the memory 602, thereby implementing the category-level recall control method in the above method embodiments.

[0128] Memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, memory 602 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 602 may optionally include memory remotely located relative to processor 601, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0129] The electronic device may also include an input device 603 and an output device 604. The processor 601, memory 602, input device 603, and output device 604 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0130] Input device 603 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 604 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0131] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0135] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A category-level recall control method, comprising: The input samples are classified based on a pre-defined multi-class classification model to obtain calibration scores for each class on the input samples. Based on the pre-configured target recall rates for each category, and combined with a pre-built mapping table between recall rate and threshold, a standard threshold corresponding to the target recall rate for each category is determined. The category with the highest calibration classification among the multiple categories is taken as the candidate predicted category of the input sample; Based on the candidate prediction category, the standard threshold of the candidate prediction category is determined from the standard threshold corresponding to the target recall rate of each category; Based on the calibration score of the candidate prediction category and the standard threshold of the candidate prediction category, it is determined whether to recall the input sample.

2. The method according to claim 1, wherein, The multi-class classification model is pre-trained in the following manner: Add trainable low-rank decomposition matrices to the key linear layers in the pre-trained model; A unified scoring space based on the conditional probability of the label sequence is constructed, and the pre-trained model is trained by combining cross-entropy loss, supervised contrast loss and boundary loss to adjust the parameters of the trainable low-rank decomposition matrix. The trained pre-trained model is then used as the multi-class classification model.

3. The method according to claim 2, wherein, The construction of a unified scoring space based on the conditional probability of the label sequence, and the training of the pre-trained model using a combination of cross-entropy loss, supervised contrastive loss, and boundary loss, includes: The training samples are input into the pre-trained model to obtain the scores of each category output by the last layer in the pre-trained model; the scores of each category are constructed based on the log-likelihood of the label sequence. Based on the scores corresponding to the true class labels of the training samples and the scores of each class, determine the multi-class cross-entropy loss based on the class scores; Based on the class score similarity of samples of the same class and the class score similarity of samples of different classes in the training samples, the supervised contrast loss is determined; Based on the scores corresponding to the true category labels of the training samples and the scores of each category, the boundary loss is determined using a large inter-category margin constraint. The joint loss is determined based on the multi-class cross-entropy loss, the supervised comparison loss, and the boundary loss; The pre-trained model is trained based on the joint loss.

4. The method according to claim 1, wherein, The input samples are classified based on a preset multi-class classification model to obtain calibration scores for each class on the input samples, including: The input samples are classified based on a preset multi-class classification model to obtain the scores of each category output by the last layer of the multi-class classification model. Based on the temperature parameters of each category, the scores of each category are calibrated to obtain the calibration scores of each category on the input sample.

5. The method according to claim 4, wherein, The temperature parameters for each category are pre-learned in the following manner: The validation samples are classified based on the multi-class classification model to obtain the scores of each category output by the last layer of the multi-class classification model; The scores of each category are calibrated using a set of learnable temperature parameters to obtain calibration scores for each category, wherein the set of temperature parameters includes the temperature parameters of each category. Each temperature parameter in the set of temperature parameters is jointly optimized by minimizing the negative log-likelihood loss based on the calibration score.

6. The method according to claim 1, wherein, The mapping table between recall and threshold is pre-constructed in the following manner: Multi-class threshold tuning is modeled as a Markov decision process (MDP), and the optimal threshold tuning strategy parameters for multi-class thresholds under the category-level recall objective are learned through trial and error using the reward function. Based on the category-level recall target and the optimal threshold adjustment strategy parameters of the multi-category threshold under the category-level recall target, the mapping data between recall rate and threshold is determined, and the mapping table between recall rate and threshold is constructed according to the mapping data.

7. The method according to claim 6, wherein, The mapping table between recall and threshold is expanded in the following ways: For each category, a set of calibration scores corresponding to each category is collected on the validation set using a multi-class classification model; Within the score range of each category, a set of discrete threshold points are selected, and for each threshold point, the corresponding recall rate is calculated on the validation set. Each threshold point and its corresponding recall rate are expanded into a mapping table between the recall rate and the threshold.

8. A category-level recall control device, comprising: The classification module is used to classify the input sample based on a preset multi-class classification model and obtain the calibration score of each class on the input sample; The first determining module is used to determine the standard threshold corresponding to the target recall rate of each category based on the pre-configured target recall rate of each category and in combination with the pre-built mapping table of recall rate and threshold. The second determining module is used to select the category with the highest calibration classification among the multiple categories as the candidate predicted category of the input sample; The third determining module is used to determine the standard threshold of the candidate prediction category from the standard thresholds corresponding to the target recall rates of each category based on the candidate prediction category; The recall determination module is used to determine whether to recall the input sample based on the calibration score of the candidate prediction category and the standard threshold of the candidate prediction category.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. An intelligent agent, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and execute the method of any one of claims 1-7 by calling the large model to obtain output information; An output module is used to output the output information obtained by the processing module.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

12. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images