Sample classification method and device based on multiple models, storage medium and electronic equipment

Abstract

The invention discloses a sample classification method and device based on multiple models, a storage medium and electronic equipment, and relates to the field of artificial intelligence, and the method comprises the steps: determining a prediction probability that a target sample belongs to each preset classification according to the sample features of the target sample through each model; according to the weight of each model, a decision threshold and the prediction probability of each model, generating a fusion prediction probability of the N models belonging to each preset classification for the target sample; according to the decision threshold and the prediction probability of each model, determining the confidence of the N models when judging that the target sample belongs to each preset classification; and according to the fusion prediction probability and the confidence coefficient, in combination with a fusion classification threshold value and a confidence coefficient threshold value, determining a target category to which the target sample belongs. The technical problem that the accuracy of a final decision result is low due to the fact that an existing multi-model integration mode takes overall prediction precision as a core target and ignores prediction uncertainty and consistency of different models is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a multi-model-based sample classification method, apparatus, storage medium, and electronic device. Background Technology

[0002] In existing technologies, to improve the accuracy and reliability of prediction results, multi-model ensemble methods are commonly used to synthesize the prediction results of different models. These methods include bagging, boosting, stacking, or simple weighted averaging / voting strategies to fuse the predictions from multiple models. These existing multi-model ensemble methods primarily aim to improve overall prediction accuracy by statistically aggregating the numerical outputs of the models to form the final decision. However, this ensemble approach, guided by a single or comprehensive accuracy metric, often prioritizes overall prediction accuracy and tends to overlook the inherent uncertainties and consistency differences between different models during the prediction process, leading to lower accuracy in the final decision.

[0003] For example, in the field of medical technology, existing methods generally ignore the uncertainties inherent in the predictions of multiple models involved in the integration when facing the same test sample, as well as the consistency information of the prediction conclusions among the models. Inconsistency in model predictions often indicates that the test sample (e.g., a medical image sample or medical data sample) is at a classification boundary or in a region of ambiguous features, casting doubt on the reliability of its prediction results. Ignoring the capture and utilization of this information means that the final fusion decision may be based on a probability set with low internal consensus and poor stability. Therefore, in practical applications, especially in borderline decision cases, it is difficult to guarantee the accuracy and credibility of the decision results, failing to meet the core requirement of refined risk control.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a multi-model-based sample classification method, apparatus, storage medium, and electronic device to at least address the technical problem that existing multi-model integration methods, which take overall prediction accuracy as the core objective, ignore the prediction uncertainties and consistency of different models, resulting in low accuracy of the final decision results.

[0006] According to one aspect of the embodiments of this application, a multi-model-based sample classification method is provided, comprising: inputting a target sample into N models; determining the predicted probability of the target sample belonging to each preset category by each model based on the sample features of the target sample, wherein N is an integer greater than 1; generating a fusion prediction probability of the target sample belonging to each preset category by N models based on the weights, decision thresholds, and the predicted probability of the target sample belonging to each preset category by each model, wherein the weights of different models are the same or different, and the decision thresholds of different models are the same or different; determining the confidence level of the N models in judging that the target sample belongs to each preset category based on the decision thresholds of each model and the predicted probability of the target sample belonging to each preset category by each model; and determining the target category to which the target sample belongs based on the fusion prediction probability and confidence level, combined with the fusion classification threshold and confidence threshold set for the N models.

[0007] Optionally, the decision threshold of each model is determined through the following steps: obtaining the predicted probability of each model when the reference sample belongs to each preset category and the true category of the reference sample; performing multiple search operations based on the predicted probability of each model when the reference sample belongs to each preset category and the true category of the reference sample to obtain the decision threshold of the model, wherein each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the decision threshold of the model.

[0008] Optionally, each search operation includes: uniformly sampling within the numerical interval to obtain M candidate thresholds, where M is an integer greater than 1; determining the misclassification rate of each model for each preset category at each candidate threshold based on the predicted probability of each model for a reference sample belonging to each preset category and the actual category to which the reference sample belongs; determining the reference threshold corresponding to this search operation from the M candidate thresholds based on the misclassification rate of each model for each preset category at each candidate threshold; and determining the numerical interval to be used in the next search operation based on the reference threshold, wherein the numerical interval to be used in the next search operation includes the reference threshold determined in this search operation, and the numerical interval to be used in the next search operation is a sub-interval of the numerical interval to be used in this search operation.

[0009] Optionally, based on the misclassification rate of each model for each preset category at each candidate threshold, a reference threshold corresponding to this search operation is determined from M candidate thresholds, including: obtaining the misclassification tolerance corresponding to each preset category; and determining the reference threshold corresponding to this search operation from M candidate thresholds based on the misclassification rate of each model for each preset category at each candidate threshold and the misclassification tolerance corresponding to each preset category.

[0010] Optionally, based on the misclassification rate of each model for each preset category at each candidate threshold and the misclassification tolerance corresponding to each preset category, a reference threshold corresponding to this search operation is determined from M candidate thresholds. This includes: when the misclassification tolerance of the i-th preset category is less than the misclassification tolerance of the j-th preset category, a candidate threshold satisfying the constraint condition is determined from M candidate thresholds as the reference threshold. The constraint condition includes: the misclassification rate of the j-th preset category is the highest when the misclassification rate of the i-th preset category is less than the preset threshold. Here, the i-th preset category and the j-th preset category are two different preset categories.

[0011] Optionally, the weights of each model are determined through the following steps: determining the target preset category from multiple preset categories based on the misclassification tolerance of each preset category; calculating the ratio of the misclassification rate of each model for the target preset category to the model temperature parameter, and then performing an exponential function operation on the ratio to obtain the initial weights of the model, wherein the model temperature parameter is used to regulate the smoothness of the model weight distribution; and normalizing the initial weights of each model to obtain the weights of each model.

[0012] Optionally, based on the weights, decision thresholds, and prediction probabilities of each model for the target sample belonging to each preset category, N models are generated as a fusion prediction probability for the target sample belonging to each preset category. This includes: calculating the difference between the prediction probability of each model for the target sample belonging to the x-th preset category and the decision threshold of that model; then calculating the product of this difference and the weights of that model to obtain the prediction distance of that model for the target sample belonging to the x-th preset category, where x is an integer greater than or equal to 1; averaging the prediction distances of the N models for the target sample belonging to the x-th preset category to obtain the average prediction distance; and using a non-linear activation function to convert the average prediction distance into a fusion prediction probability of the N models for the target sample belonging to the x-th preset category.

[0013] Optionally, based on the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, the confidence level of N models in judging the target sample belonging to each preset category is determined, including: determining the category decision result of each model for the target sample based on the relationship between the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category; determining the consistency score based on the number of similarities and differences in the category decision results of the N models for the target sample; calculating the variance based on the predicted probability of each model for the target sample belonging to each preset category and the decision threshold of that model; and determining the confidence level of N models in judging the target sample belonging to each preset category based on the variance calculation result and the consistency score.

[0014] Optionally, based on the fusion prediction probability and confidence level, and combined with the fusion classification threshold and confidence level threshold set for the N models, the target category to which the target sample belongs is determined, including: when the fusion prediction probability of the N models for the target sample belonging to the y-th preset category is detected to be greater than or equal to the fusion classification threshold, and the confidence level of the N models when judging that the target sample belongs to each preset category is greater than or equal to the confidence level threshold, the y-th preset category is determined as the target category of the target sample, where y is an integer greater than or equal to 1.

[0015] Optionally, the fusion classification threshold is determined through the following steps: obtaining the fusion prediction probability of N models for each preset classification and the true category of the reference sample; performing multiple search operations based on the fusion prediction probability of N models for each preset classification and the true category of the reference sample to obtain the fusion classification threshold, wherein each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the fusion classification threshold.

[0016] According to another aspect of the embodiments of this application, a multi-model-based sample classification apparatus is also provided, comprising: a first processing unit, configured to input a target sample into N models, and determine the predicted probability of the target sample belonging to each preset category based on the sample features of the target sample by each model, wherein N is an integer greater than 1; a second processing unit, configured to generate a fusion prediction probability of the target sample belonging to each preset category by N models based on the weights, decision thresholds, and the predicted probability of the target sample belonging to each preset category by each model, wherein the weights of different models are the same or different, and the decision thresholds of different models are the same or different; a third processing unit, configured to determine the confidence level of the N models when judging that the target sample belongs to each preset category based on the decision thresholds of each model and the predicted probability of the target sample belonging to each preset category by each model; and a fourth processing unit, configured to determine the target category to which the target sample belongs based on the fusion prediction probability and confidence level, combined with the fusion classification threshold and confidence threshold set for the N models.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device on which the computer-readable storage medium is located executes the above-described multi-model-based sample classification method.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described multi-model-based sample classification method.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the above-described multi-model-based sample classification method.

[0020] In this embodiment, the intelligent classification system can input a target sample into N models. Each model determines the predicted probability of the target sample belonging to each preset category based on the sample characteristics of the target sample, where N is an integer greater than 1. Based on the weights, decision thresholds, and the predicted probability of each model for the target sample belonging to each preset category, a fusion prediction probability of the target sample belonging to each preset category is generated for each of the N models. The weights and decision thresholds of different models may be the same or different. Then, based on the decision thresholds and the predicted probability of each model for the target sample belonging to each preset category, the confidence level of the N models in determining that the target sample belongs to each preset category is determined. Finally, based on the fusion prediction probability and confidence level, combined with the fusion classification threshold and confidence threshold set for the N models, the target category to which the target sample belongs is determined.

[0021] As described above, intelligent classification systems can generate fused prediction probabilities for N models to represent a target sample in each preset category by introducing independent weights and decision thresholds for each model and combining the prediction probability of each model for each preset category. The weight differences between models allow models with better performance or more relevant to the current task to contribute more during the fusion process, thus enabling discriminative integration of prediction information from multiple models in the probability fusion stage. Furthermore, setting a separate decision threshold for each model can calibrate the probability distribution shifts caused by training differences between models, ensuring that each model's prediction is in a better state before fusion. This helps reduce noise introduced by uneven model capabilities and inconsistent standards during the probability fusion stage, making the generated fused prediction probabilities more representative of the collective consensus of multiple models and improving the accuracy of the final decision. Moreover, intelligent classification systems can determine confidence levels based on the decision thresholds and prediction probabilities of each model and employ a dual-threshold decision rule combining fused prediction probabilities and confidence levels to directly address the uncertainty and consistency issues in model predictions. Determining the confidence level captures the degree of consistency between model predictions and the stability of each model's predictions. Combining confidence scores with fusion prediction probabilities allows the final classification decision to consider not only the predicted probability values ​​but also the consensus and stability of the predictions. This dual verification helps identify and filter high-uncertainty samples that, while potentially having high fusion probability values, exhibit significant disagreement or unstable predictions among the multiple models. For such high-uncertainty samples, the intelligent classification system can set a confidence threshold, choosing not to make a definitive classification or triggering a more cautious processing procedure. This reduces the risk of erroneous decisions due to disagreements among models, improving the accuracy of the decision results. This addresses the technical problem of existing multi-model ensemble methods that prioritize overall prediction accuracy while neglecting the prediction uncertainties and consistency of different models, resulting in low accuracy of the final decision. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a schematic diagram of an optional multi-model-based sample classification method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of an optional multi-model-based sample classification device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to an embodiment of this application, a method embodiment of a multi-model-based sample classification method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0029] According to the embodiments of this application, an intelligent classification system can be used as the execution subject of the multi-model-based sample classification method of this application embodiment. The system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the method execution subject.

[0030] Figure 1 This is a schematic diagram of a multi-model-based sample classification method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0031] Step S101: Input the target sample into N models, and determine the predicted probability of the target sample belonging to each preset category based on the sample characteristics of the target sample by each model, where N is an integer greater than 1.

[0032] Optionally, sample features can refer to numerical information that characterizes the attributes of the target sample. In an example of a clinical auxiliary diagnosis scenario, the target sample can refer to a medical image sample or a medical data sample, such as CT slices, pathological images, physiological test data, etc., and the sample features can refer to texture features, shape features, depth features extracted from the medical image sample, deep learning model-extracted features, or data features extracted from the medical data sample, etc. The N models can be models with the same architecture trained on different datasets, or models with different architectures, which is beneficial for providing diverse prediction perspectives. In addition, the target sample can also be other types of samples besides medical image samples, including but not limited to speech samples, data samples, text samples, etc., such as physiological test data, network traffic data, various analytical data, log data, etc.

[0033] Optionally, the model can refer to a machine learning or statistical model that has been trained and is capable of making probabilistic predictions about input features such as target samples, such as deep neural networks, support vector machines, and random forests. The predicted probability can characterize the confidence level of the sample belonging to a specific category, calculated by the model based on the input features, and can be a value between 0 and 1.

[0034] Optionally, each preset classification can be positive, negative, or other categories, such as multiple labels corresponding to medical image samples, including labels for distinguishing image types or labels for distinguishing different risk probabilities.

[0035] By using multiple different prediction models to independently analyze the same target sample, we can obtain an assessment of the probability that each model assigns the target sample to various preset categories. Using multiple models instead of a single model helps to capture sample features from different perspectives or based on different subsets of data, and helps to compensate for the biases or blind spots that may exist in a single model.

[0036] Step S102: Based on the weights, decision thresholds, and prediction probabilities of each model for the target sample belonging to each preset category, generate N fusion prediction probabilities for the target sample belonging to each preset category. The weights of different models may be the same or different, and the decision thresholds of different models may be the same or different.

[0037] Optionally, the weight of each model can be a non-negative numerical value, used to quantify the relative importance and credibility contribution of the model in the overall fusion decision. Models with higher weights have a greater impact on the final fusion result. The numerical value of each model's weight can be determined through offline evaluation, for example, by calculating and normalizing the model's performance metrics for target classification on independent validation sets. Alternatively, the weight allocation can be determined based on the model's historical performance on the validation set.

[0038] Optionally, the decision threshold for each model can refer to the critical value at which the model converts continuous predicted probabilities into discrete class decisions. Since different models may have different learning abilities, training data distributions, or optimization objectives, a decision threshold can be set individually for each model, allowing for more precise calibration of the model's decision boundaries on specific classification tasks.

[0039] By generating N models to predict the probability of a target sample belonging to each preset category, it is beneficial to achieve differentiated weighting and personalized calibration of the prediction results of multiple models. This allows models with better performance or better matching to the current classification task to play a greater role in the fusion process. It also helps to reduce noise introduced by inconsistent decision criteria among models, thereby improving the accuracy of the fusion results.

[0040] Step S103: Based on the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, determine the confidence level of N models when judging that the target sample belongs to each preset category.

[0041] Optionally, the confidence score can be a scalar value used to quantify the reliability of multiple models' classification judgments of the current target sample. A higher confidence score indicates high consensus among models and stable predictions, while a lower confidence score indicates greater uncertainty.

[0042] Optionally, the consensus and stability of the classification results of the target sample across multiple models can be evaluated to determine the confidence level. Confidence level reflects the consistency and certainty of multiple models' classification decisions for the target sample. Confidence level can be calculated based on two types of information: one is the consistency of decision results among models, such as statistically analyzing the proportion of models that give the same class judgment; the other is the dispersion of the deviation between the model's predicted probability and its respective decision threshold, such as calculating the variance of these deviations. A higher confidence level is achieved when most models give the same judgment and the predicted probabilities of each model are close to their respective decision thresholds; conversely, a lower confidence level is achieved when there is significant divergence among models or when the predicted probabilities deviate from the decision thresholds. Determining the confidence level can identify fuzzy samples that, although the fused predicted probability value is high, exhibit significant divergence among models or unstable predictions.

[0043] Step S104: Based on the fusion prediction probability and confidence level, and combined with the fusion classification threshold and confidence level threshold set for the N models, determine the target category to which the target sample belongs.

[0044] Optionally, the fusion classification threshold and confidence threshold can be pre-defined scalar parameters based on the analysis of historical validation set data. The fusion classification threshold is used to determine whether the fusion prediction probability is sufficient to support classifying the target sample as a certain target category; the confidence threshold is used to determine whether the reliability of the current classification judgment reaches an acceptable level. For example, the final decision rule can be set as follows: the target sample is classified as the target category only when the fusion prediction probability is greater than or equal to the fusion classification threshold and the confidence level is greater than or equal to the confidence threshold; otherwise, the target sample is classified as another category or further processing is triggered, such as manual review.

[0045] By determining the target category of a target sample based on the fused prediction probability and confidence level, combined with the fusion classification threshold and confidence level threshold set for N models, the probability information of the fused prediction can be combined with the consistency and stability information among the models. This facilitates a dual-verification decision-making mechanism, considering not only the predicted probability value but also the predictable confidence level. This filters out unreliable predictions that, while having high probability values, have low consensus among the models, helping to improve the interpretability and reliability of the decision results while ensuring accuracy. Compared to methods that directly average or vote on model outputs, the embodiments of this application can more effectively integrate the advantages of multiple models and identify and process samples with uncertain predictions through confidence levels, thereby helping to improve the overall accuracy and decision credibility of the classification system in complex or high-risk decision-making scenarios.

[0046] Optionally, due to factors such as data distribution and model structure, the model may find it difficult to simultaneously consider the trade-offs between different indicators. For ease of explanation, the following uses the False Negative Rate (FNR) and True Negative Rate (TNR) as examples to illustrate the trade-offs between indicators. It should be noted that the method in this application embodiment is not limited to specific indicators. By fusing multiple models, complementarity between models can be formed, improving the overall prediction accuracy, especially when considering the trade-offs between different indicators. This application embodiment mainly addresses how to fuse multiple models in clinical scenarios while focusing on specific clinical indicators.

[0047] In an optional embodiment, the decision threshold of each model is determined by the following steps: the intelligent classification system can obtain the predicted probability of each model when the reference sample belongs to each preset category and the actual category to which the reference sample belongs; then, based on the predicted probability of each model when the reference sample belongs to each preset category and the actual category to which the reference sample belongs, multiple search operations are performed to obtain the decision threshold of the model, wherein each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the decision threshold of the model.

[0048] Optionally, the reference sample can refer to a dataset of known true class labels used for model evaluation, parameter calibration, or optimization. Its data form can be structured feature vectors, image data, text data, etc. In the example of a clinical auxiliary diagnostic scenario, the reference sample can be historically accumulated, pathologically confirmed medical imaging data and its corresponding diagnostic result labels.

[0049] Optionally, a search operation can refer to the process of selecting candidate values ​​within a numerical range according to certain rules, and choosing the optimal value from them based on preset evaluation criteria such as misjudgment rate constraints. Each search operation generates a reference threshold. Multiple search operations are dependent on each other; that is, the range of a later search can be determined based on the results of the previous search, thus achieving a gradual focus of the search scope. For example, multiple search operations can refer to both simple search and depth-first search, but are essentially a loop of iterative search. Multiple search operations can be, but are not limited to, two times.

[0050] Intelligent classification systems, by using reference samples with known true categories, can objectively calculate the model's classification performance metrics at different candidate thresholds. Multiple search operations facilitate coarse-to-fine threshold localization. For example, the first search operation can traverse a large, pre-defined range with a sparse step size, quickly locating a roughly suitable threshold region—the reference threshold for the first search. Subsequent search operations then search around the previously found optimal threshold with smaller numerical intervals and finer step sizes, thus precisely determining the optimal decision threshold. Compared to high-precision traversal across the entire range, hierarchical search significantly reduces computational overhead, improves the efficiency of the threshold determination process, and can find highly suitable, personalized decision thresholds for each model. It fully considers the differences in prediction probability distributions caused by variations in training data, algorithm structure, or optimization objectives among different models, allowing each model to participate in subsequent fusion at its optimal decision point.

[0051] In one optional embodiment, each search operation includes: the intelligent classification system can perform uniform sampling within a numerical range to obtain M candidate thresholds, where M is an integer greater than 1; based on the predicted probability of each model for a reference sample belonging to each preset category and the actual category of the reference sample, determine the misclassification rate of the model for each preset category under each candidate threshold; based on the misclassification rate of each model for each preset category under each candidate threshold, determine the reference threshold corresponding to this search operation from the M candidate thresholds; and determine the numerical range to be used in the next search operation based on the reference threshold, wherein the numerical range to be used in the next search operation includes the reference threshold determined in this search operation, and the numerical range to be used in the next search operation is a sub-range of the numerical range to be used in this search operation.

[0052] Optionally, the false positive rate may refer to the false negative rate or the false positive rate, or it may refer to other categories besides the false negative rate and the false positive rate.

[0053] Optionally, the numerical range used in the next search operation includes the reference threshold determined in this search operation, and the numerical range used in the next search operation is a sub-range of the numerical range used in this search operation. This can refer to a re-search and fine-tuning at a higher resolution (e.g., step size 0.001) near the reference threshold determined in the first search operation (e.g., within the range of ±0.05 of the reference threshold) to determine the final optimal decision threshold for each model.

[0054] Optionally, generating multiple candidate thresholds helps create a discretized set of candidate solutions covering the current search space. Uniform sampling allows for even exploration of intervals, helping to avoid missing potential excellent threshold points. Setting the value of M controls the granularity of the search; a larger value of M results in a finer search, but also increases the computational cost.

[0055] In one optional embodiment, a reference threshold corresponding to the current search operation is determined from M candidate thresholds based on the misclassification rate of each model for each preset category at each candidate threshold. This includes: obtaining the misclassification tolerance corresponding to each preset category; and determining the reference threshold corresponding to the current search operation from M candidate thresholds based on the misclassification rate of each model for each preset category at each candidate threshold and the misclassification tolerance corresponding to each preset category.

[0056] Optionally, the misclassification tolerance can represent the acceptable level or maximum tolerance for misclassification of different preset categories in a specific application scenario. For example, in the field of medical technology, the tolerance for misclassifying an actual diseased sample as healthy (false negative) may be extremely low, while the tolerance for misclassifying a healthy sample as diseased (false positive) can be relatively high. By introducing this misclassification tolerance as a constraint, the threshold search process is no longer simply about finding a point that maximizes comprehensive indicators such as accuracy, but rather transforms into an optimization problem that satisfies prior risk constraints.

[0057] In one optional embodiment, based on the misclassification rate of each model for each preset category at each candidate threshold and the misclassification tolerance corresponding to each preset category, a reference threshold corresponding to this search operation is determined from M candidate thresholds. This includes: if the misclassification tolerance of the i-th preset category is less than the misclassification tolerance of the j-th preset category, a candidate threshold satisfying the constraint condition is determined from the M candidate thresholds as the reference threshold. The constraint condition includes: the misclassification rate of the j-th preset category is the highest under the condition that the misclassification rate of the i-th preset category is less than the preset threshold. The i-th preset category and the j-th preset category are two different preset categories.

[0058] Optionally, the preset threshold can be associated with the misclassification tolerance of the i-th preset classification, which helps to prevent the model from exceeding the limit for classification errors with lower tolerance. Achieving the highest misclassification rate for the j-th preset classification is the optimization objective of maximizing performance within the safety boundary.

[0059] For example, in the process of searching for the optimal classification threshold for a single model, the first step is to define the objective: to find the decision threshold for each model. This ensures that the following condition is met: the misclassification rate of each model for the first preset classification does not exceed the misclassification tolerance corresponding to the first preset classification, as can be found in formula (1):

[0060] Formula (1);

[0061] in, Used to express the false negative rate, which is the proportion of positive samples (FN) that are incorrectly identified as negative to the total number of actual positive samples (TP+FN). It can also represent the misclassification rate of each model for the first preset classification, which can correspond to the misclassification rate of the i-th preset classification; FN can correspond to the number of samples that the model incorrectly predicts as not belonging to the first preset classification when the reference sample that actually belongs to the first preset classification is calculated during the process of calculating the misclassification rate. This corresponds to the number of reference samples that actually belong to the first preset category correctly predicted by the model as samples belonging to the first preset category during the calculation of the misclassification rate. Used to represent the tolerance for misclassification of the i-th preset category.

[0062] At the same time, the misclassification rate of each model for the second preset classification should be as high as possible. The misclassification tolerance of the first preset classification is less than that of the second preset classification. See formula (2) for reference:

[0063] Formula (2);

[0064] in, Used to express the true negative rate, which is the proportion of the number of negative samples (TN) that are correctly identified as negative to the total number of actual negative samples (TN+FP). This can also represent the misclassification rate of each model for the second preset classification, which corresponds to the misclassification rate for the j-th preset classification. Wherein, and The sum can equal 1, which can be achieved through optimization. To indirectly minimize . This corresponds to the number of reference samples that actually belong to the second preset category being correctly predicted by the model as samples belonging to that second preset category during the TNR calculation process. This corresponds to the number of reference samples that actually belong to the second preset category being incorrectly predicted as the first preset category by the model during the TNR calculation process.

[0065] It's important to note that different models may use different training data or strategies, leading to variations in their predicted probability distributions. For example, model A using a decision threshold of 0.3 might achieve an optimal trade-off between the misclassification rates of the first and second preset classifications, while model B using a threshold of 0.5 might also achieve the same trade-off. Therefore, the same decision threshold cannot be applied to all models; each model needs its own threshold calculated.

[0066] The process of searching for the decision threshold for optimal classification using a single model may include:

[0067] First, perform the first search operation: uniformly sample within the preset numerical range to obtain M candidate thresholds θ, where M is an integer greater than 1, the preset numerical range can be [0,1], and the number of sampling points M can be set to 100.

[0068] Furthermore, for each candidate threshold, the predicted probability of the model can be binarized using the candidate threshold to obtain a preliminary category judgment, as shown in formula (3):

[0069] Formula (3);

[0070] in, Used to represent the class decision result of each model for the target sample; Used to represent the predicted probability of each model for a target sample belonging to each preset category, that is, the probability output by the model; Used to represent candidate thresholds.

[0071] Based on the predicted probability of each model for a reference sample belonging to each preset category and the true category of the reference sample, calculate the misclassification rate of each model for the first preset category and the misclassification rate for the second preset category at the candidate threshold. For example, the false negative rate can be calculated. With true negative rate .

[0072] Then, the candidate threshold with the highest misclassification rate for each model for the second preset category, provided that the misclassification rate of each model for the first preset category does not exceed the misclassification tolerance corresponding to the first preset category, is retained as the reference threshold determined in the first search operation. For example, the candidate threshold with the highest misclassification rate for each model for the second preset category can be retained. under conditions The highest classification threshold.

[0073] Then, a second search operation can be performed: near the reference threshold determined in the first search operation (e.g., the reference threshold). Within the range of ±0.05, a further search and fine-tuning is performed at a higher resolution (e.g., a step size of 0.001) to determine the final optimal decision threshold for each model.

[0074] In one optional embodiment, the weight of each model is determined through the following steps: the intelligent classification system determines the target preset category from multiple preset categories based on the misclassification tolerance of each preset category, then calculates the ratio of the misclassification rate of each model for the target preset category to the model temperature parameter, and then performs an exponential function operation on the ratio to obtain the initial weight of the model, wherein the model temperature parameter is used to regulate the smoothness of the model weight distribution; and the initial weight of each model is normalized to obtain the weight of each model.

[0075] Optionally, the target pre-classification can be selected as the class with a relatively high tolerance for misclassification. Initial weights are calculated based on the misclassification rate of the target pre-classification and the model temperature parameter. The model's misclassification rate for the target pre-classification can be used as the model's performance score. Dividing this performance score by the model temperature parameter and then applying an exponential function can non-linearly amplify or reduce performance differences. The model temperature parameter can play a role in smoothing: a larger model temperature parameter value can weaken the performance score differences between different models in the weight calculation, resulting in a more even weight distribution; a smaller model temperature parameter value can amplify performance differences, giving higher weights to better-performing models. Finally, by normalizing the initial weights of all models, the sum of all weights can be made 1, resulting in weights for each model that can be directly used for weighted fusion. This helps to allow better-performing models to have a greater impact on the final decision during multi-model fusion, thereby improving the overall performance of the fusion system. Furthermore, by adjusting the model temperature parameter, it is beneficial to achieve a flexible transition from near-average weights to highly differentiated weights, adapting to the need for a balance between utilizing model diversity and performance emphasis in different scenarios.

[0076] For example, when fusing multiple models, the prediction performance of different models varies. The contribution weight of each model in the multi-model fusion can be determined by using the misclassification rate of each model when making individual decisions for the second preset classification.

[0077] Each model has varying predictive capabilities, therefore each model's contribution to the overall prediction should not be equal. The contribution of each model to the overall prediction is calculated based on its misclassification rate performance for the second preset classification, under the constraint of the misclassification tolerance for the first preset classification. For example, the misclassification rate performance for the second preset classification can refer to the balance between the misclassification rates for the first and second preset classifications for each model. The better the balance between the misclassification rates for the first and second preset classifications for each model, the greater its weight should be.

[0078] For the model Each model The misclassification rate score for the target predefined classification is denoted as: First, calculate the exponent value after scaling adjustment based on the temperature parameter. The exponent distribution can be found in formula (4):

[0079] Formula (4);

[0080] in, Used to represent the initial weights of the model, The temperature parameter is used to represent the model temperature and to adjust the smoothness of the weight distribution for each model. Next, the initial weights of each model are normalized, as shown in formula (5):

[0081] Formula (5);

[0082] in, Formula (5) represents the normalization of the initial weights to make the sum of all weights equal to 1.

[0083] In one optional embodiment, based on the weights of each model, the decision threshold, and the prediction probability of each model for the target sample belonging to each preset category, a fusion prediction probability of N models for the target sample belonging to each preset category is generated. This includes: the intelligent classification system can calculate the difference between the prediction probability of each model for the target sample belonging to the x-th preset category and the decision threshold of the model, and then calculate the product of the difference and the weight of the model to obtain the prediction distance of the model for the target sample belonging to the x-th preset category, where x is an integer greater than or equal to 1. The average prediction distance of the N models for the target sample belonging to the x-th preset category is calculated to obtain the average prediction distance. Then, a non-linear activation function is used to convert the average prediction distance into a fusion prediction probability of the N models for the target sample belonging to the x-th preset category.

[0084] Optionally, the difference between the predicted probability and the decision threshold can characterize the extent to which the model's prediction supports classifying the target sample into the x-th preset category. A positive value indicates support, a negative value indicates opposition, and the absolute value indicates the strength of support or opposition. Multiplying this difference by a weight representing the model's credibility helps to achieve differentiated weighting of the model's contribution.

[0085] Optionally, the calculation of the average prediction distance can aggregate the weighted opinions of multiple models. Averaging helps smooth out random fluctuations or anomalous outputs of individual models, making the aggregated signal more reflective of the collective tendencies of the models.

[0086] Optionally, a nonlinear activation function can be used to convert the average prediction distance into a fused prediction probability, mapping the average prediction distance over any real number range to a probability range between 0 and 1. This nonlinear mapping enhances the discriminative power of signals near the boundary, making the final fused prediction probability more sensitive to changes in the model's collective consensus.

[0087] Optionally, based on the performance of each model under the constraint of misjudgment tolerance, weights are assigned to different models, and the prediction probabilities of multiple models are weighted and fused based on the weights of each model to calculate the overall prediction probability after fusion.

[0088] For example, the prediction results of multiple models can be combined, utilizing the decision threshold of each model. and the weights of each model This generates a comprehensive fusion prediction probability.

[0089] The process of generating fusion prediction probabilities may include:

[0090] First, for the target sample, according to each model Predicted probability that the target sample belongs to the x-th preset category Calculate each model The predicted distance for the target sample to belong to the x-th preset category Predicted distance The calculation can be referenced from formula (6):

[0091] Formula (6);

[0092] in, Used to represent each model The predicted probability of a target sample belonging to the x-th preset category and the probability of each model Decision threshold The distance between them, by calculating this distance This helps to eliminate differences in the predicted probability distributions between different models. (The distance is then used.) With each model weight Multiplication facilitates weighted processing.

[0093] Then, the average prediction distance of all N models is calculated to obtain the average prediction distance. Average prediction distance The calculation can be referenced from formula (7):

[0094] Formula (7);

[0095] in, Used to indicate the number of models.

[0096] Using non-linear activation functions such as the Sigmoid function to calculate the average prediction distance Convert into N models to fused prediction probabilities for a target sample belonging to the x-th preset category. Fusion prediction probability The calculation can be referenced from formula (8):

[0097] Formula (8);

[0098] in, Used to represent a settable constant, for example, This can be set to 10 to control the steepness of the Sigmoid function curve. The Sigmoid function normalizes the fusion prediction probability to between 0 and 1, ensuring that the range of the fusion prediction probability is consistent with the range of the prediction probability of each model. Therefore, a fusion classification threshold between 0 and 1 can also be used for classification decisions regarding the fusion prediction probability.

[0099] In one optional embodiment, based on the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, the confidence level of N models in judging that the target sample belongs to each preset category is determined. This includes: the intelligent classification system can determine the category decision result of each model for the target sample based on the relationship between the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, and determine a consistency score based on the number of similarities and differences in the category decision results of the N models for the target sample. Then, based on the predicted probability of each model for the target sample belonging to each preset category and the decision threshold of that model, variance is calculated to obtain the variance calculation result. And based on the variance calculation result and the consistency score, the confidence level of N models in judging that the target sample belongs to each preset category is determined.

[0100] Optionally, the consistency score can reflect the concentration of opinions among the model group; a higher score indicates a high degree of consensus among the models. Variance can reflect the dispersion of the predictive strength of each model; a smaller variance indicates that the models not only have consistent classification opinions but also similar predictive confidence; a larger variance indicates that although the classification opinions may be the same, the predictive confidence of each model differs significantly. By simultaneously examining the consistency of the model group's decision-making and the stability of individual predictive strength, the generated confidence index can more comprehensively capture the sources of uncertainty in the prediction process.

[0101] Optionally, based on the calculation of the weighted fusion prediction probability, an algorithm is proposed to calculate the prediction confidence of each sample as a method to further improve the prediction reliability.

[0102] For example, using only fused prediction probabilities may not be robust enough. An algorithm can be used to calculate the consistency between multiple models, which serves as the confidence level to determine the reliability of the prediction results for the target sample. The confidence level determination process may include:

[0103] First, based on the decision threshold of each model... And the predicted probability of each model for a target sample belonging to each preset category. The size relationship between them determines the class decision result of each model for the target sample. Category decision results It is generated by an indicator function, as shown in formula (9):

[0104] Formula (9);

[0105] Formula (9) means that the predicted probability of each model can be converted into a binary prediction result for a preset classification, such as a negative prediction result or a positive prediction result.

[0106] Furthermore, a consistency score is determined based on the number of similarities and differences in the class decision results of the N models for the target sample. Consistency score The proportion of models that predict the target class among N models is calculated using the following formula (10):

[0107] Formula (10);

[0108] in, This represents the category decision result of each model i for the target sample. Indicates the number of models.

[0109] Then, based on the predicted probability of each model for the target sample belonging to each preset category. And the decision threshold for each model. Variance calculation is performed to obtain the variance calculation results. Variance calculation results Calculated using formula (11):

[0110] Formula (11);

[0111] The variance It can reflect the degree of dispersion of the deviation between the predicted probability of each model and its respective decision threshold.

[0112] Then, the variance can be calculated. and consistency score Determine the confidence levels of N models when judging that the target sample belongs to each preset category. The confidence level is calculated using formula (12):

[0113] Formula (12);

[0114] Among them, when the consistency score The variance tends towards 0 or 1 (indicating consensus among N models) and the variance calculation results When the value is small, the calculated confidence level is higher.

[0115] In one optional embodiment, the target category to which the target sample belongs is determined based on the fusion prediction probability and confidence level, combined with the fusion classification threshold and confidence level threshold set for the N models. This includes: when the intelligent classification system detects that the fusion prediction probability of the N models for the target sample belonging to the y-th preset category is greater than or equal to the fusion classification threshold, and the confidence level of the N models when determining that the target sample belongs to each preset category is greater than or equal to the confidence level threshold, the y-th preset category is determined as the target category of the target sample, where y is an integer greater than or equal to 1.

[0116] Optionally, the fusion classification threshold can refer to a judgment boundary set relative to the fusion prediction probability, used to measure whether the support strength of the model group for the target sample belonging to the y-th preset classification meets the minimum requirement for making a positive judgment. If the fusion prediction probability is greater than or equal to the fusion classification threshold, it indicates that the weighted combined opinion of multiple models tends to classify the target sample into the y-th preset classification.

[0117] Optionally, the confidence threshold can refer to an acceptance standard set for the reliability measure of the model group's predictions. A confidence level greater than or equal to the confidence threshold indicates that the reliability of the current classification judgment can reach an acceptable level, the consensus among models is high, and the predictions are stable.

[0118] For example, in the final prediction stage based on dual threshold optimization using a fusion classification threshold and a confidence threshold, pre-determined parameters can be used, such as the decision threshold for each model, the weights of each model, and the fusion classification threshold. and confidence threshold Perform the following steps on the new target sample:

[0119] First, calculate the fusion prediction probability. And the confidence levels of N models when determining whether a target sample belongs to each preset category.

[0120] Then, the final judgment rule can be referred to formula (13):

[0121] Formula (13);

[0122] That is, simultaneously satisfying the fusion prediction probability Exceeding the fusion classification threshold Furthermore, the confidence levels of N models in determining that the target sample belongs to each preset category exceed the confidence threshold. Only then will the target sample be classified as a positive example or the first preset classification.

[0123] In one optional embodiment, the fusion classification threshold is determined through the following steps: the intelligent classification system can obtain the fusion prediction probabilities of N models for each preset category of the reference sample and the true category of the reference sample. Then, based on the fusion prediction probabilities of the N models for each preset category of the reference sample and the true category of the reference sample, multiple search operations are performed to obtain the fusion classification threshold, wherein each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the fusion classification threshold.

[0124] For example, after obtaining the fusion prediction probability After determining the confidence levels of N models in classifying the target sample as belonging to each preset category, it is necessary to further determine the fusion classification threshold for the overall decision. and confidence threshold Fusion classification threshold and confidence threshold The approach to determining the threshold is similar to the search for the decision threshold of each model. The goal is still to ensure that the overall misclassification rate for the first preset classification does not exceed the misclassification tolerance corresponding to the first preset classification, and to make the overall misclassification rate for the second preset classification as low as possible, or to make the accuracy rate for the second preset classification as high as possible.

[0125] At this stage, the threshold is optimized again based on the fusion prediction results, and a confidence threshold is found. This ensures that the confidence level of N models in determining whether a target sample belongs to each preset category is greater than or equal to a confidence threshold. And the fusion prediction probability Greater than or equal to the fusion classification threshold Under the given conditions, the prediction should be as accurate as possible.

[0126] This application's embodiments address the core need in clinical scenarios such as disease diagnosis and treatment plan analysis to "prioritize key clinical indicators (e.g., low missed diagnosis rate) while optimizing other indicators (e.g., reducing misdiagnosis and improving true negative rate)." It enables multi-model fusion with high interpretability and no need for model retraining. The intelligent classification system, through a process of "decision threshold search for each model → weight calculation for each model → fusion prediction probability calculation → confidence verification → final decision based on dual thresholds of classification and confidence," facilitates multi-model collaborative decision-making, thereby achieving a balance between controlling the risk of missed diagnoses and ensuring clinical safety while minimizing unnecessary examinations and promoting efficient use of medical resources.

[0127] See Figure 2 According to another aspect of the embodiments of this application, a multi-model-based sample classification device is also provided, including: a first processing unit 201, a second processing unit 202, a third processing unit 203, and a fourth processing unit 204.

[0128] The system comprises the following components: a first processing unit 201, which inputs the target sample into N models and determines the predicted probability of the target sample belonging to each preset category based on the sample features of the target sample, where N is an integer greater than 1; a second processing unit 202, which generates a fusion prediction probability of the target sample belonging to each preset category based on the weights, decision thresholds, and predicted probabilities of each model for the target sample belonging to each preset category, where the weights of different models may be the same or different, and the decision thresholds of different models may be the same or different; a third processing unit 203, which determines the confidence level of the N models in judging that the target sample belongs to each preset category based on the decision thresholds of each model and the predicted probabilities of each model for the target sample belonging to each preset category; and a fourth processing unit 204, which determines the target category to which the target sample belongs based on the fusion prediction probability and confidence level, combined with the fusion classification threshold and confidence threshold set for the N models.

[0129] Optionally, the third processing unit 203 includes: a first acquisition subunit, used to acquire the predicted probability of each model when the reference sample belongs to each preset category and the true category of the reference sample; and a first processing subunit, used to perform multiple search operations based on the predicted probability of each model when the reference sample belongs to each preset category and the true category of the reference sample to obtain the decision threshold of the model, wherein each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the decision threshold of the model.

[0130] Optionally, the first processing subunit includes: a first processing module, configured to perform uniform sampling within a numerical interval to obtain M candidate thresholds, where M is an integer greater than 1; a second processing module, configured to determine the misclassification rate of each model for each preset category under each candidate threshold based on the predicted probability of each model for a reference sample belonging to each preset category and the actual category to which the reference sample belongs; a third processing module, configured to determine the reference threshold corresponding to the current search operation from the M candidate thresholds based on the misclassification rate of each model for each preset category under each candidate threshold; and a fourth processing module, configured to determine the numerical interval to be used in the next search operation based on the reference threshold, wherein the numerical interval to be used in the next search operation includes the reference threshold determined in the current search operation, and the numerical interval to be used in the next search operation is a sub-interval of the numerical interval to be used in the current search operation.

[0131] Optionally, the third processing module includes: a first acquisition submodule, used to acquire the misjudgment tolerance corresponding to each preset category; and a first processing submodule, used to determine the reference threshold corresponding to this search operation from M candidate thresholds based on the misjudgment rate of each model for each preset category at each candidate threshold and the misjudgment tolerance corresponding to each preset category.

[0132] Optionally, the first processing submodule includes: a first processing component, configured to determine a candidate threshold satisfying a constraint condition as a reference threshold from M candidate thresholds when the misjudgment tolerance of the i-th preset category is less than the misjudgment tolerance of the j-th preset category, wherein the constraint condition includes: satisfying the condition that the misjudgment rate of the j-th preset category is the highest when the misjudgment rate of the i-th preset category is less than the preset threshold, wherein the i-th preset category and the j-th preset category are two different preset categories.

[0133] Optionally, the second processing unit 202 includes: a second processing subunit, used to determine a target preset category from multiple preset categories based on the misclassification tolerance of each preset category; a third processing subunit, used to calculate the ratio of the misclassification rate of each model for the target preset category to the model temperature parameter, and then perform an exponential function operation on the ratio to obtain the initial weight of the model, wherein the model temperature parameter is used to regulate the smoothness of the model weight distribution; and a fourth processing subunit, used to normalize the initial weight of each model to obtain the weight of each model.

[0134] Optionally, the second processing unit 202 includes: a fifth processing subunit, used to calculate the difference between the predicted probability of each model for the target sample belonging to the x-th preset category and the decision threshold of the model, and then calculate the product of the difference and the weight of the model to obtain the predicted distance of the model for the target sample belonging to the x-th preset category, where x is an integer greater than or equal to 1; a sixth processing subunit, used to calculate the average of the predicted distances of N models for the target sample belonging to the x-th preset category to obtain the average predicted distance; and a seventh processing subunit, used to convert the average predicted distance into a fused predicted probability of N models for the target sample belonging to the x-th preset category using a nonlinear activation function.

[0135] Optionally, the third processing unit 203 includes: an eighth processing subunit, used to determine the category decision result of each model for the target sample based on the relationship between the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category; a ninth processing subunit, used to determine the consistency score based on the number of similarities and differences in the category decision results of N models for the target sample; a tenth processing subunit, used to perform variance calculation based on the predicted probability of each model for the target sample belonging to each preset category and the decision threshold of that model, to obtain the variance calculation result; and an eleventh processing subunit, used to determine the confidence level of N models in judging that the target sample belongs to each preset category based on the variance calculation result and the consistency score.

[0136] Optionally, the fourth processing unit 204 includes: a twelfth processing subunit, configured to determine the y-th preset category as the target category of the target sample when the fusion prediction probability of N models for the target sample belonging to the y-th preset category is greater than or equal to the fusion classification threshold, and the confidence of N models in judging that the target sample belongs to each preset category is greater than or equal to the confidence threshold, where y is an integer greater than or equal to 1.

[0137] Optionally, the fourth processing unit 204 includes: a second acquisition subunit, used to acquire the fusion prediction probability of N models when the reference sample belongs to each preset category and the true category of the reference sample; and a thirteenth processing subunit, used to perform multiple search operations based on the fusion prediction probability of N models when the reference sample belongs to each preset category and the true category of the reference sample to obtain a fusion classification threshold, wherein each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the fusion classification threshold.

[0138] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device on which the computer-readable storage medium is located executes the above-described multi-model-based sample classification method.

[0139] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described multi-model-based sample classification method.

[0140] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the above-described multi-model-based sample classification method.

[0141] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0142] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0147] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A sample classification method based on multiple models, characterized in that, include: The target sample is input into N models, and each model determines the predicted probability of the target sample belonging to each preset category based on the sample features of the target sample, where N is an integer greater than 1; Based on the weights, decision thresholds, and prediction probabilities of each model for the target sample belonging to each preset category, the fusion prediction probabilities of the N models for the target sample belonging to each preset category are generated, wherein the weights of different models are the same or different, and the decision thresholds of different models are the same or different. Based on the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, the confidence level of the N models in judging that the target sample belongs to each preset category is determined. Based on the fusion prediction probability and the confidence level, combined with the fusion classification threshold and confidence threshold set for the N models, the target category to which the target sample belongs is determined.

2. The sample classification method based on multiple models according to claim 1, characterized in that, The decision threshold for each model is determined through the following steps: Obtain the predicted probability of each model for a reference sample belonging to each preset category, and the actual category to which the reference sample belongs; Based on the predicted probability of each model for a reference sample belonging to each preset category and the actual category of the reference sample, multiple search operations are performed to obtain the decision threshold of the model. Each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the decision threshold of the model.

3. The sample classification method based on multiple models according to claim 2, characterized in that, Each search operation includes: Uniform sampling is performed within the numerical range to obtain M candidate thresholds, where M is an integer greater than 1; Based on the predicted probability of each model for a reference sample belonging to each preset category and the actual category of the reference sample, the misclassification rate of the model for each preset category at each candidate threshold is determined. Based on the misclassification rate of each model for each preset category at each candidate threshold, the reference threshold corresponding to this search operation is determined from the M candidate thresholds; The numerical range to be used in the next search operation is determined based on the reference threshold, wherein the numerical range to be used in the next search operation includes the reference threshold determined in the current search operation, and the numerical range to be used in the next search operation is a sub-range of the numerical range used in the current search operation.

4. The sample classification method based on multiple models according to claim 3, characterized in that, Based on the misclassification rate of each model for each preset classification at each candidate threshold, a reference threshold corresponding to this search operation is determined from the M candidate thresholds, including: Obtain the misjudgment tolerance for each preset category; Based on the misjudgment rate of each model for each preset category at each candidate threshold and the misjudgment tolerance corresponding to each preset category, the reference threshold corresponding to this search operation is determined from the M candidate thresholds.

5. The sample classification method based on multiple models according to claim 4, characterized in that, Based on the misclassification rate of each model for each preset category at each candidate threshold and the misclassification tolerance corresponding to each preset category, a reference threshold corresponding to this search operation is determined from the M candidate thresholds, including: If the misjudgment tolerance of the i-th preset category is less than that of the j-th preset category, a candidate threshold satisfying the constraint condition is determined from the M candidate thresholds as the reference threshold. The constraint condition includes: the misjudgment rate of the j-th preset category is the highest under the condition that the misjudgment rate of the i-th preset category is less than the preset threshold. The i-th preset category and the j-th preset category are two different preset categories.

6. The sample classification method based on multiple models according to claim 1, characterized in that, The weights for each model are determined through the following steps: Based on the misjudgment tolerance of each preset category, the target preset category is determined from multiple preset categories; Calculate the ratio of the misclassification rate of each model for the target preset classification to the model temperature parameter, and then perform an exponential function operation on the ratio to obtain the initial weight of the model, wherein the model temperature parameter is used to adjust the smoothness of the model weight distribution; The initial weights of each model are normalized to obtain the weights of each model.

7. The sample classification method based on multiple models according to claim 1, characterized in that, Based on the weights, decision thresholds, and prediction probabilities of each model for the target sample belonging to each preset category, a fusion prediction probability of the N models for the target sample belonging to each preset category is generated, including: Calculate the difference between the predicted probability of each model for the target sample belonging to the x-th preset category and the decision threshold of the model, and then calculate the product of the difference and the weight of the model to obtain the predicted distance of the model for the target sample belonging to the x-th preset category, where x is an integer greater than or equal to 1. The average prediction distance is obtained by averaging the prediction distances of N models for the target sample belonging to the x-th preset category. The average prediction distance is converted into a fusion prediction probability of the N models for the target sample belonging to the xth preset category using a nonlinear activation function.

8. The sample classification method based on multiple models according to claim 1, characterized in that, Based on the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, the confidence levels of the N models in determining that the target sample belongs to each preset category are determined, including: Based on the relationship between the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category, the category decision result of each model for the target sample is determined. A consistency score is determined based on the number of similarities and differences in the class decision results of N models for the target sample. Based on the predicted probability of each model for the target sample belonging to each preset category and the decision threshold of the model, variance is calculated to obtain the variance calculation result. Based on the variance calculation results and the consistency score, the confidence levels of the N models in determining whether the target sample belongs to each preset category are determined.

9. The sample classification method based on multiple models according to claim 1, characterized in that, Based on the fusion prediction probability and the confidence level, combined with the fusion classification threshold and confidence threshold set for the N models, the target category to which the target sample belongs is determined, including: If the fusion prediction probability of the N models for the target sample belonging to the y-th preset category is greater than or equal to the fusion classification threshold, and the confidence of the N models in determining that the target sample belongs to each preset category is greater than or equal to the confidence threshold, then the y-th preset category is determined as the target category of the target sample, where y is an integer greater than or equal to 1.

10. The sample classification method based on multiple models according to claim 1, characterized in that, The fusion classification threshold is determined through the following steps: Obtain the fusion prediction probability of the N models for each preset category of the reference sample, and the true category of the reference sample; Based on the fusion prediction probabilities of the N models for each preset category of the reference sample and the actual category of the reference sample, multiple search operations are performed to obtain the fusion classification threshold. Each search operation is used to determine a reference threshold, and the reference threshold determined by the last search operation is used as the fusion classification threshold.

11. A sample classification device based on multiple models, characterized in that, include: The first processing unit is used to input the target sample into N models, and determine the predicted probability of the target sample belonging to each preset category based on the sample features of the target sample by each model, where N is an integer greater than 1; The second processing unit is used to generate the fusion prediction probability of the N models for the target sample belonging to each preset category based on the weights, decision thresholds and prediction probabilities of each model for the target sample belonging to each preset category, wherein the weights of different models are the same or different, and the decision thresholds of different models are the same or different. The third processing unit is used to determine the confidence level of the N models when judging that the target sample belongs to each preset category, based on the decision threshold of each model and the predicted probability of each model for the target sample belonging to each preset category. The fourth processing unit is used to determine the target category to which the target sample belongs based on the fusion prediction probability and the confidence level, combined with the fusion classification threshold and confidence level threshold set for the N models.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device on which the computer-readable storage medium is located performs the multi-model-based sample classification method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the multi-model-based sample classification method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the multi-model-based sample classification method according to any one of claims 1 to 10.

Images