Classification correction method and computer program product

Abstract

To correct predictive classification of an uncertain intermediate result using context information, thereby improving accuracy of predictive classification corresponding to the uncertain intermediate result and improving the accuracy of final classification.SOLUTION: A classification correction method includes: determining accuracy of a first intermediate result, based on the first intermediate result including a plurality of intermediate classifications and degrees of confidence thereof, output for first data by a first model of a plurality of models; determining, when the first intermediate result is determined to be uncertain, whether the statistic of features extracted by the models from an input dataset including the first input data satisfies a correction condition or not; and correcting, when the statistic satisfies the correction condition, based on context information, a first predictive classification corresponding to the uncertain first intermediate result to generate a corrected first predictive classification.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the field of artificial intelligence classification, and in particular to classification correction methods and computer program products. [Background technology]

[0002] To achieve a goal such as classification or scoring, it is often necessary to integrate multiple models for classification. In such cases, each model is typically pre-trained individually. Due to issues such as imbalance in the data used for pre-training and different task difficulty, the performance of a certain model may be relatively low. The generated intermediate results may have a high error rate when the confidence level is low, which may result in poor performance of the final classification obtained by integrating multiple models.

[0003] For example, FIG. 1 shows an example of classifying whether a person in a video is concentrating. As shown in FIG. 1, input video frames are analyzed using a model, and the analysis results are converted into different activity categories and their reliability using a softmax function. As shown in the bar graph in FIG. 1, each term in the bar graph represents an activity category, and the length of the bar represents the reliability. The activity category with the highest reliability is then selected as the activity category for the video frame using the Max(softmax) function. In the bar graph corresponding to video frames 102 to 106 shown in FIG. 1, the activity category with the highest reliability is all "looking at the screen," so the activity category for these video frames is determined to be "looking at the screen." Then, a concentration score is calculated from the activity categories of multiple video frames in the video according to a rule (the concentration score is 0 for sleeping, 0.5 for talking, and 0.25 for yawning; and if there is no sleeping, talking, or yawning, the probability of looking at the screen becomes the concentration score). In the example of Figure 1, all three activity categories in the video frames are "looking at the screen," so the final concentration score is 1, which is the probability of looking at the screen. Finally, whether the person in the video is concentrating is determined based on whether the concentration score is greater than a threshold. Because the threshold shown in Figure 1 is 0.75, the person is erroneously classified as concentrating. In video frames 102 and 104 in Figure 1, the person is not looking at the screen (represented by the dashed blocks), and the confidence level of the activity category "looking at the screen" is low. However, because "looking at the screen" has the highest confidence level among all activity categories, these two video frames 102 and 104 are erroneously recognized as "looking at the screen." In video frame 106, the person is looking at the screen (represented by the solid block), and the confidence level of "looking at the screen" is very high. Therefore, video frame 106 is correctly recognized as "looking at the screen." Finally, because the probability of looking at the screen in each video frame in the video is 1, which is greater than the threshold of 0.75, the person in this video is erroneously recognized as concentrating.This means that if the classification of an action is incorrect during the process, it may ultimately lead to a misjudgment of whether or not the person is concentrating. Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, it is common to discard results with a low confidence level. However, a single intermediate result with low confidence may be either accurate or incorrect. Discarding such intermediate results results in information loss and affects the final classification performance. For example, if two video frames 102 and 104 in FIG. 1 are discarded, the action result of video frame 106 in this video is "watching a video," so the probability of looking at the screen in this video frame is still 1, and the person in the video is mistakenly recognized as concentrating. In other words, if a low-confidence intermediate result is incorrect, discarding such a low-confidence intermediate result may ultimately lead to an incorrect determination of whether or not the person is concentrating.

[0005] In view of the above problems, the present disclosure provides a classification correction method and computer program product. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, there is provided a classification correction method including: determining certainty of a first intermediate result based on a first intermediate result output by a first model among a plurality of models for first input data, the first intermediate result including a plurality of intermediate classifications and their confidence levels; if the first intermediate result is determined to be uncertain, determining whether statistics of features extracted by the plurality of models from an input dataset including the first input data satisfy a correction condition; and if the statistics satisfy the correction condition, correcting a first predicted classification corresponding to the uncertain first intermediate result based on context information to obtain the corrected first predicted classification.

[0007] According to one embodiment of the present disclosure, the context information includes a plurality of reliable second intermediate results output by the plurality of models for the input dataset, and revising a first predicted classification corresponding to an uncertain first intermediate result based on the context information includes calculating a second predicted classification of the input dataset and a confidence level thereof based on the plurality of reliable second intermediate results, and revising the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and the confidence level thereof.

[0008] According to an embodiment of the present disclosure, revising the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its reliability includes: determining top k intermediate categories among the uncertain first intermediate result, the sum of reliability levels of which is greater than a predetermined reliability threshold; determining predicted classifications for each of the top k intermediate categories based on the reliability levels of the top k intermediate categories and the second predicted classification; and determining that the uncertain first intermediate result corresponds to the second predicted classification if the second predicted classification is included in the predicted classifications corresponding to the top k intermediate categories, where k is a positive integer.

[0009] According to an embodiment of the present disclosure, revising the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its confidence level includes determining that the uncertain first intermediate result corresponds to a classification of an opposite type to the second predicted classification, a predicted classification having the highest sum of confidence levels among the top k intermediate classifications, or a predicted classification corresponding to the intermediate classification having the highest confidence level among the top k intermediate classifications, if the second predicted classification is not included in the predicted classifications corresponding to the top k intermediate classifications.

[0010] According to an embodiment of the present disclosure, revising the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its reliability includes: determining top k intermediate categories among the uncertain first intermediate results, the sum of reliability levels of which is greater than a predetermined reliability threshold; determining a predicted classification for each of the top k intermediate categories based on reliability levels of the top k intermediate categories and the second predicted classification; determining that the uncertain first intermediate result corresponds to the second predicted classification if the second predicted classification and a predicted classification corresponding to a middle category with a second highest reliability among the top k intermediate categories are the same; and determining that the uncertain first intermediate result corresponds to the predicted classification corresponding to a middle category with a highest reliability among the top k intermediate categories if the second predicted classification and a predicted classification corresponding to a middle category with a second highest reliability among the top k intermediate categories are different, where k is a positive integer.

[0011] According to an embodiment of the present disclosure, the classification correction method further includes not correcting the uncertain intermediate result if the statistics do not satisfy the correction condition.

[0012] According to another embodiment of the present disclosure, the classification correction method further includes, if the statistics do not satisfy the correction condition, determining a classification corresponding to an intermediate classification having a maximum confidence level among the uncertain intermediate results as the corrected first predicted classification.

[0013] According to one embodiment of the present disclosure, the classification correction method further includes determining a final classification of the input data set based on the corrected first predicted classification.

[0014] According to one embodiment of the present disclosure, the data type of the input data of the plurality of models is one of video, image, audio, and text.

[0015] According to one embodiment of the present disclosure, the rule set of the modification condition is self-trained based on multiple sets of labeled data, and each set of labeled data among the multiple sets of labeled data includes multiple statistics of features extracted by the multiple models from an input sample set, and a label indicating whether the classification predicted by the multiple models based on the features extracted from the input sample set is accurate.

[0016] According to an embodiment of the present disclosure, the self-learning of the rule set of the modification condition based on the plurality of sets of labeled data includes discretizing a continuous attribute of each statistic among the plurality of statistics to obtain a discretized labeled data set; generating all candidate rules ranging from those in which a single statistic satisfies the corresponding discretization condition to those in which n statistics satisfies the corresponding discretization condition; iterating through all candidate rules in order from those in which a single statistic satisfies the corresponding discretization condition to those in which n statistics satisfies the corresponding discretization condition; determining whether labeled data corresponding to the candidate rule is included in the discretized labeled data set; and determining whether the labeled data corresponding to the candidate rule is included in the discretized labeled data set. , determining the candidate rule to be an invalid rule if the candidate rule is not included in the discretized labeled data set; and, if the labeled data corresponding to the candidate rule is included in the discretized labeled data set, determining whether the candidate rule covers only misclassified data, and if the candidate rule covers only misclassified data, adding the candidate rule to the rule set of the modified condition, removing the data covered by the candidate rule from the discretized labeled data set; and, if the candidate rule covers correctly classified data, determining the candidate rule to be an invalid rule;

[0017] According to another aspect of the present disclosure, there is provided an apparatus for use in classification correction, comprising a processor and a memory having one or more computer program modules stored therein, the one or more computer program modules, when executed by the processor, performing the classification correction method described above.

[0018] According to another aspect of the present disclosure, there is provided a computer program product comprising computer commands which, when executed by a processor, cause the processor to implement the classification modification method described above.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided having stored thereon computer commands that, when executed by a processor, cause the processor to perform the classification correction method described above.

[0020] In the above aspect of the present disclosure, whether to make a correction to the classification is determined based on the reliability of the intermediate result and feature statistics, and the context information is used to correct the predicted classification of the uncertain intermediate result, thereby improving the accuracy of the predicted classification corresponding to the uncertain intermediate result and improving the accuracy of the final classification.

[0021] The above and other objects, features, and advantages of the present disclosure will become more apparent from the detailed description of the embodiments of the present disclosure with reference to the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure, constitute a part of the specification, and are used to explain the present disclosure together with the embodiments of the present disclosure, and are not intended to limit the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same elements or steps. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 shows an example of existing techniques for classifying whether people in a video are focused or not. [Figure 2] FIG. 2 shows a flowchart of a classification correction method according to an embodiment of the present disclosure. [Figure 3] FIG. 3 shows a schematic diagram of an exemplary process for calculating a person's attention score in a video using multiple models according to an example embodiment of the present disclosure. [Figure 4] FIG. 4 shows a flowchart of a classification correction method according to an embodiment of the present disclosure. [Figure 5] FIG. 5 shows a schematic diagram of a method for correcting uncertain results in the example of centralized video classification. [Figure 6] FIG. 6 illustrates a flowchart of a process for correcting uncertain results based on context information according to an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates a flowchart of a process for calculating a predicted classification of an input dataset and its confidence level based on multiple reliable intermediate results according to an example embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates an example of two sets of video frames and their non-concentrated coefficients according to an embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates a flowchart of a process for revising a predicted classification corresponding to an uncertain intermediate result based on the predicted classification of an input data set and its confidence level according to an embodiment of the present disclosure. [Figure 10] FIG. 10 shows a schematic diagram of determining predicted classifications for each of the top-k intermediate categories according to a focused classification example of an embodiment of the present disclosure. [Figure 11] FIG. 11 illustrates a flowchart of a process for revising a predicted classification corresponding to an uncertain intermediate result based on the predicted classification of an input data set and its confidence level according to another embodiment of the present disclosure. [Figure 12] FIG. 12 illustrates a flowchart of a process for self-learning a rule set of modified conditions based on multiple sets of labeled data according to an embodiment of the present disclosure. [Figure 13] FIG. 13 illustrates a schematic diagram of generating multiple sets of labeled data in an example of convergent video prediction according to an embodiment of the present disclosure. [Figure 14] FIG. 14 shows a functional block diagram of an apparatus for use in classification correction according to an embodiment of the present disclosure. [Figure 15]FIG. 15 shows a schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although several embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather, these embodiments are provided to enable a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are merely for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] This disclosure uses flowcharts to describe steps of methods according to embodiments of the disclosure. It should be understood that previous or subsequent steps do not necessarily occur in the exact order. Conversely, various steps may be processed in the reverse order or simultaneously. At the same time, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0025] First, a flowchart of a multiple model-based classification correction method 200 according to an embodiment of the present disclosure will be described with reference to Fig. 2. As shown in Fig. 2, the multiple model-based classification correction method 200 includes the following steps S202 to S206.

[0026] In step S202, the reliability of the first intermediate result is determined based on a first intermediate result output by a first model among the multiple models for the first input data, the first intermediate result including multiple intermediate categories and their reliability. In an embodiment of the present disclosure, the model may be any artificial intelligence model. In an embodiment of the present disclosure, the first model may be a model with relatively low reliability, and the reliability of the output result is determined based on the reliability of each category of the output result. In one embodiment of the present disclosure, for a highly reliable model (e.g., the rate at which the results output by the model match the human-labeled results is above a certain threshold), the output result may be directly determined to be reliable. In another embodiment of the present disclosure, for a highly reliable model, the reliability of the output result may be determined based on the reliability of each category of the output result. In one embodiment of the present disclosure, the method of determining the reliability may, for example, compare the reliability of each category of the output result with a threshold, and determine the intermediate result to be reliable if the highest reliability is higher than the threshold. In some embodiments of the present disclosure, one intermediate result may be multiple classifications and their confidence levels obtained by inputting one input sample data into one model, and such intermediate results may be used for subsequent predictive classification after further processing.

[0027] If it is determined that the first intermediate result is uncertain, in step S204, it is determined whether the statistics of the features extracted from the input data set including the first input data by the multiple models satisfy a modification condition.

[0028] If the statistics satisfy the correction condition, in step S206, the first predicted classification corresponding to the uncertain first intermediate result is corrected based on the context information, resulting in a corrected first predicted classification. In an embodiment of the present disclosure, the context information of an input sample (or a set of input data) may refer to information of other input samples related to the input sample, and these other input samples belong to the same sample set as the input sample. In some embodiments of the present disclosure, these other input samples may reflect the overall information of the sample set to which the input sample belongs. For example, in the field of image processing, the context information may refer to information contained in pixels surrounding a pixel. In the field of video processing, the context information may refer to information contained in video frames surrounding (before and after) a video frame. In the field of text processing, the context information may refer to sentences before and after a sentence. In the field of speech recognition, the context information may refer to speech before and after a speech. In the medical field, context information for one test result may refer to other test results that a patient underwent at the same time as the test result and / or test results for the patient at different times.

[0029] Referring to FIG. 3, the classification correction method 200 will be further described using the concentration score of a student during a class in a video as an example. FIG. 3 illustrates a schematic diagram of an exemplary process for calculating the concentration score of a person in a video using multiple models according to an embodiment of the present disclosure. As shown in FIG. 3, video frames 310a-310p are input as input data to a face keypoint detection model 302, a head pose model 304, a talking / yawning recognition model 306, and a mask recognition model 308, respectively. The face keypoint detection model 302 generates a mark indicating whether a face can be cut out from the video frame. The head pose model generates a mark indicating whether the head in the video frame is facing up, down, sideways, or forward. The talking / yawning recognition model generates a mark indicating whether the person in the video frame is talking / yawning. The mask recognition model generates a mark indicating whether the person in the video frame is wearing a mask. As a result, video frames 320a-320p having the above-mentioned marks are generated. In one example, the face keypoint detection model 302 generates an intermediate result including a classification of face croppable and face uncroppable and its confidence level, and determines a mark indicating whether the face can be cropped or not from the confidence levels of these two classifications. The head pose model 304 generates an intermediate result including a classification of head up, head down, sideways, and frontal orientation and its confidence level, and generates a mark indicating whether the head is up, down, sideways, or frontal orientation from the confidence levels of these classifications. The speaking / yawning recognition model 306 generates an intermediate result including a classification of speaking and yawning and its confidence level, and determines a mark indicating whether the person is speaking / yawning from the confidence levels of these two classifications. The mask recognition model 308 generates an intermediate result including a classification of masked and unmasked and its confidence level, and generates a mark indicating whether the person is wearing a mask or not from the confidence levels of these two classifications. In an embodiment of the present disclosure, one intermediate result includes different categories recognized by one model and their confidence levels. Video frames with the above-mentioned marks and a frontal head pose are input to a facial action recognition model.However, only video frames in which the face can be cut out and the head is facing forward (shown as unshaded rectangular blocks in Figure 3; shaded blocks indicate video frames in which the face cannot be cut out or the face can be cut out but the head is not facing forward) are input to the facial action recognition model 312 for calculation.

[0030] The facial activity recognition model 312 calculates the confidence of different activity categories (which may be referred to herein as intermediate categories, e.g., looking at screen, looking away, blinking, occluding face, talking, yawning, etc.) recognized from the video frames. In one embodiment, the activity category with the highest confidence is determined as the recognized activity category, thereby obtaining activity-category-labeled video frames 330a-330p. In the example of FIG. 3, the activity category "looking at screen" is indicated by a dashed block, the activity category "blinking" is indicated by a dashed-dotted block, and the activity category "looking away" is indicated by a solid block. In one embodiment, corresponding to step S202, an activity classification whose highest confidence exceeds a preset threshold (e.g., 70%) is determined to be a certain activity classification result (referred to as a certain intermediate result in this specification and shown by a shaded block in Figure 3), and an activity classification whose highest confidence does not exceed the preset threshold is determined to be an uncertain activity classification result (referred to as an uncertain intermediate result in this specification and shown by an unshaded block in Figure 3), that is, the certainty of the first intermediate result is determined based on the first intermediate result output by the first model for the first input data. Therefore, in the example illustrated in FIG. 3 , video frame 330a represents an activity category of “blinking” that is certain, video frame 330c represents an activity category of “looking at the screen” that is uncertain, video frame 330e represents an activity category of “looking elsewhere” that is certain, and video frame 330f represents an activity category of “blinking” that is uncertain. The example in FIG. 3 illustrates a case where models are serially connected. Here, data is input to one model, which outputs an intermediate result as input to another model, which then outputs another intermediate result. Each intermediate result may be uncertain and may affect subsequent predicted classifications. Note that an embodiment of the present disclosure is also applicable to a case where models are serially connected.In other words, data is fed into multiple models in parallel, each of which produces an intermediate result, any of which may be uncertain and may affect the subsequent predicted classification.

[0031] For a video frame with an uncertain activity categorization result, it is necessary to determine whether the video frame needs to be corrected. This corresponds to step S204. That is, if it is determined that the first intermediate result is uncertain, it is determined whether the statistics of features extracted from an input dataset including first input data by multiple models satisfy a correction condition. If the video frame with an uncertain activity categorization result needs to be corrected, the predicted classification of the video frame with an uncertain activity categorization result is corrected based on the reliable correction direction of the prediction results of surrounding video frames, and video frames 340a-340p having the corrected predicted classification are obtained. This corresponds to step S206. That is, if the statistics satisfy the correction condition, the first predicted classification corresponding to the uncertain first intermediate result is corrected based on context information to obtain the corrected first predicted classification. In the example of FIG. 3, the context information refers to the video frames surrounding the video frame with an uncertain activity categorization result. The activity classification results generated by each of the surrounding video frames correspond to one predicted classification, and based on the predicted classifications corresponding to each surrounding video frame, a predicted classification corresponding to the entirety of all surrounding video frames can be obtained, which can be used as a direction to correct the predicted results of the surrounding video frames, i.e., the predicted classification of the video frames whose activity classification results are uncertain.

[0032] In one embodiment, the modified predicted classification can be converted into a single behavior categorization result. For example, if the predicted classification of a video frame is modified to a convergent classification, the modified predicted classification is converted into "watching the screen" and calculated as a probability of "watching the screen"; otherwise, the modified predicted classification is converted into "watching the screen" and not calculated as a probability of "watching the screen."

[0033] If there are no video frames that need to be corrected, a concentration score is calculated based on the behavior classification results (referred to as intermediate results in this specification) of each video frame 340a-340p. In the example of Figure 3, the concentration score is calculated based on the following formula (1). Concentration score = min(sleep score, talking score, yawn score, probability of looking at screen) (1) For example, the sleep score is 0 if the person is sleeping and 1 if they are not. The talking score is 0.5 if they are talking and 1 if they are not. The yawning score is 0.25 if they are yawning and 1 if they are not. In other words, in this example, if they are sleeping, the sleep score is low, if they are talking, the talking score is low, and if they are yawning, the yawning score is low, and the probability of looking at the screen is the percentage of frames in which the person looks at the screen in the actual video. Formula (1) indicates that as long as there is one case of non-concentration, the concentration score will be low, and the lower the concentration score, the less focused they are, and the higher the concentration score, the more focused they are. Therefore, the concentration score can be used to determine whether or not someone is concentrating.

[0034] When there is a video frame that needs to be corrected, if the original activity classification result of the video frame with uncertain activity classification result is “looking to the side” (corresponding to the non-focused classification), and the video frame with uncertain activity classification result is corrected so that the corrected predicted classification is the focused classification, the corrected predicted classification will be converted into the activity classification result of “looking at the screen”, and the “probability of looking at the screen” in formula (1) will increase, and the final concentration score may be changed.

[0035] FIG. 4 illustrates a flowchart of a classification correction method 400 according to an embodiment of the present disclosure. As illustrated in FIG. 4, the classification correction method 400 may include steps S402 to S408. Steps S402 to S406 of the classification correction method 400 in FIG. 4 are the same as steps S202 to S206 of the classification correction method 200 in FIG. 2, and therefore will not be described again here. In step S408, a final classification of the input data set is determined based on the corrected first predicted classification. In one embodiment of the present disclosure, not only is the final classification of the input data set determined, but a score for the final classification is also obtained (e.g., as shown in the example of FIG. 3). In one embodiment, if it is determined in step S404 that the statistics do not satisfy the correction condition, no correction may be required for the uncertain intermediate results. In one embodiment, if it is determined in step S404 that the statistics do not satisfy the correction condition, a predicted classification corresponding to the category with the highest confidence among the uncertain intermediate results may be determined as the corrected first predicted classification.

[0036] FIG. 5 is a schematic diagram of a method for correcting uncertain results in an example of centralized video classification. As shown in FIG. 5, the classification correction method may select the top few activity categories whose sum of confidence exceeds a predetermined threshold (e.g., 70%) for determining that the classification result is certain, and map the top few activity categories to a centralized, neutral, or non-centralized category based on the mapping relationship between the activity categories and the classification of whether they are centralized or not. Then, the proportion of the sum of confidence for each of the centralized, neutral, and non-centralized categories to the sum of confidence for the top few activity categories may be calculated, and whether the uncertain video frame is classified as a non-centralized or centralized category may be determined based on the proportion of the centralized category. As can be seen from FIG. 5, video frame 504 was recognized as the activity category with the highest confidence as "looking at the screen," but the confidence of "looking at the screen" did not exceed 70%, which corresponds to an uncertain situation (indicated by the dashed block). The confidence levels of the two previous behavioral categories recognized by video frame 504 are above 70%, and these two behavioral categories are "looking at the screen" and "covering the face," respectively, both of which correspond to the focused category. Therefore, the percentage of the focused category is 1, and the percentages of the neutral and non-focused categories are both 0. Therefore, it can be concluded that video frame 504 corresponds to the focused category. Note that video frame 502 is definitely recognized as the "tilting head" behavioral category, which corresponds to the focused category. And video frame 506 is definitely recognized as the "looking down" behavioral category, which also corresponds to the focused category. Finally, by combining the corresponding focus / non-focus classifications of video frames 502-506, it is possible to determine that the person in the video is focused. However, this conclusion is incorrect. The person in the video changes from a head-upward and upward position in video frame 502 to an intermediate position in video frame 504 to a head-downward and downward position in video frame 506. This transition of the person's state in the video clearly indicates non-focus. In this disclosure, it is noted that single-frame correction may limit the performance of the correction due to the localized nature of the information, leading to errors in the final classification conclusion.

[0037] In view of this, the present disclosure proposes a method for correcting a predicted classification corresponding to an uncertain intermediate result by taking context information into consideration. FIG. 6 shows a flowchart of a process 600 for correcting an uncertain result based on context information according to an embodiment of the present disclosure. As shown in FIG. 6, the process 600 may correspond to step S206 of the method 200 shown in FIG. 2 and step S406 of the method 400 shown in FIG. 4. In one embodiment (e.g., the embodiment illustrated in FIG. 6), the context information may correspond to multiple certain intermediate results output by multiple models based on an input dataset (e.g., in the example of FIG. 3, the video frame 330a represents an activity classification result of "blink" and is certain, and the video frame 330e represents an activity classification result of "look away" and is certain), and the input dataset includes multiple input data related to the input data of the uncertain intermediate result. The multiple input data reflect information within the domain of a single input data of the uncertain intermediate result. In one embodiment, the above-mentioned multiple reliable intermediate results include not only multiple reliable intermediate results output by one model for different input data, but also multiple reliable intermediate results output by different models for the same input data. In one embodiment, the above-mentioned multiple reliable intermediate results include results directly recognized as reliable, output by a reliable model. In process 600 as shown in FIG. 6 , in step S602, a predicted classification of the input dataset and its reliability may be calculated based on the multiple reliable intermediate results (which may be referred to as multiple reliable second intermediate results in this specification). In one embodiment of the present disclosure, the predicted classification with the highest percentage of predicted classifications among the multiple predicted classifications corresponding to the multiple reliable intermediate results may be determined as the predicted classification of the multiple reliable intermediate results, and this percentage may be set as the reliability of the predicted classification of the multiple reliable intermediate results. Those skilled in the art will appreciate that the present disclosure may also use other reasonable methods to calculate the predicted classification of the input dataset and its reliability. Then, in step S604, the predicted classification corresponding to the uncertain intermediate results may be revised based on the predicted classification of the input dataset and its reliability.

[0038] In an embodiment of the present disclosure, the intermediate result corresponds to one predicted classification, for example, each different category recognized from one intermediate result corresponds to one predicted classification. Based on the reliability of each category and the predicted classification, a predicted classification of the intermediate result (referred to herein as a second predicted classification) is obtained. As shown in the example of FIG. 3 , the category of “looking at the screen” corresponds to the focused classification, the category of “looking elsewhere” corresponds to the unfocused classification, the category of “blinking” corresponds to the neutral classification, the category of “talking” corresponds to the unfocused classification, and the category of “looking down” corresponds to the focused classification, etc. In one example, the predicted classification corresponding to the category with the highest reliability among the multiple categories of the intermediate result is determined as the predicted classification corresponding to the intermediate result. In another example, a predicted classification corresponding to the intermediate result may be obtained based on the predicted classification of the reliability of each classification of the intermediate result using other rules, for example, by selecting multiple classifications whose sum of reliability is higher than a predetermined threshold, calculating the sum of reliability of each predicted classification corresponding to the multiple classifications, and selecting the predicted classification with the highest sum of reliability as the predicted classification corresponding to the intermediate result.

[0039] FIG. 7 illustrates a flowchart of a process 700 for calculating a predicted classification of an input dataset and its reliability based on multiple reliable intermediate results according to an embodiment of the present disclosure. The illustrated process 700 may correspond to step S602, i.e., calculating a predicted classification of an input dataset and its reliability based on multiple reliable intermediate results. The example in FIG. 7 illustrates an example of a scene for recognizing whether a person in a video is concentrating. The process 700 for calculating a predicted classification of an input dataset and its reliability illustrates a process for calculating a non-concentration coefficient. In step S702, the number of reliable activity classification results, Ncca, is calculated. The higher the number of reliable activity classification results, the higher the likelihood that the student is not concentrating, which is because the student is wandering around and engaging in various activities. The reliable activity classification results may be mapped to a non-concentration classification or a concentration classification, and in step S704, the number of reliable non-concentration activity classifications, Nne, is calculated. The higher the proportion of non-concentration activities, the higher the likelihood that the student is not concentrating. In step S706, the number of switching times Ns between uncertain and certain activity categories is calculated. The higher the number of switching times, the more likely the students are not concentrated. This is because a large change in the distribution of the reliability of the prediction results indicates that the state of student movement is not stable. Then, in step S708, a non-concentration coefficient, i.e., a non-concentration reliability, may be calculated based on one or a combination of the number of certain activity categories Ncca, the number of certain non-concentration activity categories Nne, and the number of switching times Ns. In one embodiment, the non-concentration coefficient may be calculated using the following formula (2): Deconcentration coefficient = W1*Ncca+W2*[Nne / (number of frames - 1)]+W3*(Ns / number of frames) (2) Here, W1, W2, and W3 are weighting coefficients, and W1+W2+W3=1. For example, W1=0.5, W2=0.25, and W3=0.25.

[0040] FIG. 8 illustrates two exemplary sets of video frames and their decentralization coefficients according to an embodiment of the present disclosure. As shown in FIG. 8, each set of video frames includes 10 video frames. If the highest confidence level of the predicted video frame classification is equal to or greater than a predetermined threshold (70% in the example of FIG. 8), the activity classification of the video frame is deemed certain (indicated by a solid-lined block in FIG. 8); otherwise, the activity classification of the video frame is deemed uncertain (indicated by a dashed-lined block in FIG. 8). The decentralization coefficients of each set of video frames in FIG. 8 are calculated with reference to the method described in FIG. 7. The upper set of video frames in FIG. 8 has a relatively high decentralization coefficient of 0.65. However, the uncertain video frames are in an intermediate transition state, representing a decentralized action (e.g., a process of turning away from the face or bowing the head). The lower set of video frames in FIG. 8 has a relatively low decentralization coefficient of 0.17. However, uncertain video frames, due to, for example, head tilt or blurry image quality, indicate where the person in the video is actually focused.

[0041] FIG. 9 illustrates a flowchart of a process 900 for correcting a predicted classification corresponding to an uncertain intermediate result (referred to herein as a first predicted classification) based on the predicted classification of the input dataset and its confidence level according to an embodiment of the present disclosure. As shown in FIG. 9 , in step S902, among the uncertain intermediate results (referred to herein as the first intermediate results), a top k intermediate classifications whose sum of confidence levels is higher than a predetermined confidence threshold may be determined, where k is a positive integer. In step S904, a predicted classification for each of the top k intermediate classifications may be determined based on the confidence levels of the top k intermediate classifications and the predicted classification of the input dataset. In one embodiment, there is a mapping relationship between the confidence levels of the intermediate classifications and the predicted classification of the input dataset (referred to herein as a second predicted classification), and the predicted classification, and step S904 is implemented based on this mapping relationship. FIG. 10 illustrates a schematic diagram for determining a predicted classification for each of the top k intermediate classifications based on a mapping according to an example of a focused classification according to an embodiment of the present disclosure. As shown in Figure 10, the mapping table indicates whether different categories are classified as clustered or not under different decentralized confidence levels. In the example of Figure 10, two video frames in the "head tilting" category have different decentralized confidence levels (referred to as decentralized coefficients in the examples of Figures 7 and 8), with the decentralized confidence level of the left video frame being greater than 0.4 and the decentralized confidence level of the right video frame being less than 0.4. Examining the mapping table, it can be seen that the left video frame corresponds to the decentralized classification, and the right video frame corresponds to the clustered classification. Similarly, two video frames in the "yawning" category are mapped to the decentralized and clustered classifications, respectively, under different decentralized confidence conditions. Furthermore, two video frames in the "face occluding" category and two video frames in the "talking" category are similar. It can be seen from the mapping table that in the category of "looking at screen", the video frames correspond to the focused classification regardless of whether the decentralized confidence is greater than 0.4 or not, and in the categories of "looking to the side" and "looking away", the video frames correspond to the decentralized classification regardless of whether the decentralized confidence is greater than 0.4 or not.

[0042] In step S906, it may be determined whether the predicted classifications corresponding to the top k intermediate categories include a correction direction. In an embodiment of the present disclosure, the correction direction may be a predicted classification to which the uncertain intermediate result should be corrected if it satisfies a correction condition, or may be a predicted classification when the confidence level of the predicted classification of the input dataset is greater than a preset threshold. For example, in FIGS. 7 and 8, if the decentralization coefficient is greater than 0.4, the correction direction is a non-centralized classification; otherwise, the correction direction is a concentrated classification. Returning to FIG. 9, in step S908, if the predicted classifications corresponding to the top k intermediate categories include a correction direction, it may be determined that the uncertain intermediate result corresponds to the correction direction. If the predicted classifications corresponding to the top k intermediate categories do not include a correction direction, it may be determined in step S910 that the uncertain intermediate result corresponds to a classification of the opposite type to the correction direction. In one embodiment, if the confidence level of the predicted classification of the input dataset (referred to herein as a second predicted classification) is greater than a preset threshold, the predicted classification of the input dataset may be considered as the correction direction. In another embodiment, if the confidence in the predicted classification of the input data set is below a preset threshold, a classification of the opposite type to the predicted classification of the input data set may be considered as the correction direction. In one embodiment, the correction direction may be considered as the final classification of the input data set.

[0043] If the predicted classifications corresponding to the top k intermediate classifications do not include a direction of correction, then in step S910, a person skilled in the art may determine that the uncertain intermediate result corresponds to the predicted classification among the top k intermediate classifications that has the highest sum of confidence, or the predicted classification corresponding to the intermediate classification among the top k features that has the highest confidence.

[0044] FIG. 11 shows a flowchart of a process 1100 for correcting a predicted classification corresponding to an uncertain intermediate result based on the predicted classification of an input dataset and its confidence level, according to another embodiment of the present disclosure. As shown in FIG. 11, the process 1100 may include the following steps S1102 to S1110. Steps S1102 to S1104 of the process 1100 in FIG. 11 are the same as steps S902 to S904 of the process 900 in FIG. 9 , and therefore will not be described again here. In step S1106, it may be determined whether the correction direction is the same as the predicted classification corresponding to the intermediate classification with the second highest confidence level among the top-k intermediate classifications. If the correction direction is the same as the predicted classification corresponding to the intermediate classification with the second highest confidence level among the top-k intermediate classifications, it may be determined in step S1108 that the uncertain intermediate result corresponds to the correction direction. If the correction direction differs from the predicted classification corresponding to the intermediate classification with the second highest confidence among the top-k intermediate classifications, then in step S1110, it may be determined that the uncertain intermediate result corresponds to the predicted classification corresponding to the intermediate classification with the highest confidence among the top-k intermediate classifications. As mentioned above, in one embodiment, if the confidence of the predicted classification of the input data set is higher than a predetermined threshold, the predicted classification of the input data set may be considered as the correction direction. In another embodiment, if the confidence of the predicted classification of the input data set is equal to or lower than a predetermined threshold, a classification of the opposite type to the predicted classification of the input data set may be considered as the correction direction.

[0045] As described above, for an uncertain intermediate result, the predicted classification is revised only if the uncertain intermediate result satisfies the revision condition. FIG. 12 shows a flowchart of a process 1200 for self-training a rule set of revision conditions based on multiple sets of labeled data according to an embodiment of the present disclosure. In the process 1200 shown in FIG. 12, in step S1202, multiple training models extract features from a single input sample set and calculate multiple statistics of the features. The multiple training models then label the classification for the input sample set as accurate or incorrect, thereby generating a set of labeled data. Multiple sets of labeled data are obtained by labeling different input sample sets. In one embodiment, the multiple training models output multiple statistics for a single input data set, each statistic being generated for features extracted from multiple input data in the single input data set. In one embodiment, each set of labeled data includes the multiple statistics output by the multiple training models for a single input data set and a label indicating whether the predicted classification based on the features extracted from the input sample set is accurate or incorrect. As an example, hard decision is adopted for classifying the context feature set, but in other examples, other appropriate decision methods may be adopted.

[0046] In step S1204, the continuous attribute of each statistic among the plurality of statistics is discretized to obtain a discretized labeled data set. The number of the plurality of statistics is n, where n is a positive integer. In one embodiment of the present disclosure, the continuous attribute of each statistic among the plurality of statistics may be discretized according to a preset discretization condition to obtain a discretized labeled data set. In another embodiment of the present disclosure, other suitable discretization methods may be adopted, such as an equal-width discretization method, an equal-frequency discretization method, etc.

[0047] In step S1206, all candidate rules are generated, from those for which a single statistic satisfies the corresponding discretization condition to those for which n statistics satisfies the corresponding discretization condition. In step S1208, all candidate rules are cycled through in order from those for which a single statistic satisfies the corresponding discretization condition to those for which n statistics satisfies the corresponding discretization condition. For each candidate rule, in step S1210, it is determined whether labeled data corresponding to the candidate rule is included in the discretized labeled data set. If labeled data corresponding to the candidate rule is not included in the discretized labeled data set, in step S1212, the candidate rule is determined to be an invalid rule. If labeled data corresponding to the candidate rule is included in the discretized labeled data set, in step S1214, it is determined whether the candidate rule covers only incorrectly classified data. In step S1214, rules for incorrectly classified samples are filtered out. If the candidate rule covers correctly classified data, the process proceeds to step S1212, where the candidate rule is determined to be an invalid rule. If the candidate rule covers only misclassified data, then in step S1216, the candidate rule is added to the modified condition rule set, and the data covered by the candidate rule is removed from the discretized labeled data set.

[0048] FIG. 13 is a schematic diagram illustrating generating multiple sets of labeled data in an example of converged video prediction according to an embodiment of the present disclosure. FIG. 13 further illustrates the process of generating multiple sets of labeled data in step S1202 using converged video prediction as an example. Video frames of a video are first input to a face keypoint detection model 1302. The face keypoint detection model 1302 generates a mark indicating that the face cannot be cropped. Then, video frames with a mark indicating that the face can be cropped are input to a head pose model 1304, a talking / yawning recognition model 1306, and a mask recognition model 1308. The head pose model 1304 generates a mark indicating whether the head is facing up, down, sideways, or forward; the talking / yawning recognition model 1306 generates a mark indicating that the head is facing up or down; and the mask recognition model 1308 generates a mark indicating that the head is facing a mask. Video frames with a frontal head pose are input into the facial action recognition model to generate a facial action classification and its confidence score. Then, the "looking at screen" mark for the video frame is obtained from the hard decision result. Furthermore, the activity ratio is generated based on the "looking at screen" mark, mask mark, talking / yawning mark, etc. Finally, the concentration score is predicted based on the activity ratio. The concentration scores of people in the video are manually labeled to indicate whether the concentration classification is correct (represented by 1) or incorrect (represented by 0).

[0049] Meanwhile, statistics f1 to fn of context features are generated by a face keypoint detection model 1302, a head pose model 1304, a talking / yawning recognition model 1306, a mask recognition model 1308, etc. As shown in FIG. 13, f1 is the "proportion of faces that cannot be cropped," and fn is the uncertainty rate of video frames. Each video may correspond to one context feature set and a label indicating whether the convergent classification of this video is accurate or not, i.e., labeled structured data may be generated. Furthermore, as described in step S1204, a discretized labeled dataset is obtained by attribute discretizing the continuous values of the labeled structured data. For example, as shown in FIG. 13, the discretization condition for the statistic f1 is whether the continuous value of f1 is greater than 0.8. Therefore, when the continuous value of f1 is 0.8 or less, the discrete value of the statistic f1 is A1 (e.g., 0), and when the continuous value of f1 is greater than 0.8, the discrete value of the statistic f1 is A2 (e.g., 1). Similarly, the discretization condition for the statistic fn is whether the continuous value of fn is greater than 5. Therefore, when the continuous value of f1 is 5 or less, the discrete value of the statistic fn is C1 (e.g., 0), and when the continuous value of fn is greater than 5, the discrete value of the statistic fn is C2 (e.g., 1). The form of discrete division shown in FIG. 13 is an example and not a limitation. Those skilled in the art should understand that the discretization condition for each statistic may discretize the continuous value into more types of discrete values, such as two or more types of discrete values. In one embodiment of the present disclosure, the discretization condition may be set based on the meaning represented by the feature statistic, or a conventional method (e.g., equal-width discretization method, equal-frequency discretization method, etc.) may be adopted and the discretization condition or method may be adjusted by comparing the classification performance when a rule set is automatically generated.

[0050] Although the present disclosure has described the technical solution of the present disclosure using a video centralized classification scenario as an example in Figures 3, 5, 7, 8, 10, and 13, it should be understood by those skilled in the art that the technical solution of the present disclosure may be extended and applied to other scenarios. For example, the input data input to the integrated multiple models may be video, image, text, or audio. In the application scenario of text or audio processing, there are different types of models for processing text, such as word segmentation models and semantic classification models. A word segmentation model outputs different word categories and their confidence levels for input text or audio (referred to herein as intermediate results including multiple intermediate categories and their confidence levels). For a word category result with a relatively low confidence level, if the statistics of the word features extracted from the input text satisfy a correction condition, the predicted word semantic classification corresponding to the word (referred to herein as predicted classification) is corrected based on the text surrounding the word (referred to herein as context information).

[0051] In addition, in a disease diagnosis application scenario, the input data may be in different medical forms, such as MRI (Magnetic Resonance Imaging) data, FD-PET (Fluorodeoxyglucose-Positron Emission Tomography) data, CSF (Cerebrospinal Fluid) data, or genetic data. At the same time, different analysis models (corresponding to multiple models in this specification) exist for different types of input data, such as an MRI data analysis model, an FD-PET data analysis model, a CSF data analysis model, and a genetic data analysis model. Each of the MRI data, FD-PET data, CSF data, and genetic data for one patient may be considered as one input sample, and various types of data for one patient may constitute an input sample set. The MRI analysis results (referred to herein as intermediate results) obtained by the MRI data analysis model include multiple MRI diagnosis categories (referred to herein as intermediate categories) and their reliability. For MRI analysis results with relatively low reliability, if the statistical quantities of features extracted from an input sample set consisting of various types of data from the patient satisfy the correction conditions, the predicted disease classification (referred to herein as predicted classification) corresponding to the MRI analysis results may be corrected based on information (referred to herein as context information) from the FD-PET data, CSF data, and genetic data of the same patient.

[0052] The classification correction method based on multiple models according to the present disclosure can retain the contribution of all information rather than simply discarding input data with uncertain intermediate results. Furthermore, the classification correction method based on multiple models according to the present disclosure determines whether to correct the classification based on the reliability of the intermediate results and feature statistics, and corrects the predicted classification of input data with uncertain intermediate results based on context information, thereby improving the overall classification accuracy. Furthermore, the integrated model self-learns feature rules that are prone to errors and generates a rule set of correction conditions, which corrects error-prone areas when intermediate results are uncertain, thereby improving the accuracy of the final classification.

[0053] An apparatus 1400 used for classification correction according to an embodiment of the present disclosure will now be described with reference to Figure 14. Figure 14 is a schematic diagram of an apparatus used for classification correction according to an embodiment of the present disclosure. The functions of the apparatus used for classification correction in this embodiment are the same as the details of the method described above with reference to Figures 2 and 4, and therefore, for simplicity, a detailed description of the same content will be omitted here.

[0054] The classification correction apparatus of the present disclosure includes a processor 1402 and a memory 1401 having computer-readable commands stored therein. When the computer-readable commands are executed by the processor, a classification correction method is performed. The classification correction method includes: determining a certainty of a first intermediate result based on a first intermediate result output by a first model of a plurality of models for first input data, the first intermediate result including a plurality of intermediate classifications and their confidence levels; if the first intermediate result is determined to be uncertain, determining whether statistics of features extracted by the plurality of models from an input dataset including the first input data satisfy a correction condition; and if the statistics satisfy the correction condition, correcting a first predicted classification corresponding to the uncertain first intermediate result based on context information to obtain a corrected first predicted classification.

[0055] For the technical effects of the apparatus 1400 for classification based on multiple models in different embodiments, reference may be made to the technical effects of the classification correction method based on multiple models disclosed in the embodiments of the present disclosure, and redundant explanations will be omitted here.

[0056] The multiple model based classification apparatus 1400 can be applied to a variety of suitable electronic devices.

[0057] FIG. 15 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure.

[0058] 15 , the present disclosure further provides a computer-readable storage medium 1500 having computer-readable instructions 1501 stored thereon. When the computer-readable instructions are executed by a computer, the computer performs a classification correction method based on multiple models. The method includes: determining a certainty of a first intermediate result, the first intermediate result including a plurality of intermediate classifications and their confidence levels, output by a first model among the multiple models for first input data; determining whether statistics of features extracted by the multiple models from an input dataset including the first input data satisfy a correction condition if the first intermediate result is determined to be uncertain; and correcting a first predicted classification corresponding to the uncertain first intermediate result based on context information if the statistics satisfy the correction condition, to obtain the corrected first predicted classification.

[0059] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, the order of the processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, although elements of various steps of the methods described herein are presented using an example order, they are not limited to the particular order presented.

[0060] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0061] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0062] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0063] It should be understood by those skilled in the art that various aspects of the present application can be illustrated and described in terms of several classes or contexts that may be patentable, including any new and useful process, machine, manufacture, or combination of matter, or any new and useful improvement thereof. Accordingly, various aspects of the present application may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. Such hardware or software may be referred to as "data blocks," "modules," "engines," "units," "components," or "systems." Various aspects of the present application may also be expressed as a computer product including one or more computer-readable programs residing in a computer-readable medium.

[0064] In this application, certain terms are used to describe embodiments of the application. For example, "one embodiment," "one embodiment," and / or "some embodiments" refer to characteristics, configurations, or features related to at least one embodiment of the application. Therefore, it should be noted that "one embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more times in different places in this specification do not necessarily refer to the same embodiment. Furthermore, some characteristics, configurations, or features of one or more embodiments of the application may be combined as appropriate.

[0065] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant technical field, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0066] Although the invention according to the present disclosure has been described in detail above, it is clear to those skilled in the art that the invention according to the present disclosure is not limited to the embodiments described in the present disclosure. The invention according to the present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the description of the claims. Therefore, the description of the present disclosure is intended to be illustrative and explanatory and does not impose any limiting meaning on the invention according to the present disclosure.

Claims

1. determining a certainty of a first intermediate result based on a first intermediate result output by a first model of the plurality of models for the first input data, the first intermediate result including a plurality of intermediate classifications and their confidence levels; determining whether statistics of features extracted by the plurality of models from an input dataset including the first input data satisfy a correction condition if the first intermediate result is determined to be uncertain; and if the statistics satisfy the correction condition, correcting the first predicted classification corresponding to the uncertain first intermediate result based on context information to obtain a corrected first predicted classification.

2. the context information includes a plurality of reliable second intermediate results output by the plurality of models for the input dataset; Revising a first predicted classification corresponding to an uncertain first intermediate result based on the context information includes: calculating a second predicted classification of the input data set and a confidence level thereof based on the plurality of reliable second intermediate results; and revising the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its reliability.

3. modifying the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its reliability; determining a top k intermediate classifications of the uncertain first intermediate results whose sum of confidences is greater than a preset confidence threshold; determining a predicted classification for each of the top-k intermediate classifications based on the top-k intermediate classifications and the confidence level of the second predicted classification; determining that the uncertain first intermediate result corresponds to the second predicted classification if the second predicted classification is included in the predicted classifications corresponding to the top k intermediate classifications; 3. The classification correction method of claim 2, wherein k is a positive integer.

4. modifying the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its reliability; 4. The classification correction method of claim 3, further comprising: if the predicted classifications corresponding to the top k intermediate classifications do not include the second predicted classification, determining that the uncertain first intermediate result corresponds to a classification of an opposite type to the second predicted classification, a predicted classification having a highest sum of confidence among the top k intermediate classifications, or a predicted classification corresponding to the intermediate classification having a highest confidence among the top k intermediate classifications.

5. modifying the first predicted classification corresponding to the uncertain first intermediate result based on the second predicted classification and its reliability; determining a top k intermediate classifications of the uncertain first intermediate results whose sum of confidences is greater than a preset confidence threshold; determining a predicted classification for each of the top-k intermediate classifications based on the top-k intermediate classifications and the confidence level of the second predicted classification; determining that the uncertain first intermediate result corresponds to the second predicted classification if the second predicted classification is the same as the predicted classification corresponding to the intermediate classification with the second highest confidence among the top k intermediate classifications; if the second predicted classification differs from a predicted classification corresponding to a second most reliable intermediate classification among the top k intermediate classifications, determining that the uncertain first intermediate result corresponds to a predicted classification corresponding to a most reliable intermediate classification among the top k intermediate classifications; 3. The classification correction method of claim 2, wherein k is a positive integer.

6. The classification correction method of claim 1 , further comprising determining a final classification of the input data set based on the corrected first predicted classification.

7. The classification correction method of claim 1 , wherein the data type of the input data for the plurality of models is one of video, image, audio, and text.

8. self-learning a rule set of the modified conditions based on a plurality of sets of labeled data; 6. The classification correction method according to claim 1, wherein each set of labeled data among the plurality of sets of labeled data includes a plurality of statistics of features extracted by the plurality of models from one input sample set, and a label indicating whether a classification predicted by the plurality of models based on the features extracted from the input sample set is accurate.

9. self-learning the rule set of the modification condition based on the plurality of sets of labeled data; discretizing a continuous attribute of each statistic of the plurality of statistics to obtain a discretized labeled data set; generating all candidate rules ranging from those in which a single statistic satisfies the corresponding discretization condition to those in which n statistics satisfy the corresponding discretization condition; iterating through all candidate rules in order from those with a single statistic satisfying the corresponding discretization condition to those with n statistics satisfying the corresponding discretization condition, and determining whether the labelled data corresponding to the candidate rule is included in the discretised labelled data set; If the labeled data corresponding to the candidate rule is not included in the discretized labeled data set, the candidate rule is determined to be an invalid rule; If the candidate rule contains labeled data in the discretized labeled data set, determining whether the candidate rule covers only misclassified data; If the candidate rule covers only misclassified data, add the candidate rule to the modified condition rule set and remove data covered by the candidate rule from the discretized labeled data set; determining the candidate rule to be an invalid rule if the candidate rule covers correctly classified data; 9. The method of claim 8, wherein the number of said plurality of statistics is n, where n is a positive integer.

10. 1. A computer program product including computer commands, A computer program product, wherein the computer commands, when executed by a processor, cause the classification correction method of any one of claims 1 to 9 to be performed.

Images