Classification prediction model evaluation method and device, equipment, medium and program product

Abstract

The invention discloses a classification prediction model evaluation method and device, equipment, a medium and a program product, and relates to the field of machine learning. The method comprises the following steps: acquiring first sample data and a plurality of pieces of second sample data; acquiring data association degrees between the plurality of pieces of second sample data and the first sample data; based on the data association degrees corresponding to the multiple pieces of second sample data, fusing the confidence values corresponding to the multiple pieces of second sample data to obtain a confidence label corresponding to the first sample data; performing confidence value prediction on the first sample data through a sample confidence prediction network to obtain a predicted confidence value corresponding to the first sample data; and training the sample confidence prediction network based on the difference between the prediction confidence value and the confidence label to obtain a confidence prediction network. The confidence prediction network is additionally arranged and trained on the basis of classification prediction model training, so that the accuracy of a classification prediction result can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and in particular to an evaluation method, apparatus, device, medium, and program product for a classification prediction model. Background Art

[0002] Classification prediction is used to perform classification prediction on the test data under the target task, and perform the target task according to the classification prediction results.

[0003] In related technologies, a classification prediction model is trained using sample data to perform classification prediction on input data.

[0004] However, in related technologies, directly using the results output by the classification prediction model to execute corresponding tasks will lead to errors in the task execution results when there are deviations in the classification prediction results, thereby reducing the accuracy of task execution. Summary of the Invention

[0005] The present invention provides a method, apparatus, device, medium, and program product for evaluating a classification prediction model. The method can be used to evaluate the accuracy of the classification prediction results output by the classification prediction model by training a confidence prediction network, thereby improving the accuracy of the classification prediction. The technical solution is as follows:

[0006] In one aspect, a method for evaluating a classification prediction model is provided, the method comprising:

[0007] Acquire first sample data and a plurality of second sample data, wherein the plurality of second sample data are respectively annotated with a confidence value, where the confidence value is a probability output by a classification prediction model that represents the second sample data corresponds to the category;

[0008] Obtaining data association degrees between the plurality of second sample data and the first sample data respectively;

[0009] Based on the data association degrees respectively corresponding to the plurality of second sample data, the confidence values ​​respectively corresponding to the plurality of second sample data are fused to obtain a confidence label corresponding to the first sample data;

[0010] Performing confidence value prediction on the first sample data through a sample confidence prediction network to obtain a prediction confidence value corresponding to the first sample data;

[0011] Based on the difference between the prediction confidence value and the confidence label, the sample confidence prediction network is trained to obtain a confidence prediction network, which is used to evaluate the accuracy of the classification prediction result output after the classification prediction model performs classification prediction on the input data.

[0012] In another aspect, a device for evaluating a classification prediction model is provided, the device comprising:

[0013] an acquisition module, configured to acquire first sample data and a plurality of second sample data, wherein the plurality of second sample data are respectively annotated with a confidence value, wherein the confidence value is a probability output by a classification prediction model that indicates whether the second sample data corresponds to the category;

[0014] The acquisition module is further configured to acquire data association degrees between the plurality of second sample data and the first sample data respectively;

[0015] a fusion module, configured to fuse the confidence values ​​corresponding to the plurality of second sample data based on the data association degrees corresponding to the plurality of second sample data, to obtain a confidence label corresponding to the first sample data;

[0016] A prediction module, configured to predict a confidence value of the first sample data using a sample confidence prediction network to obtain a prediction confidence value corresponding to the first sample data;

[0017] A training module is used to train the sample confidence prediction network based on the difference between the prediction confidence value and the confidence label to obtain a confidence prediction network, and the confidence prediction network is used to evaluate the accuracy of the classification prediction result output after the classification prediction model performs classification prediction on the input data.

[0018] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the evaluation method of the classification prediction model as described in any of the above embodiments.

[0019] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an evaluation method for a classification prediction model as described in any of the above embodiments.

[0020] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the classification prediction model evaluation method described in any of the above embodiments.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0022] After obtaining first sample data and multiple second sample data labeled with confidence values, the confidence labels corresponding to the first sample data are obtained by obtaining the data associations of the multiple second sample data with the first sample data, and fusing the confidence values ​​corresponding to the multiple second sample data based on the data associations corresponding to the multiple second sample data. The sample confidence prediction network is trained based on the difference between the predicted confidence value and the confidence label obtained after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network based on the classification prediction model training, the accuracy of the classification prediction results can be improved. On the other hand, the confidence labels of the first sample data are calculated using the confidence values ​​labeled by the surrounding sample data to supervise the training of the sample confidence prediction network, which can improve the accuracy and efficiency of network training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0025] Figure 2 is a flowchart of an evaluation method for a classification prediction model provided by an exemplary embodiment of the present application;

[0026] Figure 3 is a flowchart of a method for evaluating a classification prediction model provided by another exemplary embodiment of the present application;

[0027] Figure 4 is a flowchart of a method for evaluating a classification prediction model provided by another exemplary embodiment of the present application;

[0028] Figure 5 is a flowchart of a method for evaluating a classification prediction model provided by another exemplary embodiment of the present application;

[0029] Figure 6 is a schematic diagram of an evaluation method for a classification prediction model provided by another exemplary embodiment of the present application;

[0030] Figure 7This is a structural block diagram of an evaluation device for a classification prediction model provided by an exemplary embodiment of the present application;

[0031] Figure 8 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of this application more clear, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0033] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on the quantity and execution order.

[0034] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0035] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI domains. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0037] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. Figure 1 As shown, the implementation environment includes a terminal 110, a server 120, and a communication network 130, wherein the terminal 110 and the server 120 are connected via the communication network 130. Optionally, the communication network 130 can be a wired network or a wireless network, which is not limited here.

[0038] In the embodiment of the present application, terminal 110 is used to send data to server 120. Optionally, a target application with classification prediction functionality is installed in terminal 110, which is not limited in this embodiment. Illustratively, the target application can be a traditional application, a cloud application, a mini-program or application module within a host application, or a web platform, which is not limited in this embodiment.

[0039] After acquiring the sample data, the terminal 110 sends the sample data to the server 120 for training the classification prediction model and the confidence prediction network. The sample data includes first sample data and multiple second sample data, and the multiple second sample data correspond to at least one category.

[0040] In some embodiments, after receiving the sample data, the server 120 predicts confidence values ​​for the plurality of second sample data using the trained classification prediction model, and outputs the probability of the corresponding category of the second sample data as the confidence value of the second sample data.

[0041] In some embodiments, data associations between multiple second sample data and the first sample data are obtained, and based on the data associations corresponding to the multiple second sample data, the confidence values ​​corresponding to the multiple second sample data are fused to obtain confidence labels corresponding to the first sample data.

[0042] In some embodiments, a confidence value prediction is performed on the first sample data through a sample confidence prediction network, and a predicted confidence value corresponding to the first sample data is output, thereby training the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label to obtain a confidence prediction network.

[0043] In some embodiments, after server 120 has trained the confidence prediction network, if a task is required, terminal 110 receives input data and sends the input data to server 120 for classification prediction. After receiving the input data, server 120 extracts the input feature representation corresponding to the input data, inputs the input feature representation into the classification prediction model, and outputs the classification prediction result corresponding to the input data. Simultaneously, the input feature representation is input into the confidence prediction network, and outputs the confidence value prediction result corresponding to the input data. The classification prediction result and the confidence value prediction result are fed back to terminal 110, where the confidence value prediction result is used to evaluate the accuracy of the classification prediction result.

[0044] In some optional embodiments, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, a smart car terminal, a smart speaker, an intelligent voice interaction device, an aircraft, etc., but is not limited thereto.

[0045] It is worth noting that server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0046] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool for on-demand, flexible, and convenient use. Cloud computing technology will become a crucial support. Backend services in technical network systems, such as those for video websites, image websites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Data of varying levels will be processed separately, requiring robust system support for all types of industry data, which can only be achieved through cloud computing. Alternatively, server 120 can also be implemented as a node in a blockchain system.

[0047] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions.

[0048] Combined with the above introduction and implementation environment, Figure 2 This is a flowchart of a classification prediction model evaluation method provided in an embodiment of the present application. The method can be applied to a terminal, a server, or both. The embodiment of the present application takes the application of the method to a server as an example for explanation. The method includes:

[0049] Step 210: Acquire first sample data and a plurality of second sample data.

[0050] The plurality of second sample data correspond to at least one category, and the plurality of second sample data are respectively marked with confidence values, where the confidence value is the probability output by the classification prediction model that represents the category corresponding to the second sample data.

[0051] Illustratively, the first sample data is sample data not labeled with a confidence value, and the second sample data is sample data labeled with a confidence value.

[0052] Optionally, the first sample data and the multiple second sample data are sample data obtained from the same data set, that is, the first sample data and the multiple second sample data are in the same batch; or, the first sample data and the multiple second sample data are obtained through different channels respectively.

[0053] Optionally, the first sample data and the second sample data are sample data of the same data type; or, the first sample data and the second sample data are sample data of different data types.

[0054] Optionally, the data type corresponding to the sample data includes at least one of text data, video frame data, image data, audio frame data, and the like.

[0055] Optionally, the first sample data and the second sample data belong to the same category of sample data, for example, the first sample data is an image of a road with an obstacle, and the second sample data is also an image of a road with an obstacle; or, the first sample data and the second sample data belong to different categories of sample data, for example, the first sample data is an image of foggy weather, and the second sample data is an image of snowy weather.

[0056] Optionally, multiple second sample data belong to the same category of sample data, for example, the second sample data a and the second sample data b are both road traffic jam images; or, multiple second sample data belong to different categories of sample data, for example, the second sample data c is a foggy weather image, and the second sample data d is a sunny day image.

[0057] The category refers to the category of data content contained in the sample data. For example, when the sample data is an image, the category refers to the category of image content contained in the image.

[0058] In some embodiments, the confidence value is a probability value of the category corresponding to the second sample data. For example, the second sample data category A is a traffic jam road image with a probability of 0.8, so the confidence value corresponding to the second sample data A is 0.8.

[0059] Schematically, the confidence value of the second sample data is obtained by prediction through a classification prediction model. The second sample data is input into a trained classification prediction model, and the prediction probabilities corresponding to the second sample data in multiple candidate categories are output, and the confidence value corresponding to the second sample data is determined from the multiple prediction probabilities.

[0060] Optionally, the confidence value may be determined by at least one of the following methods:

[0061] 1. After the classification prediction model outputs the predicted probabilities corresponding to the second sample data on multiple candidate categories, the highest predicted probability value is selected as the confidence value corresponding to the second sample data, and the candidate category corresponding to the confidence value is used as the category corresponding to the second sample data. For example, if the second sample data B is input into the classification prediction model and the outputs are prediction result 1: "Sunny road image, 0.7", prediction result 2: "Rainy road image, 0.2", and prediction result 3: "Fog road image, 0.1", 0.7 is used as the second sample data B.

[0062] 2. Pre-specify the category, and after the classification prediction model outputs the prediction probabilities corresponding to the second sample data in multiple candidate categories, the prediction probabilities corresponding to the specified category are used as the confidence values ​​corresponding to the second sample data. For example: the specified category is a road image with obstacles, and the classification prediction model outputs prediction result 1 corresponding to the second sample data C: "Road image with obstacles, 0.2", and prediction result 2: "Road image without obstacles, 0.8", then 0.2 is used as the confidence value corresponding to the second sample data.

[0063] It is worth noting that the above-mentioned method for determining the confidence value is only an illustrative example and is not limited to this embodiment of the present application.

[0064] Step 220: Obtain data correlation degrees between a plurality of second sample data and the first sample data.

[0065] Illustratively, the data association degree refers to the data association between the first sample data and a single second sample data.

[0066] Optionally, the data association includes at least one of the following types:

[0067] 1. Correlation: refers to the relationship between the data variables of the first sample data and the second sample data. The degree of association between the first sample data and the second sample data is measured by statistical methods, such as correlation coefficient, covariance, etc.

[0068] 2. Causal relationship: This refers to the relationship in which one data event causes another data event to occur, where the causal data event is called the cause and the resulting data event is called the effect. For example, if the first sample data is a "rear-end collision road condition map," the second sample data 1 is a "snowy weather map," and the second sample data 2 is a "sunny weather map," then the weather factor "snowy weather" that causes the rear-end collision has a greater impact than "sunny weather." Therefore, the causal relationship between the second sample data 1 and the first sample data is greater than the causal relationship between the second sample data 2 and the first sample data. Causal relationships can be trained using methods such as regression analysis, decision trees, and Bayesian networks to obtain a causal model, thereby predicting the causal relationship between two data sets.

[0069] 3. Similarity: This refers to the degree of similarity between the first sample data and the second sample data. It can be measured by comparing features, attributes, relationships, etc. The strength of the similarity can be used to measure the correlation between the data. If the similarity between the first sample data and the second sample data is strong, it means that the correlation between them is high.

[0070] It is worth noting that the above types of data association are only illustrative examples and are not limited to these in the embodiments of the present application.

[0071] Optionally, the data relevance is obtained by at least one of the following methods:

[0072] 1. Pre-training a feature extraction model to extract feature representations corresponding to the first sample data and the second sample data, and then determining the data correlation between the first sample data and the second sample data based on the difference between the feature representations. The data correlation between the sample data is obtained by extracting data features. Can simplify data association The acquisition process improves the efficiency of acquiring data relevance ;

[0073] 2. Obtaining a data association degree between the first sample data and the second sample data by comparing the differences between the data structure elements corresponding to the first sample data and the data structure elements corresponding to the second sample data. For example, when the first sample data and the second sample data are both images, the data association degree between the first sample data and the second sample data is obtained by comparing the grayscale values ​​of pixels in the first sample data and the second sample data. For another example, when the first sample data and the second sample data are both audio data, the data association degree between the first sample data and the second sample data is obtained by comparing the differences in phonemes (for example, sound quality, sound length, sound intensity, or timbre) in the first sample data and the second sample data. For another example, when the first sample data and the second sample data are both text data, the data association degree between the first sample data and the second sample data is obtained by comparing the characters in the first sample data and the second sample data. The data association degree between the sample data is obtained by comparing the differences in the data structure elements corresponding to the sample data. Can make When the sample data has different data types, the data structure elements corresponding to the data types are used for comparison, which improves Accuracy and acquisition efficiency of data relevance ;

[0074] 3. By setting reference sample data, by comparing a first data correlation between the first sample data and the reference sample data, and a second data correlation between the second sample data and the reference sample data, thereby obtaining a data correlation between the first sample data and the second sample data by comparing the first correlation and the second correlation. pass Compare with the same reference object, and obtain the difference between the first sample data and the second sample data according to the comparison result. Correlation, which can improve the accuracy and objectivity of correlation results .

[0075] It is worth noting that the above-mentioned method for obtaining data relevance is only an illustrative example and is not limited to this embodiment of the present application.

[0076] Among them, for the first data association acquisition method mentioned above, when the first sample data and the second sample data belong to different data types, the feature extraction model is used to map the first sample data and the second sample data to the same feature space, so as to obtain the feature representation under the same feature dimension, and then obtain the data association between the first sample data and the second sample data by comparing the feature similarity. For example: the first sample data is image data, and the second sample data is video frame data. The image data and the video frame data are mapped to the pixel point space, so as to obtain the first pixel point feature corresponding to the image data and the second pixel point feature corresponding to the video frame data, and then obtain the data association between the first sample data and the second sample data based on the similarity between the first pixel point feature and the second pixel point feature. By mapping sample data of different data types to the same feature space, the feature representation of the same dimension is extracted to perform similarity calculation. It can realize the acquisition of data correlation for sample data of different data types, thus improving the data correlation. Objectivity and accuracy.

[0077] Step 230 : Based on the data association degrees corresponding to the plurality of second sample data, the confidence values ​​corresponding to the plurality of second sample data are fused to obtain the confidence label corresponding to the first sample data.

[0078] In some embodiments, the confidence label corresponding to the first sample data refers to a confidence value obtained by fusing the data association degree and the confidence value corresponding to the second sample data.

[0079] Schematically, the confidence values ​​corresponding to the plurality of second sample data are weightedly fused using the data association degree corresponding to the second sample data as the weight, thereby obtaining the confidence value corresponding to the first sample data as the confidence label corresponding to the first sample data.

[0080] In principle, the fusion method includes at least one of the following methods:

[0081] 1. Multiply the data association degree corresponding to the second sample data by the confidence value corresponding to the second sample data to obtain the product result corresponding to the second sample data. Finally, calculate the average of the product results corresponding to multiple second sample data as the confidence label corresponding to the first sample data;

[0082] 2. Pre-set multiple different label thresholds, multiply the data association degree corresponding to the second sample data by the confidence value corresponding to the second sample data to obtain the product result corresponding to the second sample data, and finally calculate the average result of the product results corresponding to the multiple second sample data. Compare the average result with multiple label thresholds, and select the label threshold closest to the average result as the confidence label corresponding to the first sample data. By calculating After comparing the average value with the pre-set label threshold, the threshold is classified, which can make the final confidence mark Sign data regularly to reduce subsequent calculation pressure .

[0083] It is worth noting that the above-mentioned fusion method is only an illustrative example and is not limited to this embodiment of the present application.

[0084] Step 240: Confidence value prediction is performed on the first sample data through the sample confidence prediction network to obtain a prediction confidence value corresponding to the first sample data.

[0085] Illustratively, after the confidence labels corresponding to the first sample data are obtained by fusing the data association degrees and confidence values ​​corresponding to the plurality of second sample data, the sample confidence prediction network is supervised and trained using the confidence labels.

[0086] Optionally, the prediction confidence value may be obtained in at least one of the following ways:

[0087] 1. Use the sample confidence prediction network to predict the confidence value of the first sample data, and output the predicted probability values ​​corresponding to multiple candidate categories. The highest value among the predicted probability values ​​is used as the predicted confidence value;

[0088] 2. The first sample data corresponds to the first category. The first sample data and the first category are input into the sample confidence prediction network, and the prediction confidence value of the first sample data corresponding to the first category is output;

[0089] 3. Pre-specify the category, and after predicting the confidence value of the first sample data through the sample confidence prediction network, output the predicted probability values ​​corresponding to multiple candidate categories, among which the predicted probability value corresponding to the specified category is used as the predicted confidence value corresponding to the first sample data.

[0090] It is worth noting that the above-mentioned method for obtaining the prediction confidence value is only an illustrative example and is not limited to this embodiment of the present application.

[0091] Step 250 : Based on the difference between the prediction confidence value and the confidence label, the sample confidence prediction network is trained to obtain a confidence prediction network.

[0092] Among them, the confidence prediction network is used to evaluate the accuracy of the classification prediction results output by the classification prediction model after classifying and predicting the input data.

[0093] Schematically, when the confidence label corresponding to the first sample data is used to supervise the training of the sample confidence prediction network, the network parameters of the sample confidence prediction network are gradient adjusted by obtaining the difference between the predicted confidence value and the confidence label as the loss value, and finally a trained confidence prediction network is obtained.

[0094] In some embodiments, by pre-setting the loss function, substituting the predicted confidence value and the confidence label into the loss function, the target loss value is calculated, and the target loss value is used to perform gradient adjustment on the network parameters of the sample confidence prediction network, and finally a confidence prediction network is obtained.

[0095] Optionally, the loss function includes mean squared error loss function (Mean Squared Error), root mean squared error loss function (Root Mean Squared Error), mean absolute error loss function (Mean Absolute Error), logarithmic loss function (Log Loss), Hinge loss function: commonly used in classification problems such as support vector machine (SVM), used to calculate the interval of classification boundaries, cross entropy loss function (Cross-Entropy Loss), gradient descent loss function (Gradient Descent Loss) and other functions.

[0096] In some embodiments, after training, the confidence prediction network can be used to evaluate the accuracy of the classification prediction results output by the classification prediction model to be tested during the testing phase. In the testing phase, after obtaining test data, the classification prediction model performs classification prediction on the test data to obtain the classification prediction results corresponding to the test data. Furthermore, the confidence prediction network performs confidence prediction on the test data to obtain the confidence prediction results corresponding to the test data. The confidence prediction results are then combined to analyze the accuracy of the classification prediction results, thereby determining the test results of the classification prediction model.

[0097] Optionally, the classification prediction includes at least one of task types such as text classification prediction, image classification prediction, and audio classification prediction.

[0098] Schematically, the classification prediction results output by the classification prediction model include prediction labels corresponding to multiple candidate categories, and the prediction labels are used to determine the prediction categories corresponding to the classification prediction results. For example: after the classification prediction model performs classification prediction on data a, the output results are category 1: 0, category 2: 1, category 3: 0, where "0" indicates that it does not belong to the category and "1" indicates that it belongs to the category. Therefore, the prediction category obtained by the classification prediction model after performing classification prediction on data a is "category 2".

[0099] In an optional case, during the process of making confidence predictions on the test data through the confidence prediction network, the confidence prediction network outputs the prediction confidence value corresponding to the test data, thereby evaluating the prediction category of the classification prediction model according to the size of the prediction confidence value. If the prediction confidence value is greater than the preset confidence threshold, it is considered that the accuracy of the prediction category is high, otherwise the accuracy is low.

[0100] In another optional case, in the process of making confidence predictions on the test data through the confidence prediction network, the confidence prediction network outputs candidate confidence values ​​corresponding to multiple candidate categories of the test data, thereby matching the predicted category output by the classification prediction model with the multiple candidate categories output by the confidence prediction network, and taking the candidate confidence value corresponding to the matched target category as the predicted confidence value. If the predicted confidence value is greater than the preset confidence threshold, it is considered that the accuracy of the predicted category is high, otherwise the accuracy is low. For example: the confidence prediction network makes a confidence prediction on data a, and the confidence prediction results are category 1: 0.6, category 2: 0.3, and category 3: 0.1. Since the classification prediction model predicts the category of model a as category 2, the predicted confidence value is 0.3, and the preset confidence threshold is 0.7. The classification result accuracy of the current classification prediction model in the test phase is low, and the classification prediction model needs to continue training.

[0101] It is worth noting that the above-mentioned evaluation process of the classification prediction model using the confidence prediction network is the testing phase. The application phase is the same as the testing phase and is not limited to this.

[0102] In summary, the evaluation method of the classification prediction model provided in the embodiment of the present application, after obtaining the first sample data and a plurality of second sample data labeled with confidence values, obtains the data correlation between the plurality of second sample data and the first sample data, and fuses the confidence values ​​corresponding to the plurality of second sample data based on the data correlation corresponding to the plurality of second sample data, obtains the confidence label corresponding to the first sample data, and trains the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label output after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network on the basis of the classification prediction model training, the accuracy of the classification prediction results can be improved. On the other hand, the confidence values ​​labeled by the surrounding sample data are used to calculate the confidence label of the first sample data, thereby supervising the training of the sample confidence prediction network, which can improve the accuracy and efficiency of network training.

[0103] In an optional embodiment, the fusion is described in detail, schematically, please refer to Figure 3 , which shows a flow chart of an evaluation method of a classification prediction model provided by an exemplary embodiment of the present application, that is, step 220 also includes step 221, and step 230 also includes step 231, schematically, as shown Figure 3 As shown, the method includes the following steps.

[0104] Step 221 : Perform feature extraction on each of the plurality of second sample data to obtain second feature representations corresponding to each of the plurality of second sample data.

[0105] In some embodiments, the data association degree includes a feature distance between a first feature representation corresponding to the first sample data and a second feature representation corresponding to the second sample data.

[0106] Schematically, the data association degree is embodied as the feature similarity between the first sample data and the second sample data as an example for explanation, wherein the feature similarity can also be called feature distance, and the shorter the feature distance, the higher the feature similarity.

[0107] Schematically, by comparing the feature similarity between the first sample data and the second sample data, the data association between the first sample data and the second sample data is determined based on the feature similarity, wherein the higher the feature similarity, the higher the data association between the first sample data and the second sample data. That is, the similarity relationship between the feature representations is used as a standard for measuring data association. Ability to intuitively and clearly obtain the data relationship between two sample data The method of mining data correlation at the feature level can improve the accuracy and objectivity of correlation comparison. .

[0108] In some embodiments, feature extraction is performed on the first sample data using a feature extraction model to obtain a first feature representation, and feature extraction is performed on multiple second sample data using the feature extraction model to obtain second feature representations corresponding to the multiple second sample data.

[0109] In this embodiment, the feature extraction model obtained by pre-training is used to extract features from the first sample data, thereby obtaining a first feature representation corresponding to the first sample data, and the feature extraction model is used to extract features from multiple second sample data, thereby obtaining second feature representations corresponding to the multiple second sample data. , which can improve the extraction efficiency of feature representation .

[0110] Next, the training process of the feature extraction model is described in detail.

[0111] In some embodiments, third sample data is obtained, and the third sample data is annotated with a feature representation label; the third sample data is input into a sample extraction model, and a predicted feature representation corresponding to the third sample data is output; based on the difference between the predicted feature representation and the feature representation label, the sample extraction model is trained to obtain a feature extraction model.

[0112] During the training process, firstly, third sample data for training the sample extraction model is obtained, wherein the third sample data is annotated with corresponding feature representation labels for supervised training of the sample extraction model.

[0113] The third sample data is input into the sample extraction model, and the feature extraction is performed on the third sample data through the sample extraction model to obtain the predicted feature representation corresponding to the third sample data. Based on the difference between the predicted feature representation and the feature representation label, a loss value is obtained, and the model parameters of the sample extraction model are gradient adjusted using the loss value to finally obtain a trained feature extraction model.

[0114] In this embodiment, the training goal of the sample extraction model is to learn a feature mapping function: h:X→H, where X represents the input data and H represents the feature space, that is, mapping the input data into the feature space to obtain the feature representation corresponding to the data.

[0115] In this embodiment, the cross entropy loss function is obtained in advance, and the predicted feature representation and the feature representation label are substituted into the cross entropy loss function to obtain the cross entropy loss value. The model parameters of the sample extraction model are adjusted with minimizing the cross entropy loss value as the adjustment target. When the cross entropy loss value reaches a preset loss threshold, the feature extraction model is considered to be trained.

[0116] Optionally, the model parameters of the sample extraction model are adjusted using at least one of the model parameter adjustment algorithms such as stochastic gradient descent, momentum method, adaptive method, root mean square propagation method, etc.

[0117] Step 231 : Based on the feature distances between the plurality of second feature representations and the first feature representation, the confidence values ​​corresponding to the plurality of second sample data are fused to obtain the confidence label corresponding to the first sample data.

[0118] Schematically, after obtaining the first feature representation corresponding to the first sample data and the second feature representations corresponding to the plurality of second sample data, the feature distances between the plurality of second feature representations and the first feature representation are obtained, and the feature distances are used as the weight values ​​corresponding to the second sample data, and the weight values ​​corresponding to the plurality of second sample data are used to perform weighted fusion on the confidence values ​​corresponding to the plurality of second sample data, thereby obtaining the confidence value fusion result as the confidence label corresponding to the first sample data. That is, by obtaining the feature distance between the first sample data and the second sample data as the weight for weighted fusion of confidence values, A second data set having a different degree of association with the first sample data The sample data has different effects on the confidence label of the final first sample data, weakening the second sample with low data correlation The impact of data on the final confidence label, enhancing the confidence of the second sample data with high data correlation to the first sample data The influence of the heart label, thereby improving the accuracy of the confidence label .

[0119] In some embodiments, based on the feature distances between the multiple second feature representations and the first feature representation, weight values ​​corresponding to the multiple second feature representations are obtained; based on the weight values ​​corresponding to the multiple second feature representations, weighted averaging is performed on the confidence values ​​corresponding to the multiple second sample data to obtain the confidence label corresponding to the first sample data.

[0120] In this embodiment, after extracting the first feature representation corresponding to the first sample data and the second feature representation corresponding to the second sample data, the feature distance between the first feature representation and the second feature representation is calculated using a distance calculation formula. For details, please refer to Formula 1.

[0121] Formula 1: d(x,x′)=Dis(h(x),h(x′))

[0122] Wherein, d represents a distance metric function (including either Euclidean distance or cosine distance), h(x) represents a first feature representation corresponding to the first sample data, and h(x′) represents a second feature representation corresponding to the second sample data.

[0123] In this embodiment, after the characteristic distance between the first characteristic representation and the second characteristic representation is calculated, the characteristic distance is normalized to obtain the weight value corresponding to the second sample data. For details, please refer to Formula 2.

[0124] Formula 2: w(x,x′) = exp(-d(x,x′) / σ) / ∑ x′∈B(x) exp(-d(x,x′) / σ)

[0125] Where w(x,x′) represents the weight value, d(x,x′) represents the feature distance, and σ is an adjustable parameter used to control the scale of the weight distribution.

[0126] In this embodiment, after obtaining the weight values ​​corresponding to the multiple second sample data, the confidence values ​​corresponding to the multiple second sample data are weighted averaged according to the weight values ​​corresponding to the multiple second sample data, so as to obtain the confidence label corresponding to the first sample data. For details, please refer to Formula 3.

[0127] Formula 3: c(x) = ∑ x′∈B(x) w(x,x′)*conf(x′)

[0128] Wherein, B(x) represents the second sample data set. In this embodiment, the second sample data set and the first sample data belong to the sample data in the same batch, w(x, x′) represents the weight value, and conf(x′) represents the confidence value corresponding to the second sample data.

[0129] In this embodiment, the characteristic distance between the first sample data and the second sample data is used as a weight value to perform weighted average processing on the confidence values ​​corresponding to the plurality of second sample data. , which can improve the first sample data corresponding The confidence label accuracy .

[0130] In some embodiments, a preset distance condition is obtained; based on the preset distance condition, multiple feature distances are distance screened to obtain at least two target feature distances, where the target feature distance is a feature distance that meets the preset distance condition, and the target feature distance is the feature distance between the screened second sample data and the first sample data; based on the at least two target feature distances, weighted averaging is performed on the confidence values ​​corresponding to the at least two screened second sample data to obtain a confidence label corresponding to the first sample data.

[0131] In this embodiment, in addition to weighting the confidence values ​​corresponding to all second sample data, the second sample data may be screened and then the confidence values ​​corresponding to some second sample data may be weighted to obtain confidence labels corresponding to the first sample data.

[0132] In this embodiment, a distance condition is pre-set. After calculating the characteristic distances between the plurality of second characteristic representations and the first characteristic representation, the plurality of characteristic distances are compared with the distance condition, thereby screening out at least two target characteristic distances that meet the distance condition. The at least two target characteristic distances are characteristic distances corresponding to the at least two filtered second sample data. The at least two target characteristic distances are normalized to obtain at least two weight values. Based on the at least two weight values, the confidence values ​​corresponding to the filtered second sample data are weighted averaged to obtain the confidence label corresponding to the first sample data. For example, the distance condition is set to be less than 0.6. Therefore, the characteristic distance less than 0.6 is selected from the plurality of characteristic distances as the target characteristic distance, and the weight value is normalized to obtain the weight value. The confidence value corresponding to the second sample data with a characteristic distance less than 0.6 is weighted averaged in combination with the weight value to obtain the confidence label corresponding to the first sample data. That is, by setting the distance condition to screen the plurality of second sample data, the second sample data with a low data correlation with the first sample data and a low contribution to the confidence label can be screened out. Not only Improves the accuracy of confidence labels and saves server data computing overhead .

[0133] In summary, the evaluation method of the classification prediction model provided in the embodiment of the present application, after obtaining the first sample data and a plurality of second sample data labeled with confidence values, obtains the data correlation between the plurality of second sample data and the first sample data, and fuses the confidence values ​​corresponding to the plurality of second sample data based on the data correlation corresponding to the plurality of second sample data, obtains the confidence label corresponding to the first sample data, and trains the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label output after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network on the basis of the classification prediction model training, the accuracy of the classification prediction results can be improved. On the other hand, the confidence values ​​labeled by the surrounding sample data are used to calculate the confidence label of the first sample data, thereby supervising the training of the sample confidence prediction network, which can improve the accuracy and efficiency of network training.

[0134] In an optional embodiment, the training process of the sample confidence prediction network is described in detail. For illustration, please refer to Figure 4 , which shows a flow chart of an evaluation method of a classification prediction model provided by an exemplary embodiment of the present application, that is, step 210 also includes steps 2101 and 2102, and step 240 includes steps 241 to 243, as shown in FIG. Figure 4 The method includes the following steps.

[0135] Step 2101 : perform classification prediction on the plurality of second sample data respectively through the classification prediction model to obtain classification prediction results corresponding to the plurality of second sample data respectively.

[0136] The classification prediction result includes multiple prediction categories and candidate confidence values ​​corresponding to the multiple prediction categories.

[0137] Illustratively, classification prediction is performed on a plurality of second sample data respectively through a classification prediction model obtained through pre-training, thereby obtaining a classification prediction result corresponding to each second sample data.

[0138] Among them, the classification prediction result includes the candidate confidence values ​​corresponding to the second sample data in multiple candidate categories. For example: the classification prediction model performs classification prediction on sample data a to obtain the classification prediction result corresponding to sample data a, wherein the classification prediction result includes category 1: 0.2, category 2: 0.7; category 3: 0.1.

[0139] Next, the training process of the classification prediction model is described in detail.

[0140] Illustratively, third sample data is obtained, and the third sample data is annotated with a category label, where the category label refers to the category corresponding to the third sample data. The third sample data is input into a sample classification model, and a classification prediction result corresponding to the third sample data is output. Based on the difference between the category label and the classification prediction result, the sample classification model is trained to obtain a classification prediction model.

[0141] In an example, the probability of category a corresponding to the category label of the third sample data is 1. The classification prediction result output by the sample classification model includes category a: 0.6 and category b: 0.4. Therefore, the classification prediction result is category a: 0.6. By obtaining the loss function in advance, the probability value of the classification prediction result and the probability value corresponding to the category label are substituted into the loss function to obtain the loss value. The loss value is used to perform gradient adjustment on the model parameters of the sample classification model, and finally a trained classification prediction model is obtained.

[0142] In this embodiment, the classification prediction model and the feature extraction model serve as two sub-models in the classification network. Therefore, the training process of the feature extraction model and the classification prediction model can be completed synchronously. For details, please refer to the following formula 4.

[0143] Formula 4:

[0144] Among them, L h Represents the loss function corresponding to the sample extraction model, L f Represents the loss function corresponding to the sample classification model. The sample extraction model and the sample classification model are trained by calculating the minimum value (minimize) of the loss function corresponding to the sample extraction model and the loss function corresponding to the sample classification model.

[0145] Optionally, the classification prediction model may adopt at least one of the model types such as support vector machine (SVM), logistic regression, naive Bayes, decision tree, random forest and neural network.

[0146] Step 2102: Taking the candidate confidence value that meets the preset confidence value condition as the confidence value corresponding to the second sample data.

[0147] Schematically, since the classification prediction results output by the classification prediction model include candidate confidence values ​​corresponding to multiple candidate categories, a confidence value condition is set in advance, and the candidate confidence value that meets the confidence value condition among the multiple candidate confidence values ​​is used as the confidence value corresponding to the second sample data.

[0148] In this embodiment, the highest candidate confidence value is used as the confidence value corresponding to the second sample data.

[0149] In this embodiment, the confidence value of the second sample data is obtained by predicting the classification prediction model. On the one hand, it can avoid No need to manually label confidence values, which improves the efficiency of obtaining confidence values. On the other hand, the candidate confidence values ​​that meet the confidence value conditions are selected as The confidence value of the second sample data can ensure the accuracy of the confidence value .

[0150] Step 241: Obtain a preset loss function.

[0151] Among them, the preset loss function is used to calculate the difference between the predicted confidence value and the confidence label.

[0152] Schematically, a loss function is pre-selected to calculate the difference between the predicted confidence value and the confidence label.

[0153] In this embodiment, the regression mean square error loss function is selected as the preset loss function.

[0154] Step 242: Obtain a target loss value between the predicted confidence value and the confidence label using a preset loss function.

[0155] Instructively, the predicted confidence value and confidence label are substituted into the preset loss function, and the mean square error result is output as the target loss value. For details, please refer to Formula 5.

[0156] Formula 5:

[0157] Where N represents the number of data of the second sample data, conf(x i ) represents the prediction confidence value, c(x i ) indicates confidence labels.

[0158] Step 243: Train the sample confidence prediction model based on the target loss value to obtain a confidence prediction network.

[0159] Indicatively, according to the target loss value, the sample confidence prediction model is trained using stochastic gradient descent to finally obtain the confidence prediction network.

[0160] In this embodiment, The sample confidence prediction network is trained by predicting the difference between the confidence value and the confidence label. The training method can improve the training accuracy of the confidence prediction network .

[0161] In some embodiments, input data corresponding to a target task is obtained, and the target task is labeled with a confidence threshold; an input feature representation corresponding to the input data is extracted; the input feature representation is input into a classification prediction model, and a classification prediction result corresponding to the input data is output; the input feature representation is input into a confidence prediction network, and a confidence value prediction result of the prediction category corresponding to the input data is output; in response to the confidence value prediction result reaching the confidence threshold, the target task is executed based on the classification prediction result.

[0162] For example, in the application phase, when a target task is required, the input data corresponding to the target task is obtained. Feature extraction is performed on the input data using a feature extraction model to obtain an input feature representation corresponding to the input data. Classification prediction is performed on the input feature representation using a classification prediction model, and the classification prediction result corresponding to the input data is output. The input feature representation is then fed into a confidence prediction network, which outputs a confidence value prediction result corresponding to the input data. If the confidence value prediction result reaches the pre-set confidence threshold for the target task, the accuracy of the classification prediction result output by the current classification prediction model is reliable. Therefore, the target task is executed based on the classification prediction result.

[0163] In this embodiment, The confidence prediction model is used to evaluate the accuracy of the classification prediction results output by the classification prediction model. Degree, thereby improving the accuracy of task execution .

[0164] In summary, the evaluation method of the classification prediction model provided in the embodiment of the present application, after obtaining the first sample data and a plurality of second sample data labeled with confidence values, obtains the data correlation between the plurality of second sample data and the first sample data, and fuses the confidence values ​​corresponding to the plurality of second sample data based on the data correlation corresponding to the plurality of second sample data, obtains the confidence label corresponding to the first sample data, and trains the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label output after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network on the basis of the classification prediction model training, the accuracy of the classification prediction results can be improved. On the other hand, the confidence values ​​labeled by the surrounding sample data are used to calculate the confidence label of the first sample data, thereby supervising the training of the sample confidence prediction network, which can improve the accuracy and efficiency of network training.

[0165] In some embodiments, the process of determining the confidence value of the second sample data whose confidence value cannot be determined by the classification prediction result is described in detail. For example, please refer to Figure 5 , which shows a flow chart of an evaluation method of a classification prediction model provided by an exemplary embodiment of the present application, such as Figure 5 As shown, the method includes the following steps.

[0166] Step 510 : In response to the difference between multiple candidate confidence values ​​in the classification prediction result of the i-th second sample data being less than a preset confidence value difference, obtaining data associations between other second sample data and the i-th second sample data.

[0167] Wherein, i is a positive integer.

[0168] Schematically, when a classification prediction model is used to perform classification prediction on multiple second sample data, since the classification prediction results output by each second sample data include multiple candidate categories and candidate confidence values ​​corresponding to the multiple candidate categories, if the difference between the multiple candidate confidence values ​​in the classification prediction result of the i-th second sample data is too small, the confidence value corresponding to the i-th second sample data cannot be determined based on the classification prediction result.

[0169] In one example, when a classification prediction model is used to perform classification prediction on data z, if the classification prediction results output by data z are category a: 0.5 and category b: 0.5, and the prediction probabilities of the two categories are the same, then the confidence value corresponding to data z cannot be determined.

[0170] Step 520 : In response to the data correlation between the kth second sample data and the ith second sample data reaching a preset correlation threshold, the confidence value corresponding to the kth second sample data is used as the confidence value corresponding to the ith second sample data.

[0171] Wherein, k is a positive integer.

[0172] Illustratively, when the confidence value corresponding to the i-th second sample data cannot be obtained based on the classification prediction result, the feature similarity between the i-th second sample data and other second sample data is obtained to determine the k-th second sample data with a higher feature similarity to the i-th second sample data, and the confidence value corresponding to the k-th second sample data is used as the confidence value corresponding to the i-th second sample data.

[0173] In this embodiment, for the second sample data whose confidence value cannot be determined based on the classification prediction result, the feature similarity between the second sample data and other second sample data is compared, so that the confidence value of the second sample data with higher feature similarity is directly used as the confidence value corresponding to the above second sample data. Ensure the confidence of the second sample data The accuracy of the value can also avoid the situation where the second sample data is not available due to the inability to determine the confidence value based on the classification prediction result. The method is applied to the training process of the confidence prediction network, resulting in a waste of data .

[0174] In summary, the evaluation method of the classification prediction model provided in the embodiment of the present application, after obtaining the first sample data and a plurality of second sample data labeled with confidence values, obtains the data correlation between the plurality of second sample data and the first sample data, and fuses the confidence values ​​corresponding to the plurality of second sample data based on the data correlation corresponding to the plurality of second sample data, obtains the confidence label corresponding to the first sample data, and trains the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label output after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network on the basis of the classification prediction model training, the accuracy of the classification prediction results can be improved. On the other hand, the confidence values ​​labeled by the surrounding sample data are used to calculate the confidence label of the first sample data, thereby supervising the training of the sample confidence prediction network, which can improve the accuracy and efficiency of network training.

[0175] Optionally, the confidence prediction network can be applied to a variety of different scenarios, as follows:

[0176] (1) Medical scenario: After obtaining a pathological image, the image feature representation corresponding to the DeDa pathological image is extracted through the feature extraction model, and the image feature representation is classified and predicted through the disease classification model to obtain the disease classification result corresponding to the pathological image. The confidence value of the image feature representation is predicted through the confidence prediction network to obtain the predicted confidence value corresponding to the image feature representation. If the predicted confidence value meets the preset confidence value threshold, the disease classification result is considered accurate and the disease classification result is used as the disease diagnosis result corresponding to the pathological image.

[0177] (2) Video recommendation: With authorization, obtain the video content that the target account has watched in the historical time period, extract the image feature representation corresponding to the specified video frame in the video content, perform classification prediction on the image feature representation through the video classification model, and obtain the video classification result corresponding to the image feature representation. Perform confidence value prediction on the image feature representation through the confidence prediction network, and obtain the confidence value prediction result corresponding to the image feature representation. If the predicted confidence value meets the preset confidence value threshold, the video classification result is considered accurate. Therefore, based on the video category corresponding to the video classification result, other video content under the video category is recommended to the target account.

[0178] Below, the autonomous driving scenario is explained in detail.

[0179] For illustration, please refer to Figure 6 , which shows a schematic diagram of an evaluation method of a classification prediction model provided by an exemplary embodiment of the present application, such as Figure 6 As shown, this method is applied to an intelligent driving scenario as an example.

[0180] During the driving of the vehicle, road image data 610 is obtained, wherein the road image data 610 refers to image data obtained by automatically photographing the travel road by the central control server corresponding to the vehicle.

[0181] The central control server includes a classification network 620 and a confidence prediction network 630. Classification network 620 further includes a feature extractor 621 and a classification prediction model 622. Feature extractor 621 extracts features from road image data 610 to obtain an image feature representation 601 corresponding to road image data 610. Classification prediction model 622 then performs road condition classification prediction on image feature representation 601 to obtain a road condition prediction result 602 corresponding to road image data 610. Road condition prediction result 602 is implemented as "obstacle present: 1; no obstacle: 0."

[0182] The confidence prediction network 630 performs confidence prediction on the image feature representation 601 to obtain a confidence prediction result 603 corresponding to the image feature representation 601 , wherein the confidence prediction result 603 is implemented as “with obstacle: 0.7; without obstacle: 0.3”.

[0183] In addition, the central control server pre-sets the confidence value threshold to 0.6. Therefore, the confidence value corresponding to the category "there is an obstacle" is 0.7, which is greater than the confidence value threshold of 0.6. The central control server determines that there is an obstacle ahead on the current road and automatically issues an obstacle warning to remind the driver that there is an obstacle on the road ahead.

[0184] Below, the training process of the feature extractor, classification prediction model, and confidence prediction network is described in detail.

[0185] Feature Extractor

[0186] First, a feature extractor is pre-trained to extract useful feature representations from input samples. This feature extractor can be a deep neural network (such as a convolutional neural network or a recurrent neural network) or another type of machine learning model. The feature extractor is trained using a training dataset to minimize a loss function, such as the cross-entropy loss.

[0187] Classification prediction model

[0188] After pre-training the feature extractor, we need to pre-train a classifier to classify input samples based on the extracted feature representations. This classifier can be a logistic regression, support vector machine, or other machine learning model. The classifier is trained using the training dataset to minimize a loss function, such as cross-entropy loss.

[0189] For each training example, we can calculate the confidence estimate as follows:

[0190] 1. Use the pre-trained feature extractor to calculate the feature representation distance of other samples in the same batch as the first sample data.

[0191] 2. Calculate weights based on feature representation distance.

[0192] 3. Use the weights to perform weighted averaging on the confidence values ​​of the second sample data to obtain the confidence estimate of the first sample data.

[0193] Confidence prediction network

[0194] The first step is to build a confidence estimation network

[0195] In this stage, we will build a confidence estimation network to estimate the confidence value of the input sample. The confidence estimation network can be a simple multilayer perceptron (MLP) or another type of neural network. The input of the confidence estimation network is the feature representation extracted by the feature extractor, and the output is the predicted confidence value of the first sample data. During the training process, we will fix the feature extractor and train only the confidence estimation network.

[0196] The second step is to train the confidence estimation network

[0197] To train the confidence estimation network, we will use the regression mean square error (MSE) loss function to constrain the predicted confidence value of the first sample data to be the same as the estimated confidence value. The confidence estimation network is trained using stochastic gradient descent (SGD) or other optimization algorithms.

[0198] In summary, the evaluation method of the classification prediction model provided in the embodiment of the present application, after obtaining the first sample data and a plurality of second sample data labeled with confidence values, obtains the data correlation between the plurality of second sample data and the first sample data, and fuses the confidence values ​​corresponding to the plurality of second sample data based on the data correlation corresponding to the plurality of second sample data, obtains the confidence label corresponding to the first sample data, and trains the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label output after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network on the basis of the classification prediction model training, the accuracy of the classification prediction results can be improved. On the other hand, the confidence values ​​labeled by the surrounding sample data are used to calculate the confidence label of the first sample data, thereby supervising the training of the sample confidence prediction network, which can improve the accuracy and efficiency of network training.

[0199] The solution provided in this application significantly reduces computational complexity while maintaining confidence estimation accuracy. This means that autonomous driving systems can more quickly estimate confidence in prediction results, enabling faster decision-making. Furthermore, this method reduces the system's computational burden, improving overall performance and freeing up resources for other critical tasks. Autonomous driving systems utilizing this patented solution can operate more stably and efficiently in complex traffic environments. For example, when the system predicts the presence of a pedestrian or other obstacle ahead, it can immediately estimate the confidence of the prediction. If the confidence is high, it can take appropriate evasive action; conversely, if the confidence is low, the system can choose to continue observing to obtain more accurate information. This decision-making process helps improve the safety and reliability of autonomous driving systems. This invention patent has broad prospects for practical application, particularly in real-time scenarios such as autonomous driving. By employing a proxy confidence estimation network solution, the system's real-time performance and the reliability of its prediction results can be significantly improved, providing strong support for achieving safer and more efficient autonomous driving.

[0200] Figure 7 This is a structural block diagram of an evaluation device for a classification prediction model provided by an exemplary embodiment of the present application. Figure 7 As shown, the device includes the following parts:

[0201] An acquisition module 710 is configured to acquire first sample data and a plurality of second sample data, wherein the plurality of second sample data are respectively annotated with a confidence value, where the confidence value is a probability output by a classification prediction model that indicates whether the second sample data corresponds to the category;

[0202] The acquisition module 710 is further configured to acquire data association degrees between the plurality of second sample data and the first sample data respectively;

[0203] A fusion module 720 is configured to fuse the confidence values ​​corresponding to the plurality of second sample data based on the data association degrees corresponding to the plurality of second sample data to obtain a confidence label corresponding to the first sample data;

[0204] A prediction module 730 is configured to perform confidence value prediction on the first sample data using a sample confidence prediction network to obtain a prediction confidence value corresponding to the first sample data;

[0205] The training module 740 is used to train the sample confidence prediction network based on the difference between the prediction confidence value and the confidence label to obtain a confidence prediction network, and the confidence prediction network is used to evaluate the accuracy of the classification prediction result output after the classification prediction model performs classification prediction on the input data.

[0206] In some embodiments, the data association degree includes a feature distance between a first feature representation corresponding to the first sample data and a second feature representation corresponding to the second sample data;

[0207] The fusion module 720 is configured to fuse the confidence values ​​corresponding to the plurality of second sample data based on the feature distances between the plurality of second feature representations and the first feature representation, to obtain the confidence label corresponding to the first sample data.

[0208] In some embodiments, the fusion module 720 is used to obtain weight values ​​corresponding to the multiple second feature representations based on the feature distances between the multiple second feature representations and the first feature representation; and perform weighted averaging processing on the confidence values ​​corresponding to the multiple second sample data based on the weight values ​​corresponding to the multiple second feature representations to obtain confidence labels corresponding to the first sample data.

[0209] In some embodiments, the fusion module 720 is used to obtain a preset distance condition; perform distance screening on multiple feature distances based on the preset distance condition to obtain at least two target feature distances, where the target feature distance is a feature distance that meets the preset distance condition, and the target feature distance is the feature distance between the filtered second sample data and the first sample data; perform weighted averaging processing on the confidence values ​​corresponding to the at least two filtered second sample data based on at least two target feature distances to obtain a confidence label corresponding to the first sample data.

[0210] In some embodiments, the training module 740 is used to obtain a preset loss function, which is used to calculate the difference between the predicted confidence value and the confidence label; obtain the target loss value between the predicted confidence value and the confidence label through the preset loss function; and train the sample confidence prediction model based on the target loss value to obtain the confidence prediction network.

[0211] In some embodiments, the training module 740 is used to substitute the prediction confidence value and the confidence label into the preset loss function, and output the mean square error result as the target loss value.

[0212] In some embodiments, the acquisition module 710 is used to perform classification prediction on the multiple second sample data respectively through the classification prediction model to obtain classification prediction results corresponding to the multiple second sample data respectively, and the classification prediction results include multiple prediction categories and candidate confidence values ​​corresponding to the multiple prediction categories respectively; the candidate confidence value that meets the preset confidence value conditions is used as the confidence value corresponding to the second sample data.

[0213] In some embodiments, the acquisition module 710 is further configured to, in response to a difference between a plurality of candidate confidence values ​​in the classification prediction result of the i-th second sample data being less than a preset confidence value difference, acquire data association degrees between other second sample data and the i-th second sample data, where i is a positive integer;

[0214] In response to the data correlation between the kth second sample data and the ith second sample data reaching a preset correlation threshold, the confidence value corresponding to the kth second sample data is used as the confidence value corresponding to the ith second sample data, where k is a positive integer.

[0215] In some embodiments, the acquisition module 710 is used to perform feature extraction on the first sample data through a feature extraction model to obtain the first feature representation, and to perform feature extraction on the multiple second sample data respectively through the feature extraction model to obtain second feature representations corresponding to the multiple second sample data respectively.

[0216] In some embodiments, the acquisition module 710 is used to obtain third sample data, where the third sample data is marked with a feature representation label; input the third sample data into a sample extraction model, and output a predicted feature representation corresponding to the third sample data; based on the difference between the predicted feature representation and the feature representation label, train the sample extraction model to obtain the feature extraction model.

[0217] In some embodiments, the training module 740 is used to obtain input data corresponding to a target task, wherein the target task is marked with a confidence threshold; extract an input feature representation corresponding to the input data; input the input feature representation into the classification prediction model, and output a classification prediction result corresponding to the input data; and input the input feature representation into the confidence prediction network, and output a confidence value prediction result of the input data corresponding to the prediction category; in response to the confidence value prediction result reaching the confidence threshold, execute the target task based on the classification prediction result.

[0218] In summary, the evaluation device of the classification prediction model provided by the embodiment of the present application, in summary, the evaluation method of the classification prediction model provided by the embodiment of the present application, after obtaining the first sample data and a plurality of second sample data labeled with confidence values, obtains the data correlation between the plurality of second sample data and the first sample data, and fuses the confidence values ​​corresponding to the plurality of second sample data based on the data correlation corresponding to the plurality of second sample data, obtains the confidence label corresponding to the first sample data, and trains the sample confidence prediction network based on the difference between the predicted confidence value and the confidence label output after predicting the confidence value of the first sample data through the sample confidence prediction network, thereby obtaining a confidence prediction network for evaluating the classification prediction accuracy of the classification prediction model. That is, on the one hand, by adding and training a confidence prediction network on the basis of the classification prediction model training, the accuracy of the classification prediction result can be improved. On the other hand, the confidence value labeled by the surrounding sample data is used to calculate the confidence label of the first sample data, thereby supervising the training of the sample confidence prediction network, which can improve the network training accuracy and training efficiency.

[0219] It should be noted that the evaluation device for the classification prediction model provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the evaluation device for the classification prediction model provided in the above embodiment and the evaluation method embodiment of the classification prediction model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0220] Figure 8The following is a block diagram of a computer device 800 according to an exemplary embodiment of the present application. The computer device 800 may be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. The computer device 800 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0221] Typically, the computer device 800 includes a processor 801 and a memory 802 .

[0222] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0223] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the training method and data classification method of the classification model provided in the method embodiment of the present application.

[0224] In some embodiments, the computer device 800 may optionally include other components, which those skilled in the art will appreciate. Figure 8 The structure shown in the figure does not constitute a limitation on the computer device 800, and the computer device 800 may include more or fewer components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0225] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, which can be a computer-readable storage medium included in the memory in the above embodiments; or a separate computer-readable storage medium that is not installed in the terminal. The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the classification model training method and data classification method described in any of the above embodiments.

[0226] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0227] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0228] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for evaluating a classification prediction model, characterized in that: The method comprises: Obtaining first sample data and a plurality of second sample data, wherein the plurality of second sample data are respectively annotated with confidence values, where the confidence values ​​are probabilities output by a classification prediction model that characterize the categories corresponding to the second sample data; Obtaining data association degrees between the plurality of second sample data and the first sample data respectively; Based on the data association degrees respectively corresponding to the plurality of second sample data, the confidence values ​​respectively corresponding to the plurality of second sample data are fused to obtain a confidence label corresponding to the first sample data; Performing confidence value prediction on the first sample data through a sample confidence prediction network to obtain a prediction confidence value corresponding to the first sample data; Based on the difference between the prediction confidence value and the confidence label, the sample confidence prediction network is trained to obtain a confidence prediction network, which is used to evaluate the accuracy of the classification prediction result output after the classification prediction model performs classification prediction on the input data.

2. The method according to claim 1, characterized in that The data association degree includes a feature distance between a first feature representation corresponding to the first sample data and a second feature representation corresponding to the second sample data; The step of fusing the confidence values ​​corresponding to the plurality of second sample data based on the data association degrees corresponding to the plurality of second sample data to obtain the confidence label corresponding to the first sample data includes: The confidence values ​​corresponding to the plurality of second sample data are fused based on the feature distances between the plurality of second feature representations and the first feature representation, to obtain the confidence label corresponding to the first sample data.

3. The method according to claim 2, characterized in that The fusing the confidence values ​​corresponding to the plurality of second sample data based on the feature distances between the plurality of second feature representations and the first feature representation to obtain the confidence label corresponding to the first sample data includes: Obtaining weight values ​​corresponding to the plurality of second feature representations based on feature distances between the plurality of second feature representations and the first feature representation; Based on the weight values ​​corresponding to the multiple second feature representations, weighted averaging processing is performed on the confidence values ​​corresponding to the multiple second sample data to obtain a confidence label corresponding to the first sample data.

4. The method according to claim 2, characterized in that The step of fusing the confidence values ​​corresponding to the plurality of second sample data based on the data association degrees corresponding to the plurality of second sample data to obtain the confidence label corresponding to the first sample data includes: Get the preset distance condition; Performing distance screening on multiple characteristic distances based on the preset distance condition to obtain at least two target characteristic distances, wherein the target characteristic distance is a characteristic distance that meets the preset distance condition, and the target characteristic distance is a characteristic distance between the second sample data and the first sample data after screening; Based on at least two target feature distances, weighted averaging is performed on the confidence values ​​corresponding to the at least two filtered second sample data to obtain a confidence label corresponding to the first sample data.

5. The method according to any one of claims 1 to 4, characterized in that: The step of training the sample confidence prediction network based on the difference between the prediction confidence value and the confidence label to obtain the confidence prediction network includes: Obtaining a preset loss function, wherein the preset loss function is used to calculate the difference between the prediction confidence value and the confidence label; Obtaining a target loss value between the prediction confidence value and the confidence label through the preset loss function; The sample confidence prediction model is trained based on the target loss value to obtain the confidence prediction network.

6. The method according to claim 5, characterized in that The obtaining a target loss value between the prediction confidence value and the confidence label by using the preset loss function includes: Substitute the predicted confidence value and the confidence label into the preset loss function, and output the mean square error result as the target loss value.

7. The method according to any one of claims 1 to 5, characterized in that: After obtaining the first sample data and the plurality of second sample data, the method further includes: Performing classification prediction on the plurality of second sample data respectively by using the classification prediction model to obtain classification prediction results corresponding to the plurality of second sample data respectively, wherein the classification prediction results include a plurality of prediction categories and candidate confidence values ​​corresponding to the plurality of prediction categories respectively; The candidate confidence value that meets the preset confidence value condition is used as the confidence value corresponding to the second sample data.

8. The method according to claim 7, characterized in that After performing classification prediction on the plurality of second sample data respectively through the classification prediction model to obtain classification prediction results corresponding to the plurality of second sample data respectively, the method further includes: In response to a difference between a plurality of candidate confidence values ​​in a classification prediction result of the i-th second sample data being less than a preset confidence value difference, obtaining data association degrees between other second sample data and the i-th second sample data, where i is a positive integer; In response to the data correlation between the kth second sample data and the ith second sample data reaching a preset correlation threshold, the confidence value corresponding to the kth second sample data is used as the confidence value corresponding to the ith second sample data, where k is a positive integer.

9. The method according to any one of claims 1 to 5, characterized in that: The obtaining of data associations between the plurality of second sample data and the first sample data respectively includes: The first sample data is subjected to feature extraction by a feature extraction model to obtain the first feature representation, and the plurality of second sample data are subjected to feature extraction by the feature extraction model to obtain second feature representations corresponding to the plurality of second sample data respectively.

10. The method according to claim 9, characterized in that Before extracting features from the first sample data using a feature extraction model to obtain the first feature representation, the method further includes: Acquire third sample data, where the third sample data is annotated with a feature representation label; Inputting the third sample data into a sample extraction model, and outputting a prediction feature representation corresponding to the third sample data; Based on the difference between the predicted feature representation and the feature representation label, the sample extraction model is trained to obtain the feature extraction model.

11. The method according to any one of claims 1 to 5, characterized in that: After training the sample confidence prediction network based on the difference between the prediction confidence value and the confidence label to obtain the confidence prediction network, the method further includes: Obtain input data corresponding to a target task, wherein the target task is annotated with a confidence threshold; Extracting an input feature representation corresponding to the input data; Inputting the input feature representation into the classification prediction model, outputting a classification prediction result corresponding to the input data, and inputting the input feature representation into the confidence prediction network, outputting a confidence value prediction result corresponding to the prediction category of the input data; In response to the confidence value prediction result reaching the confidence threshold, performing the target task based on the classification prediction result.

12. An evaluation device for a classification prediction model, characterized in that: The device comprises: an acquisition module, configured to acquire first sample data and a plurality of second sample data, wherein the plurality of second sample data are respectively annotated with a confidence value, wherein the confidence value is a probability output by a classification prediction model that indicates whether the second sample data corresponds to the category; The acquisition module is further configured to acquire data association degrees between the plurality of second sample data and the first sample data respectively; a fusion module, configured to fuse the confidence values ​​corresponding to the plurality of second sample data based on the data association degrees corresponding to the plurality of second sample data, to obtain a confidence label corresponding to the first sample data; A prediction module, configured to predict a confidence value of the first sample data using a sample confidence prediction network to obtain a prediction confidence value corresponding to the first sample data; A training module is used to train the sample confidence prediction network based on the difference between the prediction confidence value and the confidence label to obtain a confidence prediction network, and the confidence prediction network is used to evaluate the accuracy of the classification prediction result output after the classification prediction model performs classification prediction on the input data.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the evaluation method of the classification prediction model as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image object recognition method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implements the evaluation method of the classification prediction model as described in any one of claims 1 to 11.

Images