Information classification method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an information classification method and device, electronic equipment, a computer readable storage medium and a computer program product, and is at least applied to the field of artificial intelligence or information classification, and the method comprises the steps: obtaining to-be-classified information and a preset training set in response to an information classification request; determining a classification prediction result of the to-be-classified information by using a pre-trained classification model; the pre-trained classification model is obtained based on training of a training set, and the training set comprises a plurality of training samples and a label category of each training sample; determining a label class probability of each training sample in the training set; the label class probability is the probability that the training sample predicted through the classification model belongs to the label class; based on the label class probability of each training sample, determining a target confidence degree of the to-be-classified information; and determining a target category of the to-be-classified information based on the target confidence of the to-be-classified information and the classification prediction result. According to the invention, the accuracy of information classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an information classification method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence (AI) technology, information classification can be applied to various real-world scenarios, such as autonomous driving, medical diagnosis, and financial risk assessment. However, in the face of complex and ever-changing real-world scenarios, models must possess strong generalization capabilities to adapt to various unknown situations. Therefore, the reliability of the model's predicted categories for unknown test samples is crucial.

[0003] In related technologies, the confidence of a test sample is typically estimated by predicting the maximum probability from samples surrounding it. The confidence of a test sample refers to the model's confidence in its predicted category. However, this approach is susceptible to overconfidence in surrounding samples, leading to inaccurate confidence estimates. This, in turn, makes it impossible to determine the credibility of the model's predictions, reducing the accuracy of information classification using the model. Summary of the Invention

[0004] The embodiments of the present application provide an information classification method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of information classification.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides an information classification method, which includes: in response to an information classification request, obtaining information to be classified and a preset training set; using a pre-trained classification model, determining a classification prediction result of the information to be classified; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and a label category of each training sample; determining the label class probability of each training sample in the training set; the label class probability is the probability that the training sample belongs to the label category predicted by the classification model; based on the label class probability of each training sample, determining the target confidence of the information to be classified; based on the target confidence of the information to be classified and the classification prediction result, determining the target category of the information to be classified.

[0007] An embodiment of the present application provides an information classification device, comprising: an information acquisition module, configured to acquire information to be classified and a preset training set in response to an information classification request; a prediction module, configured to determine a classification prediction result of the information to be classified using a pre-trained classification model; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and a label category of each training sample; a label probability determination module, configured to determine the label class probability of each training sample in the training set; the label class probability is the probability that the training sample belongs to the label category predicted by the classification model; a confidence determination module, configured to determine a target confidence of the information to be classified based on the label class probability of each training sample; and a category determination module, configured to determine a target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

[0008] In some embodiments, the prediction module is also used to: use the feature extraction layer of the classification model to extract features of the information to be classified to obtain a first feature vector of the information to be classified; use the classification layer of the classification model to perform classification processing on the first feature vector to obtain the predicted probability of the information to be classified in each preset category; arrange the predicted probability of the information to be classified in each preset category in a preset category order to form a vector, and determine it as the classification prediction result of the information to be classified.

[0009] In some embodiments, the confidence determination module is further configured to: determine a confidence weight of each of the training samples; and determine a target confidence of the information to be classified based on the confidence weight of each of the training samples and the label class probability.

[0010] In some embodiments, the confidence determination module is further used to: determine the characteristic distance between the information to be classified and each of the training samples; normalize the characteristic distance between the information to be classified and each of the training samples to obtain the corresponding confidence weight of each of the training samples.

[0011] In some embodiments, the confidence determination module is also used to: use the feature extraction layer of the classification model to perform feature extraction on each of the training samples to obtain a second feature vector of each of the training samples; obtain a first feature vector of the information to be classified, the first feature vector being obtained by performing feature extraction on the information to be classified through the feature extraction layer of the classification model; and determine the feature distance between the information to be classified and each of the training samples based on the first feature vector and the second feature vector of each of the training samples.

[0012] In some embodiments, the confidence determination module is further used to: for each of the training samples, multiply the confidence weight by the label class probability to obtain a weighted probability; and add the weighted probabilities of each of the training samples to obtain the target confidence of the information to be classified.

[0013] In some embodiments, the category determination module is also used to: obtain a preset confidence threshold; when the target confidence of the information to be classified is greater than or equal to the confidence threshold, determine the preset category corresponding to the maximum prediction probability in the classification prediction result as the target category of the information to be classified; when the target confidence of the information to be classified is less than the confidence threshold, generate a reminder message and send the reminder message to the terminal; the reminder message is used to remind the target object to determine the target category of the information to be classified.

[0014] In some embodiments, the confidence determination module is also used to: for each preset category, based on the label category of each training sample, obtain at least one training sample under the preset category from the training set; multiply the confidence weight and label class probability of each training sample under the preset category to obtain the corresponding weighted category probability of each training sample; add the weighted category probabilities of all training samples under the preset category to obtain the target confidence of the information to be classified under the preset category.

[0015] In some embodiments, the category determination module is also used to: obtain the target confidence of the information to be classified under each of the preset categories to obtain a target confidence set; determine the maximum target confidence from the target confidence set; if the preset category corresponding to the maximum target confidence is consistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified; if the preset category corresponding to the maximum target confidence is inconsistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum target confidence is determined as the target category of the information to be classified.

[0016] In some embodiments, the information classification device also includes a training module, which is used to: utilize the feature extraction layer of the classification model to be trained to perform feature extraction processing on the training samples in the training set to obtain the third feature vector of the training samples; utilize the classification layer of the classification model to be trained to perform classification processing on the third feature vector of the training samples to obtain the training classification results of the training samples; based on the training classification results and label categories of the training samples, iteratively update the model parameters of the classification model to be trained until the classification model to be trained converges to obtain the pre-trained classification model.

[0017] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for implementing the information classification method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the information classification method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the information classification method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] When classifying information to be classified, a pre-trained classification model can be used to determine the classification prediction result of the information to be classified, and the target category of the information to be classified can be determined based on the target confidence of the information to be classified and the classification prediction result. Among them, the target confidence of the information to be classified is determined based on the label class probability of the training sample used when pre-training the classification model. The label class probability is the probability that the training sample belongs to the label category predicted by the classification model. In the embodiment of the present application, when determining the target confidence of the information to be classified, the label class probability of the training sample under the label category is used, that is, the probability that the training sample belongs to the true category. Compared with the related technology of using the maximum predicted probability of the surrounding samples to approximate the confidence of the test sample, in the embodiment of the present application, after the training sample is classified by the pre-trained classification model, the classification result is obtained, and when the classification result is wrong, since the label class probability of the training sample is less than the maximum predicted probability output by the classification model, a lower target confidence of the information to be classified will be further determined, thereby reducing the impact of overconfidence of the training sample, estimating the target confidence of the information to be classified more accurately, and then determining the target category of the information to be classified by the target confidence, which can improve the accuracy of information classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an optional architectural diagram of the information classification system provided in the embodiment of the present application;

[0023] Figure 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 3 This is an optional flowchart of the information classification method provided in the embodiment of the present application;

[0025] Figure 4 This is another optional flowchart of the information classification method provided in the embodiment of the present application;

[0026] Figure 5 This is an optional flowchart of the information classification method provided in the embodiment of the present application;

[0027] Figure 6 This is an optional flowchart of the information classification method provided in the embodiment of the present application;

[0028] Figure 7 This is a schematic diagram of the target recognition process during the autonomous driving process provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0032] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0033] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] 1) Confidence: refers to the confidence level of the classification model, that is, the confidence of the classification model in the classification results of the sample.

[0036] 2) Overconfidence: The model will overconfidently give high prediction probabilities for samples of unknown categories or location domains (in other words, categories that do not appear in the training set or samples with a different distribution from the training set), causing unknown samples to be incorrectly classified into known categories.

[0037] 3) Generalization ability: The generalization ability of a model refers to the ability of a model to correctly understand and predict new, unseen data.

[0038] In the related art, the confidence of the test sample is usually approximated by the samples around the test sample. Specifically, the word embedding similarity (such as the cosine similarity of the vector) between the test sample and the surrounding samples is used as an indicator, and the weighted average of the maximum probability values ​​of the surrounding samples predicted by the classification model is used as the confidence of the test sample. However, this method is affected by the overconfidence of the surrounding samples. For example, around the test sample, if all the samples have inaccurate confidence estimates or some samples have inaccurate confidence estimates (such as the true category of sample A is cat, but the maximum probability value predicted by the classification model is 0.9, and the category corresponding to the maximum probability value 0.9 is dog, and the predicted probability is 0.9), then the confidence obtained by the test sample through the surrounding samples is also inaccurate.

[0039] Based on the problems existing in the related art, an embodiment of the present application provides an information classification method, which is a confidence estimation method based on true class probability in the scenario where the classification model performs confidence estimation on the test sample. When estimating the confidence level of the classification model in the test sample, this method uses the true class probability of the surrounding samples instead of the maximum class probability, making full use of the true class probability information of the surrounding samples, reducing the impact of overconfidence, thereby improving the accuracy of the confidence estimation of the classification model on the test sample, and further improving the generalization ability and practical application value of the classification model, and improving the accuracy of information classification.

[0040] Specifically, in the information classification method provided in the embodiment of the present application, first, in response to an information classification request, the information to be classified and a preset training set are obtained; then, the classification prediction result of the information to be classified is determined using a pre-trained classification model; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and the label category of each training sample; the label class probability of each training sample in the training set is determined; based on the label class probability of each training sample, the target confidence of the information to be classified is determined; finally, based on the target confidence of the information to be classified and the classification prediction result, the target category of the information to be classified is determined.

[0041] Here, first, an exemplary application of the information classification device of the embodiment of the present application is described, and the information classification device is an electronic device for implementing the information classification method. In one implementation, the information classification device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal or as a server. In one implementation, the information classification device provided in the embodiment of the present application can be implemented as a laptop, tablet computer, desktop computer, mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, intelligent robot, smart home appliance and smart car-mounted device, etc., any terminal with data processing and information classification functions; in another implementation, the information classification device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. Next, an exemplary application when the information classification device is implemented as a server will be described.

[0042] See also Figure 1 , Figure 1 This is an optional architectural diagram of the information classification system provided in an embodiment of the present application. In order to achieve accurate information classification of the information to be classified, an information classification application can be provided. For example, the information classification application can be an application dedicated to information classification, or it can be a functional module in other applications (such as a target recognition module in an autonomous driving application, etc.).

[0043] The information classification system 100 provided in the embodiment of the present application includes at least a terminal 400, a network 300, and a server 200. The information classification application is running on the terminal 400, and the server 200 is the server of the information classification application. The server 200 can constitute the information classification device of the embodiment of the present application, that is, the information classification method of the embodiment of the present application is implemented by the server 200. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0044] See also Figure 1When classifying information to be classified, a user can perform an interactive operation on the client side of the information classification application via terminal 400. This interactive operation can, for example, be a click operation to initiate target recognition. After receiving the user's interactive operation, the client side can encapsulate the terminal identifier or client identifier into an information classification request and send the information classification request to server 200 via terminal 400. Upon receiving the information classification request, server 200 responds to the information classification request by obtaining the information to be classified and a pre-set training set. It then uses a pre-trained classification model to determine a classification prediction result for the information to be classified. The pre-trained classification model is trained based on the training set, which includes multiple training samples and a label category for each training sample. It then determines a label class probability for each training sample in the training set, which is the probability that the training sample belongs to the label category predicted by the classification model. It then determines a target confidence for the information to be classified based on the label class probability for each training sample. It then determines a target category for the information to be classified based on the target confidence for the information to be classified and the classification prediction result. After determining the target category for the information to be classified, server 200 can send the target category to terminal 400 to display the target category to the user.

[0045] In some embodiments, the information classification method of the embodiment of the present application can also be executed by the terminal 400 itself, that is, after the terminal 400 receives the interactive operation input by the user through the client, the terminal 400 obtains the information to be classified and the preset training set; uses the pre-trained classification model to determine the classification prediction result of the information to be classified; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and the label category of each training sample; determines the label class probability of each training sample in the training set, and the label class probability is the probability that the training sample belongs to the label category predicted by the classification model; based on the label class probability of each training sample, determines the target confidence of the information to be classified; based on the target confidence of the information to be classified and the classification prediction result, determines the target category of the information to be classified. After determining the target category of the information to be classified, the target category is displayed on the client interface of the terminal 400.

[0046] The information classification method provided in the embodiment of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 200 can be a cloud server. The cloud server responds to the information classification request and obtains the information to be classified and a preset training set; or, the cloud server uses a pre-trained classification model to determine the classification prediction result of the information to be classified; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and the label category of each training sample; or, the cloud server determines the label class probability of each training sample in the training set; or, the cloud server determines the target confidence of the information to be classified based on the label class probability of each training sample; or, the cloud server determines the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

[0047] In some embodiments, a cloud storage may be provided to store a preset training set and a pre-trained classification model. In this way, when an information classification request is received, the preset training set and the pre-trained classification model can be directly obtained from the cloud storage, thereby improving the efficiency of information classification.

[0048] It's important to note that cloud technology refers to a managed technology that unifies hardware, software, and network resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool for on-demand, flexible, and convenient use. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as those for video sites, image sites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Data of varying levels will be processed separately, and data from various industries will require robust system support, which can be achieved through cloud computing.

[0049] See also Figure 2 , Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2The electronic device shown may be an information classification device, which includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0050] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0052] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0053] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0054] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0055] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; a network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; a presentation module 453 is used to enable information to be presented (for example, a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (for example, a display screen, a speaker, etc.); an input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0056] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 An information classification device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an information acquisition module 4551, a prediction module 4552, a label probability determination module 4553, a confidence determination module 4554, and a category determination module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0057] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the information classification method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0058] The information classification method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the information classification method provided in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0059] Figure 3 This is an optional flow chart of the information classification method provided in the embodiment of the present application. Figure 3 The steps shown are explained as Figure 3 As shown, the information classification method is described by taking the server as an example, and the method includes the following steps S101 to S105:

[0060] Step S101: In response to an information classification request, obtain information to be classified and a preset training set.

[0061] Here, the preset training set is used to train the classification model, and the classification model is used to classify the information to be classified. The information to be classified refers to new information that does not appear in the training set, including but not limited to text, pictures, videos, audio, etc. The information to be classified may belong to known categories in the training set, or it may belong to unknown categories. For example, in an autonomous driving scenario, the information to be classified can be photos obtained by collecting images of unknown objects (such as other vehicles, street lights, obstacles, puddles, etc.) waiting for target recognition on the road where the vehicle is traveling.

[0062] Step S102: using the pre-trained classification model to determine the classification prediction result of the information to be classified.

[0063] The pre-trained classification model is trained based on a training set, which includes multiple training samples and a label category for each training sample.

[0064] Here, the multiple training samples in the training set are classified information samples, including but not limited to text, images, videos, audio, etc. The label categories of the training samples are real categories obtained by manual annotation. For example, in an autonomous driving scenario, training sample 1 is a photo of a street lamp, and its label category can be "street lamp". A pre-trained classification model is a model that is pre-trained based on the training set and is used for information classification. The classification model may include a feature extraction layer and a classification layer. The feature extraction layer is used to perform feature extraction processing on the input data of the classification model to obtain a feature vector. It should be noted that the embodiment of the present application does not specifically limit the structure of the feature extraction layer. For example, the feature extraction layer can be a deep neural network (such as a convolutional neural network or a recurrent neural network) or other types of machine learning models. The classification layer is used to classify the input data (information to be classified or training samples) based on the feature vectors extracted by the feature extraction layer. It should be noted that the embodiment of the present application does not specifically limit the structure of the classification layer. For example, the classification layer can be a machine learning model such as a linear classifier, a support vector machine, or a decision tree. If the information to be classified is input into the pre-trained classification model, the output of the classification layer is the classification prediction result of the information to be classified. The classification prediction result is expressed as a vector, which includes the predicted probability of the information to be classified under each category.

[0065] In some embodiments, multiple labeled training samples and the label category of each training sample can be first obtained. The multiple training samples and the label category of each training sample constitute a training set. A classification model to be trained is obtained. First, the feature extraction layer of the classification model to be trained is used to perform feature extraction processing on each training sample in the training set to obtain a third eigenvector for each training sample. Then, the classification layer of the classification model to be trained is used to perform classification processing on the third eigenvector of each training sample to obtain a training classification result for each training sample. Here, the training classification result is expressed as a vector, and the training classification result includes the predicted probability of the training sample under each preset category. Based on the training classification result and label category of each training sample, a loss function (e.g., cross entropy loss, mean squared error loss, etc.) is calculated. A gradient descent method (e.g., stochastic gradient descent, Adam (Adaptive Moment Estimation) optimizer, etc.) is used to update the model parameters of the feature extraction layer and the classification layer according to the loss function to minimize the loss function. After multiple iterations, when the loss function reaches a minimum value, the classification model to be trained converges, and a pre-trained classification model is obtained.

[0066] It should be noted that when the classification model predicts the probability of a training sample under each preset category, each preset category can include known categories in the training set, that is, all label categories obtained after manually annotating all training samples in the training set. Each preset category can also include multiple categories obtained by the classification model based on unsupervised training.

[0067] Step S103: Determine the label class probability of each training sample in the training set.

[0068] The label class probability is the probability that the training sample belongs to the label category predicted by the classification model.

[0069] Here, for any training sample, the sample prediction probability that the training sample belongs to each preset category can be obtained through the pre-trained classification model. The classification model outputs the preset category corresponding to the maximum sample prediction probability among the sample prediction probabilities that the training sample belongs to each preset category as the predicted category of the training sample. However, since the accuracy of the classification model cannot reach 100%, the predicted category of the training sample output by the classification model may not be the true category of the training sample. The label category is the true category obtained by manually labeling the training sample. The label class probability is the sample prediction probability of the training sample under the label category output by the classification model. For any training sample, the label category of the training sample is obtained, and based on the label category, the sample prediction probability of the training sample under each preset category output by the pre-trained classification model is used to determine the probability that the training sample belongs to the label category as the label class probability.

[0070] Step S104: determining the target confidence of the information to be classified based on the label class probability of each training sample.

[0071] Here, the target confidence of the information to be classified is the confidence of the classification model in the classification prediction result output by the classification information to be classified. The higher the target confidence of the information to be classified, the more reliable the classification prediction result output by the classification model is, and the closer the target category obtained from the classification prediction result is to the true category of the information to be classified.

[0072] In the embodiment of the present application, after obtaining the label class probability for each training sample, the confidence weight for each training sample can be determined. Then, based on the confidence weight and label class probability for each training sample, the target confidence level for the information to be classified is determined. The confidence weight for each training sample can be obtained by normalizing the feature distance between the training sample and the information to be classified.

[0073] In some embodiments, the product of the confidence weight of each training sample and the label class probability may be summed to obtain the target confidence of the information to be classified.

[0074] In some embodiments, for each preset category, the product of the confidence weight of each training sample belonging to the preset category and the label class probability may be summed to obtain the target confidence of the information to be classified under the preset category.

[0075] Step S105 : determining the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

[0076] In some embodiments, if the product of the confidence weight and the label class probability of each training sample is summed to obtain the target confidence of the information to be classified, a preset confidence threshold can be obtained. The embodiment of the present application does not specifically limit the value of the confidence threshold, and can be set according to experience and actual conditions. For example, the general confidence threshold is 0.98. When the target confidence of the information to be classified is greater than or equal to the confidence threshold, the preset category corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified. Alternatively, when the target confidence of the information to be classified is less than the confidence threshold, the classification prediction result of the information to be classified output by the classification model is unreliable, and a reminder message is generated. The embodiment of the present application does not specifically limit the specific content and form of expression of the reminder message, as long as it can characterize that the classification prediction result is unreliable. Send a reminder message to the terminal. After seeing the reminder message, the target object (such as a user) can determine the target category of the information to be classified by itself.

[0077] In some embodiments, if the target confidence level for each preset category of the information to be classified is obtained, the preset category corresponding to the maximum target confidence level can be compared for consistency with the preset category corresponding to the maximum predicted probability in the classification prediction result of the information to be classified. If the preset category corresponding to the maximum target confidence level is consistent with the preset category corresponding to the maximum predicted probability in the classification prediction result, the preset category corresponding to the maximum predicted probability in the classification prediction result is determined as the target category of the information to be classified. Alternatively, if the preset category corresponding to the maximum target confidence level is inconsistent with the preset category corresponding to the maximum predicted probability in the classification prediction result, the preset category corresponding to the maximum target confidence level is determined as the target category of the information to be classified.

[0078] The information classification method provided in the embodiments of the present application can first use a pre-trained classification model to determine a classification prediction result for the information to be classified, and then determine a target category for the information to be classified based on the target confidence of the information to be classified and the classification prediction result. The target confidence of the information to be classified is determined based on the label class probability of the training samples used when training the pre-trained classification model, and the label class probability is the probability that the training sample belongs to the label category predicted by the classification model. In an embodiment of the present application, when determining the target confidence of the information to be classified, the label class probability of the training sample under the label category is used, that is, the probability that the training sample belongs to the true category. Compared with the related technology of using the predicted maximum probability of the surrounding samples to approximate the confidence of the test sample, in an embodiment of the present application, after classifying the training sample through a pre-trained classification model, a classification result is obtained, and when an error occurs in the classification result, since the label class probability of the training sample under the label category is less than the predicted maximum probability output by the classification model, a lower target confidence of the information to be classified will be determined, thereby reducing the impact of overconfidence of the training sample, and estimating the target confidence of the information to be classified more accurately, and then determining the target category of the information to be classified through the target confidence, which can improve the accuracy of information classification.

[0079] In some embodiments, the information classification system includes at least a terminal and a server, wherein an information classification application runs on the terminal. The information classification application can be, for example, an application dedicated to information classification, or it can be a functional module in other applications (such as a target recognition module in an autonomous driving application, etc.).

[0080] Figure 4 This is another optional flow chart of the information classification method provided in the embodiment of the present application, such as Figure 4 As shown, the method includes the following steps S201 to S210:

[0081] Step S201: The terminal receives an interactive operation from a user.

[0082] Here, the interactive operation may be a click operation to start target recognition, etc. The user may perform any interactive operation on the client side of the information classification application through the terminal.

[0083] Step S202: The terminal generates an information classification request in response to the interactive operation.

[0084] When the terminal receives a user interaction, it determines that the information to be classified currently needs to be classified. Therefore, in response to the interaction, it generates an information classification request. The information classification request is used to request the server to use a pre-trained classification model to perform classification prediction on the information to be classified, obtain a classification prediction result, and determine the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

[0085] Step S203: The terminal sends an information classification request to the server.

[0086] Step S204: The server obtains the information to be classified and a preset training set in response to the information classification request.

[0087] Here, the training set includes multiple training samples and the label category of each training sample. The multiple training samples in the training set are classified information samples. The label categories of the training samples are the real categories obtained by manual annotation. The information to be classified refers to new information that does not appear in the training set. The information to be classified may belong to known categories in the training set or to unknown categories. The information to be classified and the training samples may be presented in forms including but not limited to text, images, videos, audio, etc.

[0088] Taking the application scenario of autonomous driving as an example, the training samples can be photos of various objects on various roads collected historically. The information to be classified can be photos obtained by collecting images of unknown objects (such as other vehicles, street lights, obstacles, puddles, etc.) waiting for target recognition on the road where the vehicle is traveling. For example, the training set includes training sample 1 "pictures of street lights", training sample 2 "pictures of trees", training sample 3 "pictures of vehicles", and training sample 4 "pictures of nails". The label category of training sample 1 and training sample 2 can be "environment", the label category of training sample 3 is "vehicle", and the label category of training sample 4 is "dangerous goods". While the vehicle is driving, it collects photos of an unknown object in front of the road in real time as information to be classified.

[0089] In step S205 , the server uses a pre-trained classification model to determine a classification prediction result of the information to be classified.

[0090] The pre-trained classification model is obtained by training based on the training set. In an embodiment of the present application, the pre-trained classification model can be trained in the following manner: first, using the feature extraction layer of the classification model to be trained, the training samples in the training set are subjected to feature extraction processing to obtain the third eigenvector of the training samples; then, using the classification layer of the classification model to be trained, the third eigenvector of the training samples is subjected to classification processing to obtain the training classification results of the training samples; finally, based on the training classification results and label categories of the training samples, the model parameters of the classification model to be trained are iteratively updated until the classification model to be trained converges to obtain the pre-trained classification model.

[0091] Here, the pre-trained classification model may include a feature extraction layer and a classification layer. The feature extraction layer may be a deep neural network (e.g., a convolutional neural network or a recurrent neural network) or another type of machine learning model. The classification layer may be a machine learning model such as a linear classifier, a support vector machine, or a decision tree. During the nth iteration (n is a positive integer), the feature extraction layer of the classification model to be trained may be used to perform feature extraction on each training sample in the training set, obtaining a third eigenvector corresponding to each training sample. The classification layer of the classification model to be trained may be used to perform classification on the third eigenvector corresponding to each training sample, obtaining a sample prediction probability for each training sample in each preset category. For each training sample, the sample prediction probabilities for the training sample in each preset category are arranged and combined in a preset category order, resulting in a vector, which is the training classification result. Based on the training classification results of the training samples and the vectors corresponding to the label categories of the training samples, the loss of the classification model (e.g., cross entropy loss, mean squared error loss, etc.) may be calculated. Use the gradient descent method to update the model parameters of the feature extraction layer and the classification layer based on the loss result to obtain the updated classification model. Use the updated classification model as the classification model to be trained in the n+1th iteration process. Continue the n+1th iteration process until the loss result reaches the minimum value and the classification model to be trained converges. Stop the above iteration process to obtain the pre-trained classification model.

[0092] For example, if the training set includes training sample 1 "pictures of streetlights", training sample 2 "pictures of trees", training sample 3 "pictures of vehicles" and training sample 4 "pictures of nails". The label category of training sample 1 and training sample 2 can be "environment", the label category of training sample 3 is "vehicle", and the label category of training sample 4 is "dangerous goods". The preset category order is: "environment", "vehicle", "dangerous goods". The sample prediction probability of training sample 1 in the "environment" category is 0.4, the sample prediction probability in the "vehicle" category is 0.3, and the sample prediction probability in the "dangerous goods" category is 0.3, then the training classification result of training sample 1 is the vector (0.4, 0.3, 0.3). The label category of training sample 1 is "environment", then the vector corresponding to the label category is (1, 0, 0). The cross entropy loss function can be calculated based on the vector (0.4, 0.3, 0.3) and the vector (1, 0, 0) to obtain the loss result of the classification model to be trained.

[0093] In some embodiments, see Figure 5 , Figure 5 It is shown that in step S205, the classification prediction result of the information to be classified is determined by using the pre-trained classification model, which can be achieved by the following steps S2051 to S2053:

[0094] Step S2051 : Using the feature extraction layer of the classification model, feature extraction is performed on the information to be classified to obtain a first feature vector of the information to be classified.

[0095] The embodiment of the present application does not specifically limit the structure of the feature extraction layer and the feature extraction method, and does not specifically limit the length of the first feature vector.

[0096] Step S2052: using the classification layer of the classification model, classify the first feature vector to obtain the predicted probability of the information to be classified in each preset category.

[0097] Here, the preset categories can be known categories in the training set, that is, all label categories obtained after manually annotating all training samples in the training set. The preset categories can also be multiple categories obtained by the classification model based on unsupervised training. For example, the training set includes training sample 1: "Picture of a streetlight," training sample 2: "Picture of a tree," training sample 3: "Picture of a vehicle," and training sample 4: "Picture of a nail." The label category of training samples 1 and 2 can be "environment," the label category of training sample 3: "vehicle," and the label category of training sample 4: "dangerous goods." During the training process, the classification model also obtains "Category 1" based on unsupervised training. The preset categories are then "environment," "vehicle," "dangerous goods," and "Category 1." Using the classification layer of the classification model, the first eigenvector of the information to be classified is classified to obtain the predicted probability that the information to be classified belongs to "environment," the predicted probability that it belongs to "vehicle," the predicted probability that it belongs to "dangerous goods," and the predicted probability that it belongs to "Category 1." The predicted probabilities range from 0 to 1.

[0098] Step S2053 , the predicted probabilities of the information to be classified under each preset category are arranged in a preset category order to form a vector, and the vector is determined as the classification prediction result of the information to be classified.

[0099] For example, if the preset category order is "environment," "vehicle," and "dangerous goods," the predicted probability that the information to be classified belongs to "environment" is 0.3, the predicted probability that it belongs to "vehicle" is 0.3, and the predicted probability that it belongs to "dangerous goods" is 0.4. The classification prediction result of the information to be classified is the vector (0.3, 0.3, 0.4).

[0100] The embodiment of the present application obtains a classification prediction result by arranging the predicted probabilities of the information to be classified under each preset category in order, and corresponds the predicted probabilities to the positions of the preset categories one by one, so as to facilitate the subsequent determination of the target category of the information to be classified from the classification prediction results and improve the efficiency of information classification.

[0101] Step S206: The server determines the label class probability of each training sample in the training set.

[0102] The label class probability is the probability that the training sample belongs to the label category predicted by the classification model.

[0103] In an embodiment of the present application, determining the label class probability of each training sample in the training set can be achieved in the following manner: first, obtaining the sample classification result of each training sample output by a pre-trained classification model; the sample classification result includes the sample prediction probability of the training sample under each preset category; and obtaining the label category of each training sample; then, for each training sample, the sample prediction probability of the training sample under the label category in the sample classification result is determined as the label class probability of the training sample.

[0104] Here, the label class probability of a training sample is the sample prediction probability of the training sample under its label class, as output by the pre-trained classification model. For each training sample, the training sample can be input into the pre-trained classification model to obtain the sample prediction probability of the training sample under each preset class. The vector obtained by combining the sample prediction probabilities of the training sample under each preset class in the preset class order is the sample classification result of the training sample. The label sample of the training sample is obtained from the training set. If the class order in the sample classification result is known, the position of the label sample in the sample classification result can be determined, and the sample prediction probability at this position is determined as the label class probability of the training sample.

[0105] For example, the preset category order is "environment", "vehicle", and "dangerous goods". The sample classification result of training sample 1 is a vector (0.2, 0.3, 0.5). The label category of training sample 1 is "environment", so the label class probability of training sample 1 is 0.2.

[0106] In step S207 , the server determines the target confidence of the information to be classified based on the label class probability of each training sample.

[0107] In some embodiments, see Figure 6 , Figure 6 It shows that in step S207, based on the label class probability of each training sample, the target confidence of the information to be classified is determined, which can be achieved by the following steps S2071 to S2072:

[0108] Step S2071: Determine the confidence weight of each training sample.

[0109] In an embodiment of the present application, determining the confidence weight of each training sample can be achieved in the following manner: first, determining the characteristic distance between the information to be classified and each training sample; then, normalizing the characteristic distance between the information to be classified and each training sample to obtain the corresponding confidence weight of each training sample.

[0110] Here, for each training sample, the characteristic distance between the information to be classified and the training sample is the characteristic representation distance between the first eigenvector of the information to be classified and the second eigenvector of the training sample obtained by the pre-trained classification model, such as Euclidean distance, cosine similarity, Mahalanobis distance, etc. After obtaining the characteristic distance between the information to be classified and each training sample, the characteristic distance between the information to be classified and each training sample can be normalized to obtain the confidence weight of each training sample. It should be noted that the embodiment of the present application does not specifically limit the specific method of normalization, as long as the confidence weight is greater, the smaller the characteristic distance between the training sample and the information to be classified. For example, a subtraction function, an inverse function, a Gaussian function, an exponential function, etc. can be used for normalization. Taking the normalized exponential function as an example here, the calculation formula of the confidence weight can satisfy the following formula (1).

[0111] w i =exp(-d(x i ,x′) / σ) / Σ j exp(-d(x j ,x′) / σ) Formula (1)

[0112] Among them, w i Represents the training sample x i The confidence weight of , σ is a positive number used to control the scale of the weight distribution, ∑ j represents the sum of all training samples in the training set, x j is the training sample j, x′ is the information to be classified, and both i and j are positive integers.

[0113] In an embodiment of the present application, determining the characteristic distance between the information to be classified and each training sample can be achieved in the following manner: first, using the feature extraction layer of the classification model, feature extraction is performed on each training sample to obtain the second feature vector of each training sample; then, the first feature vector of the information to be classified is obtained, and the first feature vector is obtained by performing feature extraction on the information to be classified through the feature extraction layer of the classification model; finally, based on the first feature vector and the second feature vector of each training sample, the characteristic distance between the information to be classified and each training sample is determined.

[0114] Here, for each training sample, a feature extraction layer of a pre-trained classification model can be used to extract features from the training sample to obtain a second feature vector for the training sample. For example, the Euclidean distance between the second feature vector of the training sample and the first feature vector of the information to be classified can be calculated as the feature distance between the information to be classified and the training sample. The calculation formula for the Euclidean distance satisfies the following formula (2).

[0115] d(x i,x′)=||h(x i )-h(x′)||2 Formula (2)

[0116] Among them, d(x i ,x′) represents the information to be classified x′ and the training sample x i The characteristic distance between them; ||·||2 represents the L2 norm, which is used to calculate the Euclidean distance between two eigenvectors; h(x i ) is the training sample x i The second eigenvector of , h(x′) is the first eigenvector of the information to be classified.

[0117] The embodiment of the present application determines the confidence weight of each to-be-classified sample based on the characteristic distance between the information to-be-classified and each training sample, so that the confidence weight of the training sample that is closer to the information to-be-classified is higher, that is, the greater the proportion in the target confidence calculation process of the information to be classified, which can improve the accuracy of calculating the target confidence of the information to be classified, thereby improving the accuracy of information classification.

[0118] Step S2072: Determine the target confidence of the information to be classified based on the confidence weight and label class probability of each training sample.

[0119] In some embodiments, the target confidence of the information to be classified is determined based on the confidence weight and label class probability of each training sample. This can be achieved in the following way: first, for each training sample, the confidence weight is multiplied by the label class probability to obtain a weighted probability; then, the weighted probability of each training sample is added to obtain the target confidence of the information to be classified.

[0120] For example, if the training set includes training sample 1 (a picture of a streetlight), training sample 2 (a picture of a tree), training sample 3 (a picture of a vehicle), and training sample 4 (a picture of a nail), the label class probability of training sample 1 can be multiplied by its confidence weight to obtain the weighted probability of training sample 1. The weighted probabilities of training sample 2, training sample 3, and training sample 4 are calculated sequentially using the above method. The sum of the weighted probabilities of training sample 1, training sample 2, training sample 3, and training sample 4 is then added together to determine the target confidence level for the information to be classified.

[0121] In some embodiments, the target confidence of the information to be classified is determined based on the confidence weight and label class probability of each training sample, which can also be achieved in the following way: first, for each preset category, based on the label category of each training sample, at least one training sample under the preset category is obtained from the training set; and the confidence weight and label class probability of each training sample under the preset category are multiplied to obtain the corresponding weighted category probability of each training sample; then the weighted category probabilities of all training samples under the preset category are added to obtain the target confidence of the information to be classified under the preset category.

[0122] For example, if all the preset categories are "environment", "vehicle", and "dangerous goods". The label category of training sample 1 and training sample 2 can be "environment", the label category of training sample 3 can be "vehicle", and the label category of training sample 4 can be "dangerous goods". Then, for the preset category "environment", training sample 1 and training sample 2 under the preset category "environment" can be obtained, and the confidence weight of training sample 1 is multiplied by its label class probability to obtain the weighted category probability of training sample 1. The confidence weight of training sample 2 is multiplied by its label class probability to obtain the weighted category probability of training sample 2. The weighted category probabilities of training sample 1 and training sample 2 are added together, and the sum obtained is determined as the target confidence of the information to be classified under the preset category "environment". For the preset category "vehicle", training sample 3 under the preset category "vehicle" can be obtained, and the target confidence of the information to be classified under the preset category "vehicle" can be calculated according to the above method. For the preset category “dangerous goods”, a training sample 4 under the preset category “dangerous goods” can be obtained, and the target confidence of the information to be classified under the preset category “dangerous goods” can be calculated according to the above method.

[0123] The embodiment of the present application determines the target confidence of the information to be classified by using the label class probability and confidence weight of the training sample, which can solve the problem of overconfidence of the training sample, thereby improving the estimation accuracy of the target confidence of the classification model on the information to be classified, helping to improve the performance of the classification model in practical applications, and further improving the accuracy of information classification.

[0124] In step S208 , the server determines the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

[0125] In some embodiments, determining the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result can be achieved in the following way: first, obtaining a preset confidence threshold; then, when the target confidence of the information to be classified is greater than or equal to the confidence threshold, determining the preset category corresponding to the maximum prediction probability in the classification prediction result as the target category of the information to be classified; or, when the target confidence of the information to be classified is less than the confidence threshold, generating a reminder message and sending the reminder message to the terminal; the reminder message is used to remind the target object to determine the target category of the information to be classified.

[0126] The embodiment of the present application is only applied when the product of the confidence weight and the label class probability of each training sample is summed to obtain the target confidence of the information to be classified.

[0127] For example, the preset confidence threshold is 0.98. If the target confidence of the information to be classified is 0.99, then the target confidence of the information to be classified is greater than the confidence threshold of 0.98, indicating that the classification prediction result output by the classification model has high credibility. If the classification prediction result is a vector (0.3, 0.3, 0.4), and the preset category order is "environment," "vehicle," and "dangerous goods," then the maximum prediction probability in the classification prediction result is 0.4, corresponding to the preset category "dangerous goods," and "dangerous goods" is determined as the target category of the information to be classified. If the target confidence of the information to be classified is 0.4, then the target confidence of the information to be classified is less than the confidence threshold of 0.98, indicating that the classification prediction result output by the classification model has low credibility. In this case, the true category of the information to be classified may not exist in the preset categories, and the classification model cannot accurately predict it. For example, if the information to be classified is a photo of a pedestrian, the true category of the information to be classified is "person." A warning message "Unable to recognize information to be classified" can be generated and sent to the terminal. After seeing the reminder message, the user can determine the target category of the information to be identified.

[0128] This embodiment of the application improves information classification accuracy by comparing the target confidence level with a preset confidence threshold. If the target confidence level is less than the confidence threshold, the classification prediction result is not used, and manual classification is performed instead. This is particularly useful in autonomous driving scenarios, reducing the risk of misidentification and improving safety.

[0129] In some embodiments, determining the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result can also be achieved in the following way: first, obtaining the target confidence of the information to be classified under each preset category to obtain a target confidence set; then, determining the maximum target confidence from the target confidence set; if the preset category corresponding to the maximum target confidence is consistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified; if the preset category corresponding to the maximum target confidence is inconsistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum target confidence is determined as the target category of the information to be classified.

[0130] The embodiments of the present application are only applied when the target confidence of the information to be classified is obtained under each preset category. For example, the preset category order is "environment", "vehicle", and "dangerous goods". The target confidence of the information to be classified under the preset category "environment" is 0.95, the target confidence under the preset category "vehicle" is 0.5, and the target confidence under the preset category "dangerous goods" is 0.99. Then the target confidence of 0.95 under the preset category "environment" of the information to be classified, the target confidence of 0.5 under the preset category "vehicle", and the target confidence of 0.99 under the preset category "dangerous goods" can be combined into a target confidence set. The maximum target confidence in the target confidence set is 0.99, and the preset category corresponding to the maximum target confidence is "dangerous goods". If the classification prediction result obtained by the classification model is a vector (0.3, 0.3, 0.4). Then the preset category corresponding to the maximum prediction probability in the classification prediction result is "dangerous goods". At this time, the preset category corresponding to the maximum target confidence is consistent with the preset category corresponding to the maximum prediction probability in the classification prediction result. The preset category "dangerous goods" corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified. Alternatively, if the classification prediction result obtained by the classification model is a vector (0.4, 0.3, 0.3), then the preset category corresponding to the maximum prediction probability in the classification prediction result is "environment". At this time, the preset category corresponding to the maximum target confidence is inconsistent with the preset category corresponding to the maximum prediction probability in the classification prediction result. The preset category "dangerous goods" corresponding to the maximum target confidence is determined as the target category of the information to be classified.

[0131] The embodiment of the present application can improve the accuracy of information classification by correcting the classification prediction result based on the preset category corresponding to the maximum target confidence.

[0132] Step S209: The server sends the target category of the information to be classified to the terminal.

[0133] In step S210, the terminal displays the target category of the information to be classified on the current interface.

[0134] In the embodiment of the present application, since the label class probability of the training sample under the label category is used when determining the target confidence of the information to be classified, that is, the probability that the training sample belongs to the true category, when the pre-trained classification model misclassifies the training sample, the label class probability of the training sample under the label category will be less than the predicted maximum probability output by the classification model, and a lower target confidence of the information to be classified is determined, thereby reducing the impact of overconfidence of the training sample, and estimating the target confidence of the information to be classified more accurately, and then determining the target category of the information to be classified by the target confidence, which can improve the accuracy of information classification.

[0135] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0136] The present application provides an information classification method that can be applied to any scenario where a classification model is used. As a confidence estimation method based on true class probabilities, this information classification method can bring significant benefits to a variety of application areas. First, this method can improve the accuracy of confidence estimates of non-standard models on test samples, thereby improving the predictive ability of the classification model. In many practical application scenarios, such as autonomous driving, medical diagnosis, and financial risk assessment, the credibility of the classification model's prediction results is crucial. The information classification method provided by the present application helps improve the accuracy and reliability of prediction results in these scenarios and reduce the risk of misjudgment. The present application helps improve the generalization ability of the classification model. When faced with complex and changing real-world scenarios, the classification model needs to have strong generalization capabilities to adapt to various unknown situations. The information classification method provided by the present application uses feature representation distance as a weight for confidence estimation, enabling the classification model to better adapt to different scenarios, thereby improving its generalization ability. This is crucial for many practical applications. For example, in autonomous driving scenarios, the system needs to be able to cope with various unknown road environments and emergencies. The information classification method provided in the embodiments of this application can improve user experience and trust. When the system can more accurately estimate confidence, users will have greater trust in the system's decisions, thereby improving the user experience. For example, in the field of medical diagnosis, accurate confidence estimation can help doctors better judge the reliability of diagnostic results and thus formulate more appropriate treatment plans.

[0137] The information classification method provided in the embodiment of the present application is also a method for solving the confidence estimation problem. Specifically, first, a classification network (classification model) is trained, which includes a feature extractor for extracting feature representation h(x) and a classifier f(x) for classification based on h(x); secondly, for the test sample x′, h(x) is used to calculate the feature representation distance d(x, x′) between the test sample x′ (information to be classified) and the training sample x in the training set as the weight for estimating confidence; finally, the true class probability p(y) of the samples surrounding the test sample (training samples in the training set) is used to represent confidence, and the confidence of the test sample (target confidence) is estimated according to the weight d(x, x′). This method makes full use of the true class probability information of the surrounding samples, reduces the impact of overconfidence, and thus improves the accuracy of confidence estimation.

[0138] The following is an example of an application scenario of autonomous driving.

[0139] Target recognition is a crucial link in the autonomous driving system, and the credibility of the target recognition results directly affects the safety performance of the autonomous driving system. In autonomous driving scenarios, target recognition errors may lead to serious safety accidents. By adopting the confidence classification method proposed in the embodiment of this application (confidence estimation method based on true class probability), the confidence of the test sample can be estimated more accurately, significantly improving the credibility and accuracy of target recognition, thereby improving the safety and stability of the entire autonomous driving system.

[0140] Figure 7 This is a schematic diagram of the target recognition process in the automatic driving process provided by the embodiment of the present application. Figure 7 As shown, during the autonomous driving process, the following interactive steps S301 to S302 are included:

[0141] Step S301: The vehicle-mounted autonomous driving client sends a test sample to the server.

[0142] Here, vehicles equipped with autonomous driving technology can be equipped with an onboard autonomous driving client. While driving, the vehicle can identify unknown objects on the road ahead to ensure autonomous driving safety. Test samples are unknown objects in the vehicle's path that require target identification, such as other vehicles, streetlights, obstacles, and puddles. After the vehicle's sensors detect the test samples in real time, the onboard autonomous driving client can send them to the server.

[0143] In step S302, the server determines the prediction result of the test sample through the classification model and returns it to the on-board autonomous driving client.

[0144] Here, the classification model is a model that is pre-trained based on a training set and is used for autonomous driving target recognition. The classification model may include a feature extractor and a classifier. The purpose of the feature extractor is to extract useful feature representations from the input data to facilitate subsequent classification tasks. The feature extractor can be a deep neural network (such as a convolutional neural network or a recurrent neural network) or another type of machine learning model. h(x) = feature extractor (x), and the function h(x) is used to represent the feature extractor (feature extraction layer), where x represents the input sample and h(x) represents the feature representation extracted from the input sample x. The task of the classifier (classification layer) is to classify the input sample according to the feature representation h(x) extracted by the feature extractor. The classifier can be a machine learning model such as a linear classifier, a support vector machine, or a decision tree. f(x) = classifier(h(x)), the classifier is represented by the function f(x), where x represents the input sample, and f(x) represents the result of classifying the input sample x according to the feature representation h(x) extracted by the feature extractor (i.e., a vector: [predicted probability of category 1, predicted probability of category 2, ..., predicted probability of category k]).

[0145] The following describes the training process of the classification model.

[0146] During the training process, a training dataset is used, which contains multiple input samples x i (training sample) and the corresponding true class label y i (label category). The goal is to minimize the loss function of the classification model on the training data set. The loss function can be cross entropy loss, mean squared error loss, etc. Use gradient descent methods (such as stochastic gradient descent, Adam (Adaptive Moment Estimation) optimizer, etc.) to update the parameters of the feature extractor and classifier to minimize the loss function. Classification loss function = L(f(x i ),y i ), where L represents the loss function (cross entropy loss), f(x i ) represents the classifier's response to the input sample x i The result of classification, y i Represents the input sample x i The true class label.

[0147] The server stores a pre-trained classification model, and the server can use the classification model to determine the prediction result of the test sample. First, the feature representation distance between the test sample x' and the sample in the training set can be calculated (the weight of estimated confidence). For each test sample x', the feature extractor h(x) can be used to calculate its feature representation h(x'). Calculate the feature representation h(x') of the test sample x' and each sample x in the training set. iThe feature representation h(x i ). Here, d(x, x′) is used to represent the feature representation distance. h(x′) = feature extractor processing (x′), d(x i ,x′)=distance metric(h(x i ),h(x′)). The distance metric can be Euclidean distance, cosine similarity, Mahalanobis distance, etc. Taking Euclidean distance as an example, the calculation formula of Euclidean distance is as follows: d(x i ,x′)=||h(x i )-h(x′)||2, where ||·||2 represents the L2 norm, which is used to calculate the Euclidean distance between two feature representations. The calculated distance between the test sample x′ and each sample x in the training set is i After the feature representation distance is obtained, the weight of the estimated confidence can be calculated based on the distance.

[0148] Then, the true class probability p(y) (label class probability) of the samples around the test sample is used to represent the confidence, and the confidence of the test sample is estimated based on the weight. The feature representation distance d(x, x′) is converted into a weight (confidence weight) using a normalized exponential function (Softmax function). The weight calculation formula is as follows: i =exp(-d(x i ,x′) / σ) / Σ j exp(-d(x j ,x′) / σ), where w i Represents sample x i The weight of , σ is a positive number used to control the scale of weight distribution, ∑ j represents the sum of all samples in the training set.

[0149] It should be noted that the above steps of calculating the feature representation distance (feature distance) and weight can also be replaced by training an additional confidence model. The input of the confidence model is the test sample, and the output is the weight of each training sample. The confidence model can be any deep neural network (such as a convolutional neural network or a recurrent neural network) or other types of machine learning models. This method can reduce the amount of computation in the information process, improve the efficiency of information classification, and thus improve the real-time performance of target recognition during autonomous driving.

[0150] In one embodiment, after obtaining each sample x i After the weights are calculated, each sample x i The true class probability of , and the weighted sum of their corresponding weights, are used to obtain the confidence of the test sample. p(y|x′)=∑ i w i *p(y|x i), where p(y|x′) is the confidence of the test sample x′, p(y|x i ) is the sample x i The true class probability is calculated based on experience. A confidence threshold is set (usually 0.98). If the confidence of the test sample is less than the confidence threshold, the prediction result of the test sample is determined to be "unreliable prediction". At this time, the server abandons the classification result of the test sample by the classification model and sends an unreliable prediction indication message to the on-board autonomous driving client. When the user sees the unreliable prediction indication message, he can judge the category of the test sample by himself to improve the accuracy of target recognition and thus improve the safety of autonomous driving. Alternatively, if the confidence of the test sample is greater than or equal to the confidence threshold, the prediction result of the test sample is the predicted category output by the classification model. The server sends the predicted category to the on-board autonomous driving client so that the vehicle can perform other autonomous driving execution actions according to the predicted category, such as deceleration.

[0151] In another embodiment, after obtaining each sample x i After the weights are calculated, the confidence that the test sample belongs to category k can be calculated for each category k. p(y=k|x′)=∑ i w i *p(y=k|x i ), where p(y=k|x′) represents the confidence that the test sample x′ belongs to category k, and p(y=k|x i ) represents the training sample x i The true class probability of belonging to class k. Finally, the confidence of the test sample is estimated based on the weighted average true class probability p(y=k|x′). Specifically, the class with the largest confidence value is selected as the prediction result for the test sample x′. y′=argmax k p(y = k | x′), where y′ represents the predicted class of the test sample x′. Through the above steps, the true class probabilities and feature representation distances of the training set samples can be used to estimate the confidence of the test sample, thus addressing the overconfidence problem. This approach can improve the accuracy of the confidence estimation of the classification model on the test sample, helping to improve the performance of the model in practical applications.

[0152] By using the true class probabilities of surrounding samples instead of the maximum class probability to represent confidence, the present embodiment mitigates the effects of overconfidence, resulting in more accurate target recognition, reduced risk of misidentification, and improved generalization capabilities of autonomous driving systems. In real-world road environments, autonomous driving systems may encounter a variety of complex and changing situations. By using feature representation distance as a weight for estimating confidence, the present embodiment enables the classification model to better adapt to different scenarios, improving generalization capabilities and enabling the autonomous driving system to maintain high recognition accuracy even in unknown situations. The present embodiment also helps improve the real-time performance of autonomous driving systems. Traditional confidence estimation methods are affected by overconfidence in surrounding samples and may require more computing resources to correct these errors, thus affecting the real-time performance of autonomous driving systems. However, the present embodiment reduces computational complexity through more accurate confidence estimation, improving the real-time performance of autonomous driving systems. Applying the information classification method provided by the present embodiment to products such as autonomous driving can bring significant positive effects, such as improving the credibility of target recognition, generalization capabilities, real-time performance, reliability, and user experience, and has broad application prospects and practical value.

[0153] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0154] The following continues to describe the exemplary structure of the information classification device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the information classification device 455 of the memory 450 may include: an information acquisition module 4551, which is used to obtain the information to be classified and a preset training set in response to an information classification request; a prediction module 4552, which is used to determine the classification prediction result of the information to be classified using a pre-trained classification model; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and the label category of each training sample; a label probability determination module 4553, which is used to determine the label class probability of each training sample in the training set; the label class probability is the probability that the training sample belongs to the label category predicted by the classification model; a confidence determination module 4554, which is used to determine the target confidence of the information to be classified based on the label class probability of each training sample; a category determination module 4555, which is used to determine the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

[0155] In some embodiments, the prediction module is also used to: use the feature extraction layer of the classification model to extract features of the information to be classified, and obtain a first feature vector of the information to be classified; use the classification layer of the classification model to classify the first feature vector to obtain the predicted probability of the information to be classified in each preset category; arrange the predicted probability of the information to be classified in each preset category in a preset category order to form a vector, and determine it as the classification prediction result of the information to be classified.

[0156] In some embodiments, the confidence determination module is further configured to: determine a confidence weight for each training sample; and determine a target confidence for the information to be classified based on the confidence weight and label class probability of each training sample.

[0157] In some embodiments, the confidence determination module is further used to: determine the feature distance between the information to be classified and each training sample; normalize the feature distance between the information to be classified and each training sample to obtain a corresponding confidence weight for each training sample.

[0158] In some embodiments, the confidence determination module is also used to: use the feature extraction layer of the classification model to extract features from each training sample to obtain a second feature vector for each training sample; obtain a first feature vector of the information to be classified, the first feature vector being obtained by extracting features from the information to be classified through the feature extraction layer of the classification model; and determine the feature distance between the information to be classified and each training sample based on the first feature vector and the second feature vector of each training sample.

[0159] In some embodiments, the confidence determination module is further used to: for each training sample, multiply the confidence weight by the label class probability to obtain a weighted probability; and add the weighted probabilities of each training sample to obtain the target confidence of the information to be classified.

[0160] In some embodiments, the category determination module is also used to: obtain a preset confidence threshold; when the target confidence of the information to be classified is greater than or equal to the confidence threshold, the preset category corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified; when the target confidence of the information to be classified is less than the confidence threshold, generate a reminder message and send the reminder message to the terminal; the reminder message is used to remind the target object to determine the target category of the information to be classified.

[0161] In some embodiments, the confidence determination module is also used to: for each preset category, based on the label category of each training sample, obtain at least one training sample under the preset category from the training set; multiply the confidence weight and label class probability of each training sample under the preset category to obtain the corresponding weighted category probability of each training sample; add the weighted category probabilities of all training samples under the preset category to obtain the target confidence of the information to be classified under the preset category.

[0162] In some embodiments, the category determination module is also used to: obtain the target confidence of the information to be classified under each preset category to obtain a target confidence set; determine the maximum target confidence from the target confidence set; if the preset category corresponding to the maximum target confidence is consistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified; if the preset category corresponding to the maximum target confidence is inconsistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum target confidence is determined as the target category of the information to be classified.

[0163] In some embodiments, the information classification device also includes a training module, which is used to: utilize the feature extraction layer of the classification model to be trained to perform feature extraction processing on the training samples in the training set to obtain a third eigenvector of the training samples; utilize the classification layer of the classification model to be trained to perform classification processing on the third eigenvector of the training samples to obtain a training classification result of the training samples; based on the training classification results and label categories of the training samples, iteratively update the model parameters of the classification model to be trained until the classification model to be trained converges to obtain a pre-trained classification model.

[0164] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the information classification method described in the embodiment of the present application.

[0165] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or computer programs, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the information classification method provided by the embodiment of the present application, for example, Figure 3 Shown is the information classification method.

[0166] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0167] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0168] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0169] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0170] In summary, the embodiments of the present application can improve the confidence estimation accuracy of the model on the test samples, thereby improving the model's predictive ability and generalization ability, improving the accuracy of information classification, and further improving user experience and trust, bringing significant beneficial effects to various application fields.

[0171] The above are merely examples of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An information classification method, characterized in that: The method comprises: In response to an information classification request, obtaining information to be classified and a preset training set; Determine a classification prediction result of the information to be classified using a pre-trained classification model; the pre-trained classification model is trained based on the training set, and the training set includes multiple training samples and a label category for each training sample; Determining a label class probability for each of the training samples in the training set; the label class probability is a probability that the training sample belongs to the label class as predicted by the classification model; Determining the target confidence of the information to be classified based on the label class probability of each of the training samples; The target category of the information to be classified is determined based on the target confidence of the information to be classified and the classification prediction result.

2. The method according to claim 1, characterized in that The method of using a pre-trained classification model to determine a classification prediction result of the information to be classified includes: Using the feature extraction layer of the classification model, extracting features from the information to be classified to obtain a first feature vector of the information to be classified; Using the classification layer of the classification model, classify the first feature vector to obtain a predicted probability of the information to be classified under each preset category; The predicted probability of the information to be classified under each preset category is arranged in a preset category order to form a vector, which is determined as the classification prediction result of the information to be classified.

3. The method according to claim 1, characterized in that Determining the target confidence of the information to be classified based on the label class probability of each training sample includes: Determining a confidence weight for each of the training samples; Based on the confidence weight of each training sample and the label class probability, a target confidence of the information to be classified is determined.

4. The method according to claim 3, characterized in that Determining the confidence weight of each training sample includes: Determining a characteristic distance between the information to be classified and each of the training samples; Normalizing the characteristic distance between the information to be classified and each of the training samples to obtain a corresponding confidence weight for each of the training samples.

5. The method according to claim 4, characterized in that Determining the characteristic distance between the information to be classified and each of the training samples includes: Using the feature extraction layer of the classification model, extract features from each of the training samples to obtain a second feature vector for each of the training samples; Obtaining a first feature vector of the information to be classified, where the first feature vector is obtained by performing feature extraction on the information to be classified through a feature extraction layer of the classification model; Based on the first feature vector and the second feature vector of each training sample, a feature distance between the information to be classified and each training sample is determined.

6. The method according to claim 3, characterized in that Determining the target confidence of the information to be classified based on the confidence weight of each training sample and the label class probability includes: For each of the training samples, multiplying the confidence weight by the label class probability to obtain a weighted probability; The weighted probability of each training sample is added together to obtain the target confidence of the information to be classified.

7. The method according to claim 1, characterized in that The determining the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result includes: Get the preset confidence threshold; When the target confidence of the information to be classified is greater than or equal to the confidence threshold, determining the preset category corresponding to the maximum prediction probability in the classification prediction results as the target category of the information to be classified; When the target confidence of the information to be classified is less than the confidence threshold, a reminder message is generated and sent to the terminal; the reminder message is used to remind the target object to determine the target category of the information to be classified.

8. The method according to claim 3, characterized in that Determining the target confidence of the information to be classified based on the confidence weight of each training sample and the label class probability includes: For each preset category, based on the label category of each training sample, obtaining at least one training sample under the preset category from the training set; Multiplying the confidence weight of each training sample under the preset category by the label class probability to obtain the weighted category probability of each training sample; The weighted category probabilities of all training samples under the preset category are added together to obtain the target confidence of the information to be classified under the preset category.

9. The method according to claim 1, characterized in that The determining the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result includes: Obtaining the target confidence of the information to be classified under each preset category to obtain a target confidence set; determining a maximum target confidence from the target confidence set; If the preset category corresponding to the maximum target confidence is consistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, then the preset category corresponding to the maximum prediction probability in the classification prediction result is determined as the target category of the information to be classified; If the preset category corresponding to the maximum target confidence is inconsistent with the preset category corresponding to the maximum prediction probability in the classification prediction result, the preset category corresponding to the maximum target confidence is determined as the target category of the information to be classified.

10. The method according to any one of claims 1 to 9, characterized in that Before determining the classification prediction result of the information to be classified using the pre-trained classification model, the method further includes: Using a feature extraction layer of a classification model to be trained, performing feature extraction processing on the training samples in the training set to obtain a third feature vector of the training samples; Using the classification layer of the classification model to be trained, classify the third eigenvector of the training sample to obtain a training classification result of the training sample; Based on the training classification results and label categories of the training samples, the model parameters of the classification model to be trained are iteratively updated until the classification model to be trained converges, thereby obtaining the pre-trained classification model.

11. An information classification device, characterized in that: The device comprises: An information acquisition module, configured to obtain information to be classified and a preset training set in response to an information classification request; A prediction module, configured to determine a classification prediction result of the information to be classified using a pre-trained classification model; the pre-trained classification model is trained based on the training set, the training set including a plurality of training samples and a label category for each training sample; A label probability determination module is used to determine the label class probability of each training sample in the training set; the label class probability is the probability that the training sample belongs to the label class predicted by the classification model; A confidence determination module, configured to determine a target confidence of the information to be classified based on the label class probability of each training sample; The category determination module is used to determine the target category of the information to be classified based on the target confidence of the information to be classified and the classification prediction result.

12. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the information classification method according to any one of claims 1 to 10 when executing the computer-executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the information classification method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the information classification method according to any one of claims 1 to 10 is implemented.

Images