Classification model training method, related method, device, equipment and medium

Abstract

The invention discloses a classification model training method, a related method, a device, equipment and a medium. Comprising the steps of obtaining an unknown prediction result, a newly-added prediction result and a known prediction result for sample features of a first sample object based on a preset class increasing model; determining classification loss according to a target classification result and a category label obtained by classification of the unknown prediction result, the newly added prediction result and the known prediction result; a class increasing classification model is obtained through classification loss training; inputting the second sample object into the known class classification model and the increased class classification model, and optimizing the known class classification result according to the increased class classification result and the correction parameter; and determining fine tuning loss according to the known class optimization result, the increased class classification result and the class label, performing fine tuning on the correction parameter according to the fine tuning loss to obtain a fine-tuned correction parameter, and obtaining a target classification model based on the fine-tuned correction parameter, the known class classification model and the increased class classification model. According to the method, the classification accuracy can be still improved under the condition that the classification categories are newly added.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a classification model training method and related methods, devices, equipment and media. Background Art

[0002] Continuous learning stems from the need to apply machine learning in dynamic environments, especially when data continues to increase. These methods have been widely used in fields such as natural language processing and autonomous driving, helping models maintain stability and accuracy in real-time learning. In addition, the application of continuous learning in various classification tasks is also growing, demonstrating its flexibility and effectiveness in dealing with constantly changing data sets.

[0003] For example, in image classification, the model can recognize new categories through continuous learning while maintaining the ability to recognize old categories. In medical analysis, doctors may encounter new types of diseases, and continuous learning can help the model quickly adapt to new consultation needs. In text classification, as social media and news content change rapidly, the model can be updated in time through continuous learning to reflect emerging topics and emotions.

[0004] At present, the technologies related to continuous learning mainly include replay-based, regularization-based and model-based methods. However, these methods may cause fluctuations in the overall performance of the model during the continuous learning process. For example, the classification model is prone to inconsistency in the balance between new and old knowledge, which affects the classification of known categories and is not suitable for long-term incremental tasks, resulting in low accuracy of the final classification results. Summary of the invention

[0005] The embodiments of the present application provide a classification model training method and related methods, devices, equipment and media, which can improve the accuracy of classification results output by the target classification model when a new classification category is added.

[0006] According to one aspect of the present application, a method for training a classification model is provided, the method comprising: obtaining a first sample object, extracting features of the first sample object based on a preset augmented class model, and performing unknown category prediction, newly added category prediction and known category prediction according to the extracted sample features, to obtain corresponding unknown prediction results, newly added prediction results and known prediction results; performing classification according to the unknown prediction results, the newly added prediction results and the known prediction results, and determining a classification loss based on a target classification result obtained by classification and a category label corresponding to the first sample object; and training the preset augmented class model by using the classification loss. , and obtain a trained augmented class classification model; obtain a second sample object, and input the second sample object into the known class classification model and the augmented class classification model for classification, and optimize the known class classification result obtained by classification based on the augmented class classification result and the correction parameter obtained by classification to obtain a known class optimization result; determine the fine-tuning loss according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and fine-tune the correction parameter based on the fine-tuning loss to obtain the fine-tuned correction parameter, so as to obtain the target classification model based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model.

[0007] According to one aspect of the present application, a classification method is provided, the method comprising: obtaining a target object to be classified; inputting object information corresponding to the target object into a target classification model, the target classification model being obtained according to the training method of the above-mentioned classification model, classifying the object information by a known class classification model in the target classification model to obtain a first target classification result, and classifying the object information by the augmented class classification model to obtain a second target classification result; correcting the first target classification result according to a correction parameter in the target classification model to obtain a target correction result; and determining the object category corresponding to the target object based on the target correction result and the second target classification result.

[0008] According to one aspect of the present application, a training device for a classification model is provided, which includes: a classification prediction module, a loss determination module, a model training module, a classification optimization module and a parameter fine-tuning module.

[0009] A classification prediction module, used to obtain a first sample object, extract features of the first sample object based on a preset class-increasing model, and perform unknown category prediction, newly added category prediction and known category prediction according to the extracted sample features, to obtain corresponding unknown prediction results, newly added prediction results and known prediction results;

[0010] a loss determination module, configured to classify the unknown prediction result, the newly added prediction result, and the known prediction result, and determine the classification loss based on the target classification result obtained by the classification and the category label corresponding to the first sample object;

[0011] A model training module, used for training the preset incremental classification model through the classification loss to obtain a trained incremental classification model;

[0012] a classification optimization module, used for obtaining a second sample object, inputting the second sample object into the known class classification model and the increased class classification model for classification, and optimizing the known class classification result obtained by classification based on the increased class classification result and the correction parameter, to obtain a known class optimization result;

[0013] A parameter fine-tuning module is used to determine the fine-tuning loss according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and fine-tune the correction parameter based on the fine-tuning loss to obtain the fine-tuned correction parameter, so as to obtain the target classification model based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model.

[0014] Optionally, the preset incremental classification model includes a feature extraction module and an object classification module, and the sample features include multimodal sample features; the classification prediction module may include a feature extraction unit and a category prediction unit. The feature extraction unit is used to output the multimodal sample information corresponding to the first sample object to the feature extraction module, and based on the feature extraction module, perform feature extraction on the multimodal sample information to obtain multimodal sample features; the category prediction unit is used to input the multimodal sample features into the object classification module, and based on the object classification module, perform unknown category prediction, newly added category prediction and known category prediction on the multimodal sample features, respectively, to obtain unknown prediction results, newly added prediction results and known prediction results corresponding to the first sample object.

[0015] Optionally, the feature extraction unit can be specifically used to: when it is detected that new modal information exists in the multimodal information, perform feature extraction on the new modal information to obtain new modal features; perform feature extraction on known modal information in the multimodal information except the new modal information to obtain multiple known modal features; perform feature fusion on the new modal features and the multiple known modal features to obtain multimodal sample features corresponding to the first sample object.

[0016] Optionally, the loss determination module may include a combination prediction unit, a result classification unit, an uncertainty calculation unit, and a classification loss determination unit. The combination prediction unit is used to perform prediction combination based on the unknown prediction result, the newly added prediction result, and the known prediction result to obtain a combined prediction result; the result classification unit is used to classify the combined prediction result based on a nonlinear activation function to obtain a target classification result corresponding to the first sample object; the uncertainty calculation unit is used to calculate a first uncertainty based on the target classification result and the category label corresponding to the first sample object; the classification loss determination unit is used to calculate the conditional distribution and boundary distribution corresponding to the first sample object based on the first uncertainty, and determine the classification loss by the conditional distribution and the boundary distribution.

[0017] Optionally, the uncertainty calculation unit can be specifically used for: when the category label corresponding to the first sample object belongs to a newly added category, performing a negative logarithm calculation based on the added category confidence under the newly added category in the target classification result to obtain a first uncertainty; when the category label corresponding to the first sample object belongs to a known category, performing a negative logarithm calculation based on the known category confidence under the known category in the target classification result and the number of known categories to obtain the first uncertainty.

[0018] Optionally, the classification loss determination unit may include a conditional subunit, a joint subunit, and a loss subunit. The conditional subunit is used to determine the conditional distribution and the boundary distribution corresponding to the first sample object according to the first uncertainty; the joint subunit is used to determine the joint distribution corresponding to the first sample object through the conditional distribution and the boundary distribution; the loss subunit is used to determine the classification loss by obtaining an optimization function based on the joint distribution through gradient equivalent optimization.

[0019] Optionally, the loss subunit can be specifically used to: perform feature extraction on the category label corresponding to the first sample object to obtain category label features; perform feature extraction on each modal information of the first sample object to obtain modal features corresponding to each modal information; construct a target comparison function based on the modal features corresponding to each modal information and the category label features; perform gradient equivalent optimization on the joint distribution to obtain an optimization function, and determine the classification loss based on the optimization function and the target comparison function.

[0020] Optionally, the model training module can be specifically used to: iteratively train the feature extraction module and the object classification module in the preset classification model through the classification loss until the training end condition is met, so as to obtain a trained enhanced classification model.

[0021] Optionally, the classification optimization module may include a model classification unit and a result optimization unit. The model classification unit is used to input the second sample object into the known class classification model for classification to obtain a known class classification result, and input the second sample object into the augmented class classification model for classification to obtain an augmented class classification result; and the result optimization unit is used to classify and optimize the second known class confidence under the known class in the known class classification result based on the correction parameter and the first known class confidence under the known class in the augmented class classification result to obtain a known class optimization result.

[0022] Optionally, the correction parameter includes a scaling parameter and a bias parameter, and the result optimization unit can be specifically used to: multiply the scaling parameter and the first known class confidence under the known category in the incremental classification result to obtain a scaling product term; add the scaling product term and the bias parameter to obtain a correction term; add the correction term and the second known class confidence under the known category in the known class classification result to obtain a known class optimization result.

[0023] Optionally, the parameter fine-tuning module may include a known determination unit, a newly added determination unit, and a parameter fine-tuning unit. The known determination unit is used to determine the second uncertainty based on the negative logarithm of the known class optimization result when the class label corresponding to the second sample object belongs to a known class; the newly added determination unit is used to determine the second uncertainty based on the negative logarithm of the increased class confidence under the newly added class in the increased class classification result when the class label corresponding to the second sample object belongs to a newly added class; the parameter fine-tuning unit is used to perform parameter fine-tuning on the correction parameter based on the fine-tuning loss determined by the second uncertainty to obtain the fine-tuned correction parameter.

[0024] Optionally, the parameter fine-tuning unit can be specifically used to: calculate the fusion condition distribution corresponding to the second sample object according to the second uncertainty, and determine the fine-tuning loss based on the fusion condition distribution; iteratively fine-tune the correction parameter based on the fine-tuning loss until the fine-tuning end condition is met, so as to obtain the fine-tuned correction parameter.

[0025] According to one aspect of the present application, a classification device is provided, which includes: an object acquisition module, an object classification module, a result correction module and a result determination module.

[0026] An object acquisition module is used to acquire a target object to be classified;

[0027] An object classification module is used to input the object information corresponding to the target object into a target classification model, wherein the target classification model is obtained by the training method of the above-mentioned classification model, classify the object information by a known class classification model in the target classification model to obtain a first target classification result, and classify the object information by the augmented class classification model to obtain a second target classification result;

[0028] A result correction module, used to correct the first target classification result according to the correction parameters in the target classification model to obtain a target correction result;

[0029] A result determination module is used to determine the object category corresponding to the target object based on the target correction result and the second target classification result.

[0030] Optionally, the object information includes multimodal object information; the object classification module can be specifically used to: extract features from the object information based on a known class classification model in the target classification model to obtain a first multimodal feature, and classify according to the first multimodal feature to obtain a first target classification result; extract features from the object information based on an incremental classification model in the target classification model to obtain a second multimodal feature, and classify according to the second multimodal feature to obtain a second target classification result.

[0031] According to one aspect of the present application, a computer-readable storage medium is provided, which stores a computer program, wherein when the computer program is executed by a processor, the above-mentioned classification model training method and related methods are executed.

[0032] According to one aspect of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is called by the processor, the training method and related methods of the above-mentioned classification model are executed.

[0033] According to one aspect of the present application, a computer program product is provided, which includes a computer program stored in a storage medium; a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned classification model training method and related methods.

[0034] The present application can obtain a first sample object, extract features of the first sample object based on a preset augmented class model, and perform unknown category prediction, newly added category prediction and known category prediction respectively according to the extracted sample features, and obtain corresponding unknown prediction results, newly added prediction results and known prediction results, and further, classify according to the unknown prediction results, newly added prediction results and known prediction results, and determine the classification loss based on the target classification result obtained by classification and the category label corresponding to the first sample object, and then, train the preset augmented class model through the classification loss to obtain a trained augmented class classification model. Thus, an augmented class classification model can be obtained by performing separate training for the newly added category, which can effectively avoid the classification conflict between the known category and the newly added category caused by the coupling of multiple models, compared with the related technology of directly adding a new model to the original known category classification model, thereby improving the accuracy of subsequent classification.

[0035] Furthermore, a second sample object can be obtained and input into the known class classification model and the augmented class classification model for classification, respectively. Based on the augmented class classification result and the correction parameter obtained by classification, the known class classification result obtained by classification is classified and optimized to obtain a known class optimization result. Thus, the known class classification result output by the known class classification model can be optimized to obtain a more accurate known class optimization result. Then, the fine-tuning loss is determined according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and the correction parameter is fine-tuned based on the fine-tuning loss to obtain the fine-tuned correction parameter. Therefore, the target classification model determined based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model can avoid inaccurate classification results caused by classification bias.

[0036] In this way, the incremental classification model is first obtained by separate training, and then the target classification model is determined based on the fine-tuned correction parameters, the known class classification model and the incremental classification model. The two-stage continuous learning can avoid classification conflicts between different classification models. The incremental classification model can learn the classification boundary between the known class classification and the newly added class classification through the known class prediction, so that the classification accuracy of the known class classification model for the known class will not be affected by the incremental classification model. In addition, the incremental classification model can also weaken the influence of the characteristics of the unknown class on the newly added class classification through the unknown class prediction. Therefore, the target classification model can still maintain high recognition performance for the newly added class during the classification process, while reducing the influence of the unknown class on the final classification result, thereby further improving the accuracy of the final classification result.

[0037] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A schematic diagram of a system architecture provided in an embodiment of the present application is shown.

[0040] Figure 2 A schematic diagram of system deployment provided in an embodiment of the present application is shown.

[0041] Figure 3 An application scenario diagram provided by an embodiment of the present application is shown.

[0042] Figure 4 Another application scenario diagram provided by an embodiment of the present application is shown.

[0043] Figure 5 A flow chart of a classification model training method provided in an embodiment of the present application is shown.

[0044] Figure 6 A flow chart of a classification method provided in an embodiment of the present application is shown.

[0045] Figure 7 A framework diagram of a target classification model provided in an embodiment of the present application is shown.

[0046] Figure 8 A flow chart of a sorting method provided in an embodiment of the present application is shown.

[0047] Fig. 9 A scene diagram of a sorting method provided in an embodiment of the present application is shown.

[0048] Fig.10 It is a module block diagram of a classification model training device provided in an embodiment of the present application.

[0049] Fig.11 It is a module block diagram of a classification device provided in an embodiment of the present application.

[0050] Fig.12It is a module block diagram of a computer device provided in an embodiment of the present application.

[0051] Fig.13 It is a module block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The implementation methods described with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0053] In some processes described in the specification, claims and the above drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, the descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order, sequence or quantity.

[0054] It is worth noting that in the specific implementation of the present application, related data such as multimodal sample information is involved. When the above embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with relevant laws, regulations and standards. For example, when the embodiment of the present application needs to obtain multimodal sample information, a separate license or separate consent for the multimodal sample information can be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining a separate license or separate consent, the necessary related data for the normal operation of the embodiment of the present application can be obtained.

[0055] The training method and related methods of the classification model proposed in this application mainly involve continuous learning, which aims to solve the problem that the model may forget previous knowledge when receiving new data. Its theoretical basis covers a variety of technologies, such as memory enhancement methods, knowledge distillation, and transfer learning, which aims to make the learning process more efficient by saving important information and dynamically adjusting model weights. Through these methods, the model can maintain performance in a changing data environment and flexibly adapt to new tasks. The application areas of continuous learning are very wide, covering many fields such as autonomous driving, medical consultation, and financial forecasting. In autonomous driving, continuous learning allows the vehicle to continuously adjust the algorithm according to new road conditions and driving behaviors, thereby improving safety and performance.

[0056] In actual applications, the model needs to maintain the ability to recognize existing categories when encountering new categories. To meet this challenge, researchers have developed a variety of strategies, such as using memory mechanisms to retain the characteristics of old categories, knowledge distillation methods to maintain the output distribution of old categories, etc. Recent strategies also include regularization-based schemes and dynamic structure-based schemes. Among them, the regularization-based scheme protects old knowledge by adding constraints, but it may lead to the plasticity-stability dilemma. That is, when learning a new task, the model may adjust parameters to adapt to the new data, but at the same time it will affect the previously learned knowledge, thereby reducing the overall performance. This makes it difficult to balance the stability and flexibility of the model when dealing with multiple categories.

[0057] The solution based on dynamic structure is to process new classes by adding sub-models. However, with the increase of sub-models, the entire classification model will become bloated, resulting in an increase in training costs. At the same time, the coupling between sub-models may cause conflicts, destroy the effectiveness of feature representation, and thus mislead the final prediction. Due to these problems, the effect in long-term incremental tasks may not be as expected, resulting in the classification model having a low accuracy of the final classification results. In order to solve the above problems, the inventors have proposed a training method and related methods for the classification model provided in the embodiments of the present application after research. The system architecture of the above-mentioned method involved in this application and the related application scenarios are described below.

[0058] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. Figure 1As shown, the above method provided in the embodiment of the present application can be applied to the system 100. The data acquisition device 110 is used to acquire training data, which is a training sample used for network training, and the training sample includes corresponding multimodal sample information and category labels. For example, the training sample is an express package, and the multimodal sample information is information such as images and texts describing the express package. The category label marks the classification of the express package, and the category label can be manually labeled. After acquiring the training data, the data acquisition device 110 can store the training data in the database 120, and the training device 130 can train the preset neural network based on the training data in the database 120 until the preset neural network meets the training end condition, and obtain the trained target model 101, that is, the target classification model.

[0059] Optionally, the training data maintained in the database 120 does not necessarily all come from the data acquisition device 110, but may also be received from other devices. For example, the execution device 140 may also serve as a data acquisition terminal, directly using the acquired data as new training data and storing it in the database 120. In addition, the training device 130 does not necessarily train the preset neural network based entirely on the training data maintained by the database 120, but may also train the preset neural network based on training data obtained from the cloud or other devices. For example, the training device 130 may obtain the latest sorted package information from the express package sorting platform, including package images and text and other information. The above description should not be used as a limitation on the embodiments of the present application.

[0060] Among them, the training end condition can be: the total loss value of the target loss is less than the preset value, the total loss value of the target loss is within the threshold range, or the number of training times reaches the preset number of times, etc. The target model can be a deep neural network or a network composed of multiple neural networks. Exemplarily, the target model 101 is a target classification model capable of multimodal classification. The target model 101 can be composed of network modules such as a convolutional neural network (CNN), a long short-term memory network (Long Short-Term Memory), and a Transformer that can extract multimodal information and perform classification tasks, which are not limited here.

[0061] When the processing module 141 of the execution device 140 performs calculations and other related processing, the execution device 140 can call data, programs, etc. in the data storage system 150 for corresponding calculation processing, and store data and instructions such as processing results obtained by the calculation processing in the data storage system 150. For example, the execution device 140 can classify the target objects to be classified using the target model 101, and store the classification results in the data storage system 150.

[0062] The training device 130 and the execution device 140 can be computer devices such as servers or terminals. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), blockchains, and big data and artificial intelligence platforms. The terminal can be a laptop, a tablet, a desktop computer, etc.

[0063] It should be noted that Figure 1 It is only a schematic diagram of a system architecture provided in the embodiment of the present application. The system architecture and application scenarios described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Figure 1 The data storage system 150 in the example is an internal memory relative to the execution device 140. In other cases, the data storage system 150 may also be placed outside the execution device 140.

[0064] See also Figure 2 , Figure 2 Schematic diagram of a system deployment is shown. For example, the data processing method provided in the embodiment of the present application can also be deployed in Figure 2 In the system of C / S (Client / Server) architecture shown in FIG. , the system may include a server, a gateway, the Internet, and devices.

[0065] A server is a computer system that can provide certain services to the server side. Compared with ordinary computer equipment, servers have higher requirements in terms of stability, security, and performance. Figure 2 As shown, the server 210 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a high-performance computer (e.g., a virtual machine), a combination of portions of multiple high-performance computers (e.g., virtual machines), etc.

[0066] A gateway is also called an internetwork connector or a protocol converter. Figure 2 As shown, gateway 220 realizes network interconnection at the transport layer through Internet 230, and is a computer system or device that acts as a conversion. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions.

[0067] Devices can send requests to servers to access or use resources, data, or services on the server. Devices are usually interfaces for users to interact with the system. Users operate on the client and generate requests. Figure 2 As shown, the device may include a smart phone 240a, a notebook computer 240b, a desktop computer 240c, a personal digital assistant (PDA), a server, etc. In addition, it may be a single device or a collection of multiple devices.

[0068] Devices can communicate with the Internet in wired or wireless ways to exchange data. Figure 2 As shown, the smart phone 240a can communicate with the Internet 230 through the wireless router 250. For another example, the message sent by the laptop 240b to the server 210 can be transmitted to the corresponding server 210 through the gateway 220. The message sent by the server 210 to the device 240 can also be transmitted to the corresponding desktop computer 240c through the gateway 220.

[0069] The training method and classification method of the classification model of the embodiment of the present application can be completely implemented on the device, can be completely implemented on the server, or can be implemented partially on the device and the other part on the server.

[0070] As an implementation method, the training method and classification method of the classification model can be completely implemented in the device. For example, the device can be used as a training device to train locally to obtain a target classification model. Then, the device can also be used as an execution device to locally classify the target object to be classified using the target classification model to obtain the object category corresponding to the target object. This implementation method can achieve local intelligence only on the device without the help of a server.

[0071] As another implementation, the training method and classification method of the classification model can be completely implemented on the server. Exemplarily, the server can be used as a training device to train a target classification model. Further, the server can also be used as an execution device to classify the target object to be classified using the target classification model to obtain the object category corresponding to the target object. For example, the device has a need for item sorting tasks. The intelligent item sorting system deployed on the device needs to classify items. At this time, the device can send the item information (such as images, text, etc.) corresponding to the items to be sorted to the server, and then the server uses the target classification model to extract features and classify items for the item information corresponding to the items to be sorted, and returns the classification results obtained by classification to the device, so that the intelligent item sorting system of the device can sort the items to the corresponding logistics thread based on the received classification results.

[0072] As another implementation, the training method and classification method of the classification model can be implemented in part on the device and in part on the server. For example, the server can be used as a training device for the target classification model, and the device can be used as an execution device using the target classification model. For example, the server can obtain the target classification model through classification training and parameter fine-tuning based on the first training sample and the second training sample. Then, the device can deploy the trained target classification model locally, and then directly use the target classification model to perform the classification task.

[0073] It should be noted that Figure 2 This is only a schematic diagram of a system deployment provided by the embodiment of the present application. The system deployment scheme described in the embodiment of the present application is only to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, a device may generally refer to one of multiple devices. This embodiment only refers to Figure 2 The three terminals shown in the figure are used as examples. It is known to those skilled in the art that, with the evolution of system deployment solutions, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0074] The embodiments of the present application can be applied in various scenarios, such as Figure 3 The item sorting scene shown, Figure 4 The smart medical scenarios shown, etc.

[0075] (1) Item sorting scenario.

[0076] Item sorting is a key link in modern logistics and supply chain management, which mainly involves the rapid and efficient classification and organization of large quantities of goods. In item sorting scenarios, automated equipment such as conveyor belts, sorting machines and robots are usually used to improve the speed and accuracy of sorting. Staff need to accurately identify and classify different types of items according to order requirements, which requires a clear understanding of the types, specifications and storage locations of the items. With the rapid development of e-commerce and express delivery industries, the efficiency and accuracy of item sorting are particularly important, which directly affects customer satisfaction and the operating costs of enterprises. Therefore, it is necessary to continuously develop innovative sorting technologies and processes to adapt to changes in market demand and improve overall logistics efficiency and service quality.

[0077] At present, the solution based on manual sorting is simple, convenient and easy to implement, with low technical requirements, but it has the disadvantages of low efficiency, high labor cost and inability to adapt to large-scale sorting tasks. Manual sorting cannot provide sufficient data processing capabilities and response speed, resulting in limited processing efficiency and accuracy. To this end, the training method and related methods of the classification model proposed in this application can be applied to the item sorting scenario to improve the efficiency and accuracy of item sorting. Specifically, a corresponding item sorting system can be developed based on the training method and related methods of the classification model of this application. In the item sorting scenario, there is a situation where new categories of items appear, so the item sorting system should be able to be updated stably and efficiently with the emergence of new categories to adapt to the sorting of new categories of items.

[0078] Exemplarily, the item sorting system provided by the item sorting center may include a system backend deployed on a server 210 and a system frontend 241 deployed on a desktop computer 240. System administrators may add new categories of items, such as category labels and remark information, on the new category interface of the system frontend 241, or may batch upload new categories of items through files. Then, by clicking the Start Adding button, the system frontend 241 sends the generated adding instruction to the server 210 through the network, and the server 210 may train a classification model based on the new category of the added item according to the adding instruction.

[0079] Specifically, the server 210 can obtain training data about the new category from the database, for example, the first sample object and the second sample object, and then the server 210 can obtain the incremental classification model capable of classifying the new category of items based on the first sample object training, and further, the server 210 can fine-tune the correction parameters based on the incremental classification model and the known category classification model through the second sample object, and obtain the target classification model based on the fine-tuned correction parameters, the known category classification model and the incremental classification model. Then, the item sorting system is updated based on the target classification model, so that the item sorting system can accurately classify the items of the known category while accurately classifying the items of the new category.

[0080] (2) Smart medical scenarios.

[0081] With the rapid growth of medical technology and data, in smart medical scenarios, when new consultation-related data continues to emerge, continuous learning can allow the medical consultation model to flexibly update and learn new information while retaining the original knowledge. This method is particularly suitable for consultations on chronic disease management and rare diseases, because data in these areas are usually variable and scarce. To this end, the training method and related methods of the classification model proposed in this application can be applied to smart medical scenarios to improve the accuracy of smart consultations. Exemplarily, smart medical service providers can periodically update the smart consultation system so that the coverage of disease consultations by the smart consultation system gradually increases.

[0082] Specifically, the intelligent medical service provider can obtain disease samples of new categories, which include corresponding multimodal information, such as medical test data, medical imaging data and consultation data, and also include corresponding disease category labels. Based on the disease samples of new categories and disease samples of known categories, the intelligent medical service provider can train a disease classification model that can classify both known categories of diseases and new categories through the training method of the classification model of the present application on the server 210. Furthermore, the intelligent consultation system is updated based on the disease classification model. Furthermore, the user can update the client 241 corresponding to the intelligent consultation system-the medical assistant on the mobile terminal, and then use the updated medical assistant to experience the intelligent consultation service in the intelligent consultation interactive interface.

[0083] It should be noted that Figure 3 and Figure 4 These are only two schematic diagrams of application scenarios provided by the embodiments of the present application. The application scenarios described in the embodiments of the present application are only for the purpose of more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. For example, the training method and related methods of the classification model proposed in this application can also be used in other scenarios where new categories need to be classified. It is known to those of ordinary skill in the art that, with the evolution of application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0084] According to one embodiment of the present application, a classification model training method is provided. The classification model training method can be executed by a computer device (the computer device can be a server, an edge computing device, or other terminal device with certain processing capabilities), and the computer device has at least storage, computing, and communication functions. Figure 5 FIG. 2 shows a flow chart of the training method of the classification model provided in this embodiment. Figure 5 As shown, the training method of the classification model may specifically include:

[0085] Step 110: Obtain a first sample object, extract features of the first sample object based on a preset class-increasing model, and perform unknown category prediction, new category prediction, and known category prediction according to the extracted sample features to obtain corresponding unknown prediction results, new category prediction results, and known prediction results.

[0086] At present, the relevant technology performs classification prediction for newly added categories by adding sub-models. However, with the increase of sub-models, the overall classification model will become bloated, resulting in an increase in training costs. At the same time, the coupling between the sub-models may cause conflicts, destroy the effectiveness of feature representation, and thus mislead the final prediction. To this end, the present application proposes a two-stage classification model training method. First, a new classification model is trained separately for the newly added category, and then the trained new model and the model for classification prediction for known categories are fused together to form a unified target classification model, so that the entire training cost will not increase with the increase of new categories, and alleviate the conflicts that may be caused by the coupling between classification models of different categories, so as to improve the accuracy of classification prediction.

[0087] Among them, the first sample object refers to the sample object used in the stage of separate training for the new classification model, for example, express parcels to be sorted in the item sorting scenario, images to be recognized in the image recognition scenario, and multimodal information in the intelligent medical consultation scenario. The training set to which the first sample object belongs includes sample objects of the newly added category and sample objects of the known category, and the number of samples of the newly added category is much larger than the number of samples of the known category. The preset category-increasing model refers to a model pre-set for category prediction of the newly added category. The preset category-increasing model can be composed of a feature extractor and a classifier, for example, a feature extractor in the form of a convolutional neural network and a classifier in the form of a fully connected layer. Optionally, a nonlinear activation function can also be set to normalize the output of the classifier. In addition, the feature extractor can also be composed of a type of encoder for feature extraction of multimodal information. The classifier can include three classification networks for respectively performing unknown category prediction, newly added category prediction, and known category prediction. Then, the preset category-increasing model can output unknown prediction results, newly added prediction results, and known prediction results.

[0088] This is because in order to prevent the catastrophic forgetting phenomenon that may occur in the classification model during the continuous learning process, that is, to ensure that the classification model can still maintain high recognition performance for the newly added categories. At the same time, to reduce the impact of unknown categories on the final classification, this application proposes to perform unknown category prediction, newly added category prediction and known category prediction on the extracted sample features during the stage of separate training for the new classification model, thereby obtaining the corresponding unknown prediction results, newly added prediction results and known prediction results.

[0089] Considering that in actual classification tasks, classification models must face diversified classification tasks according to changes in product requirements, however, in the relevant technologies of continuous learning, it is difficult for the model to effectively learn multimodal data, thereby limiting the classification model's ability to learn new categories. For example, information from different modalities cannot be effectively shared, resulting in the isolation of knowledge and the inability of the classification model to form a comprehensive understanding. To this end, this application proposes a training method for a two-stage classification model based on multimodality.

[0090] In some embodiments, multimodal sample information corresponding to the first sample object can be obtained, and then feature extraction can be performed on the sample information corresponding to the first sample object based on a preset class-increasing model, and unknown category prediction, new category prediction and known category prediction can be performed respectively according to the extracted sample features to obtain corresponding unknown prediction results, new prediction results and known prediction results. Among them, the preset class-increasing model may include a feature extraction module and an object classification module. The feature extraction module can be used to extract features from the multimodal sample information corresponding to the first sample object to obtain multimodal sample features corresponding to the first sample object. The object classification module is used to perform unknown category prediction, new category prediction and known category prediction according to the multimodal sample features to obtain corresponding unknown prediction results, new prediction results and known prediction results. The specific steps of category prediction may include:

[0091] (1) Outputting the multimodal sample information corresponding to the first sample object to a feature extraction module, and performing feature extraction on the multimodal sample information based on the feature extraction module to obtain multimodal sample features.

[0092] As an implementation method, the feature extraction module may include multiple feature extraction networks, each of which is used to perform feature extraction specifically for one modal information. Exemplarily, the multimodal sample information corresponding to the first sample object may include image modal sample information, text modal sample information, and audio modal sample information. Furthermore, feature extraction of image modal sample information can be performed based on a convolutional neural network to obtain image sample features, feature extraction of audio modal sample information can be performed based on a long short-term memory network to obtain audio sample features, and feature extraction of text modal sample information can be performed based on a Transformer model to obtain text sample features. Furthermore, the features of each modal sample are fused together to obtain multimodal sample features.

[0093] With the continuous enrichment of data sources, single-modality data can no longer meet the needs of many practical applications. For example, in the medical field, it may be necessary to combine images, text, and physiological data to make more accurate medical inquiries. In instant messaging analysis, text, image, and video content need to be considered simultaneously. To this end, this application proposes to continuously learn the newly added modalities so that the model can dynamically adapt to information from different modalities, thereby improving its understanding and reasoning capabilities. In this way, not only can more comprehensive information be obtained, but also the generalization ability and robustness of the model can be improved. In addition, considering that with the introduction of new modalities, the model can use existing knowledge when processing new tasks, reducing the amount of data required to learn new modalities, so that the model can better cope with challenges and improve application effects in complex and dynamic data environments. This application also proposes continuous learning of "increasing modalities" in addition to "increasing categories".

[0094] As another implementation, when it is detected that there is new modal information in the multimodal information, feature extraction can be performed on the new modal information to obtain the new modal features, and feature extraction can be performed on the known modal information in the multimodal information except the new modal information to obtain multiple known modal features, and then the new modal features and the multiple known modal features are feature fused to obtain the multimodal sample features corresponding to the first sample object. Exemplarily, when it is detected that there is new modal information in the multimodal information, a feature extraction network of the corresponding modality can be obtained for the new modal information, and feature extraction can be performed on the new modal information to obtain the new modal features. For example, the new modality is a video frame embedding vector. Further, the video frame embedding vector can be concatenated with the image embedding vector, text embedding vector, and audio embedding vector corresponding to the known modal information as the multimodal sample features corresponding to the first sample object.

[0095] In this way, by adding new modal information, the classification model can avoid dependence on known modal information, so that the classification model can obtain richer feature information, thereby improving the classification model's ability to understand and handle complex problems. In addition, information between different modalities can complement and enhance each other through fusion, promoting the learning and reasoning of the classification model on multimodal data, and also improving the flexibility and comprehensive ability of the classification model, so that it can adapt to more complex application scenarios.

[0096] (2) The multimodal sample features are input into the object classification module. Based on the object classification module, the multimodal sample features are respectively predicted for unknown categories, newly added categories and known categories, and the unknown prediction results, newly added prediction results and known prediction results corresponding to the first sample object are obtained.

[0097] As an implementation, the object classification module may include an unknown prediction network, a newly added prediction network, and a known prediction network. Exemplarily, the unknown prediction network is used to predict unknown categories for multimodal sample features to obtain unknown prediction results, which may indicate the uncertainty that the first sample object belongs to an unknown category. The newly added prediction network is used to predict newly added categories for multimodal sample features to obtain newly added prediction results, which may indicate the probability that the first sample object belongs to the newly added category. The known prediction network is used to predict known categories for multimodal sample features to obtain known prediction results, which may indicate the probability that the first sample object belongs to a known category.

[0098] For example, the training set S to which the first sample object belongs is D t ∪V t ,in, D t Indicates the tth newly added category The corresponding training set, V t Indicates known categories The corresponding training set, x represents the sample information corresponding to the first sample object, such as multimodal information of images, texts, audios or their combination, and y represents the true label corresponding to the first sample object. Among them, the known category is the category that has been identified by the model when performing the classification task, and the newly added category is the newly added category to be identified when the model performs the classification task. The multimodal sample information x corresponding to the first sample object is output to the feature extraction module, and the multimodal sample information is subjected to feature extraction based on the feature extraction module to obtain the multimodal sample features, and the multimodal sample features are input to the object classification module. Based on the object classification module, the multimodal sample features are subjected to unknown category prediction, newly added category prediction and known category prediction respectively, and the unknown prediction result P corresponding to the first sample object can be obtained. unk , New prediction result P new and the known prediction result P kno .

[0099] Step 120: Classify the unknown prediction results, the newly added prediction results, and the known prediction results, and determine the classification loss based on the target classification result obtained by classification and the category label corresponding to the first sample object.

[0100] In some embodiments, a nonlinear activation function may be used to normalize unknown prediction results, newly added prediction results, and known prediction results to obtain a target classification result, and the uncertainty of the first sample object belonging to the corresponding category label is determined based on the target classification result. The uncertainty is used to represent the uncertainty of the sample object belonging to the corresponding category label. Because uncertainty is usually associated with probability distribution, uncertainty can be naturally converted into the form of probability distribution by using a negative log-likelihood function, and probability distribution makes the classification result easy to interpret, and probability distribution can be used to construct the performance of the loss assessment model. Then, the classification loss can be determined according to the uncertainty to perform model training, which may specifically include:

[0101] (1) Based on the unknown prediction results, the newly added prediction results and the known prediction results, a prediction combination is performed to obtain a combined prediction result.

[0102] As an implementation method, the probabilities corresponding to the unknown prediction results, the newly added prediction results, and the known prediction results can be combined together to obtain a combined prediction result in a matrix form. For example, the combined prediction result can be expressed as a matrix M = [P unk P new P kno ], where the size of the matrix M is d×(K+2), P unk represents the probability that the first sample object belongs to an unknown category, P new The size is d×K, which indicates the probability that the first sample object belongs to K newly added categories, P kno Represents the probability that the first sample object belongs to the known category.

[0103] (2) Classify the combined prediction result based on the nonlinear activation function to obtain the target classification result corresponding to the first sample object.

[0104] As an implementation method, the combined prediction result can be input into a nonlinear activation function for normalization to obtain a target classification result corresponding to the first sample object, and the target classification result includes the normalized probability corresponding to each sample category of the first sample object. unk P new P kno ] is input into the softmax activation function to obtain the target classification result g θ (x), where the target classification result g θ (x) can be represented in the form of a vector, and the k+1th element of the vector represents the confidence g that the first sample object belongs to the corresponding category label θ (x)[k].

[0105] (3) Calculate a first uncertainty based on the target classification result and the category label corresponding to the first sample object.

[0106] As an implementation, the first uncertainty refers to the uncertainty that the first sample object belongs to the corresponding category label, and the first uncertainty can be calculated based on the negative logarithm of the confidence that the first sample object belongs to the corresponding category label in the target classification result. This is because uncertainty is usually associated with probability distribution, and by using the negative log-likelihood function, uncertainty can be naturally converted into probability, which makes the result easy to interpret and can be used to construct the performance of the loss assessment model for model training.

[0107] Specifically, when the category label corresponding to the first sample object belongs to a newly added category, a negative logarithm calculation is performed based on the confidence of the newly added category in the target classification result to obtain the first uncertainty. When the category label corresponding to the first sample object belongs to a known category, a negative logarithm calculation is performed based on the confidence of the known category in the known category in the target classification result and the number of categories of the known category to obtain the first uncertainty. The calculation formula of the first uncertainty is as follows:

[0108]

[0109] Among them, U θ (x, y) represents the first uncertainty corresponding to the first sample object (x, y), δ represents the surjective function that can map the label y to the corresponding category index, To add a new type, can be mapped to category indices by δ: 1, 2, …, K. g θ (x)[δ(y)] indicates that the predicted category label of the first sample object x is a new category The confidence of the true category label y. θ (x)[0] indicates that the predicted category label of the first sample object x belongs to any known category Because the supervision of known categories is limited (the number of known category samples in the training set is small), in order to make the uncertainty of the preset augmented model for the known category prediction greater, the uncertainty of the preset augmented model for the known category prediction is expressed as -log(g θ (x)[0] / M), where M is the number of known categories.

[0110] (4) Calculate the conditional distribution and the boundary distribution corresponding to the first sample object based on the first uncertainty, and determine the classification loss by the conditional distribution and the boundary distribution.

[0111] After obtaining the first uncertainty, the conditional distribution and boundary distribution corresponding to the first sample object can be calculated based on the probability expression of the uncertainty, wherein the conditional distribution corresponding to the first sample object reflects the association between the first sample object and a given category label, that is, the probability of belonging to a given category label. The boundary distribution corresponding to the first sample object reflects the overall probability of the first sample object under all given category labels.

[0112] As an implementation method, the conditional distribution and boundary distribution corresponding to the first sample object can be determined according to the first uncertainty, and the joint distribution corresponding to the first sample object can be determined through the conditional distribution and the boundary distribution, and then the optimization function is obtained based on the joint distribution through gradient equivalent optimization to determine the incremental classification loss. For example, the probability expression based on uncertainty is:

[0113]

[0114] Among them, the above The conditional probability expression representing uncertainty, the above Represents the boundary probability expression of uncertainty, and then, according to the conditional probability expression and boundary probability expression of uncertainty, calculates the conditional distribution and boundary distribution corresponding to the first sample object:

[0115]

[0116] Furthermore, since the conditional distribution corresponding to the first sample object usually leads to overconfident classification predictions, in order to avoid this situation, the joint distribution corresponding to the first sample object can be determined by the conditional distribution and the boundary distribution, and then the optimization function is obtained based on the joint distribution through gradient equivalent optimization to determine the incremental classification loss. Exemplarily, the expected formula for the joint distribution corresponding to the first sample object is as follows:

[0117]

[0118] Joint Distribution The expectation of can not only encourage the model to learn to distinguish all known categories, but also help model the input distribution, making the model sensitive to the drift of the input distribution. Therefore, when unknown categories appear, it can help the model mitigate overconfident predictions and reduce their impact on the final prediction. The expectation of is difficult to be directly used for the calculation of gradients in the model training phase, so the gradient equivalent optimization of the joint distribution can be performed to obtain the optimization function as the incremental classification loss. The formula for the incremental classification loss is as follows:

[0119]

[0120] in, The preset category-increasing model can accurately distinguish the newly added categories and establish a clear decision boundary between the newly added categories and the known categories. The prediction of known categories can better utilize the shared structure of all known categories and reduce the risk of overfitting. Overfitting is usually caused by insufficient training samples of known categories. Therefore, the preset category-increasing model can be better compatible with the prediction of known categories while predicting the newly added categories. The preset augmentation model can be encouraged to retain a certain degree of confidence for unknown categories, so that when unknown categories appear, their impact on the final classification prediction can be weakened. In this way, by being compatible with the prediction of known categories, the catastrophic forgetting of old knowledge is avoided, and the impact of unknown category predictions on the final prediction is weakened, thereby improving the accuracy of the final classification.

[0121] Considering that the extraction of multimodal features usually needs to be able to handle the redundancy and complementarity of multimodal information, it is necessary to design a method that ensures that the common information of different modalities can be shared while maintaining the effective extraction of unique features of each modality. To this end, this application proposes a joint contrastive learning and continuous learning loss for joint training of the preset augmentation model. Through contrastive learning, the preset augmentation model can capture the relationship between modalities, enhance the learning of multimodal information by the preset augmentation model, especially for the newly acquired modal information, use the contrast mechanism between modalities to extract shared features, and then conduct joint continuous learning based on the shared features, thereby improving the accuracy of model classification.

[0122] As an implementation method, feature extraction can be performed on the category label corresponding to the first sample object to obtain the category label feature, and feature extraction can be performed on each modal information of the first sample object to obtain the modal feature corresponding to each modal information. Further, according to the modal feature and category label feature corresponding to each modal information, a target comparison function is constructed. The formula of the target comparison function is as follows:

[0123]

[0124] Where sim(·) represents the similarity. The modal features corresponding to each modal information of the first sample object are fused to obtain the fused modal features z, w i The class label feature corresponding to the real class label of the first sample object is represented. Further, the joint distribution is optimized by gradient equivalent to obtain an optimization function, and the classification loss is determined according to the optimization function and the target comparison function, and the formula is as follows:

[0125] L new =L+L ′

[0126] Among them, L newrepresents the classification loss, L represents the optimization function, and L ′ Represents the target contrast function formula. In this way, based on the incremental classification loss determined according to the optimization function and the target contrast function, the preset incremental model is trained so that the preset incremental model can capture the modal relationship and enhance the learning of multimodal information by the preset incremental model, especially the newly acquired modal information. On this basis, classification learning is performed in combination with continuous learning, thereby improving the accuracy of the final classification.

[0127] Step 130: Train the preset incremental classification model through classification loss to obtain a trained incremental classification model.

[0128] As an implementation method, the feature extraction module and the object classification module in the preset classification model can be iteratively trained by the classification loss until the training end condition is met to obtain the trained incremental classification model. In each iterative training, it is judged whether the classification loss meets the training end condition. Among them, the training end condition can be that the total loss value of the classification loss is less than the preset value, the total loss value of the classification loss no longer changes, or the number of training times reaches the preset number of times. Optionally, an optimizer can be used to optimize the classification loss, wherein the optimizer is an algorithm for updating model parameters to minimize the loss function, and is used to find the local minimum or global minimum of the loss function by iteratively adjusting the model parameters. Common optimizers include Stochastic Gradient Descent (SGD), Adam, Adagrad, RMSprop, etc. Each optimizer has its specific update rules and hyperparameters, such as learning rate, etc. Specifically, the learning rate and the training period (Epoch) can be set based on experimental experience.

[0129] When the classification loss does not meet the training end condition, the parameters of the feature extraction module and the object classification module are adjusted until the classification loss meets the training end condition. When the classification loss meets the training end condition, the training of the feature extraction module and the object classification module is stopped to obtain the trained incremental classification model, which can be used to merge with the known class classification model to form a unified classifier, and then the known categories and the newly added categories can be classified.

[0130] Step 140: Obtain a second sample object, input the second sample object into the known class classification model and the augmented class classification model for classification, and optimize the known class classification result obtained by classification based on the augmented class classification result and correction parameters to obtain a known class optimization result.

[0131] When the training of the preset incremental class model is completed, the trained incremental class classification model can be obtained. Then, the incremental class classification model and the known class classification model can be used as a unified classifier for performing multi-classification tasks of various categories. Among them, the known class classification model is used to classify known categories.

[0132] Considering that the method of determining the classification loss according to the uncertainty formula in the stage of training the preset incremental model will lead to greater task bias. This is because different classification models may produce prediction results with different entropies, and the unified model tends to choose the prediction result with larger entropy. Therefore, in order to effectively alleviate this bias, the uncertainty calculation process can be redefined according to the correction parameters, that is, the known class classification results are optimized by combining the correction parameters with the incremental classification results to obtain the optimized known class optimization results, so that the unified classification model can accurately perform multi-category classification tasks.

[0133] In some embodiments, classification can be performed based on a known class classification model and an augmented class classification model, and based on the augmented class classification results and correction parameters obtained by classification, the known class classification results obtained by classification are optimized to obtain known class optimization results. The specific steps may include:

[0134] (1) Inputting the second sample object into the known class classification model for classification to obtain the known class classification result, and inputting the second sample object into the augmented class classification model for classification to obtain the augmented class classification result.

[0135] As an implementation method, the second sample object refers to a sample object used for fine-tuning the correction parameters in the model fusion stage of the known class classification model and the augmented class classification model. The second sample object is input into the known class classification model for classification to obtain the known class classification result, and the second sample object is input into the augmented class classification model for classification to obtain the augmented class classification result.

[0136] (2) Based on the correction parameter and the first known class confidence under the known class in the incremental classification result, the second known class confidence under the known class in the known class classification result is classified and optimized to obtain the known class optimization result.

[0137] As an implementation, the correction parameter may include a scaling parameter and a bias parameter. The scaling parameter and the first known class confidence under the known class in the incremental classification result are multiplied to obtain a scaling product term, and the scaling product term is added to the bias parameter to obtain a correction term, and then the correction term and the second known class confidence under the known class in the known class classification result are added to obtain a known class optimization result.

[0138] Step 150: Determine the fine-tuning loss according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and fine-tune the correction parameter based on the fine-tuning loss to obtain the fine-tuned correction parameter, so as to obtain the target classification model based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model.

[0139] In some embodiments, the second uncertainty can be determined based on the known class optimization result, the augmented class classification result, and the class label corresponding to the second sample object, and then the fine-tuning loss can be determined based on the second uncertainty so as to fine-tune the correction parameters. Specifically, when the class label corresponding to the second sample object belongs to a known class, the second uncertainty can be determined based on the negative logarithm of the known class optimization result. When the class label corresponding to the second sample object belongs to a newly added class, the second uncertainty can be determined based on the negative logarithm of the augmented class confidence under the newly added class in the augmented class classification result. The formula for the second uncertainty is as follows:

[0140]

[0141] in, represents the second uncertainty corresponding to the second sample object (x, y), Represents the optimization result of known class, Indicates the confidence of the second known class under the known category in the known class classification result, g θ (x)[0] represents the confidence of the first known class under the known category in the incremental classification result, ξ represents the scaling parameter, and ψ represents the bias parameter.

[0142] Further, the correction parameters are fine-tuned based on the fine-tuning loss determined by the second uncertainty to obtain the fine-tuned correction parameters. Specifically, the correction parameters can be fine-tuned based on the fine-tuning loss determined by the second uncertainty to obtain the fine-tuned correction parameters, and the fusion condition distribution corresponding to the second sample object is calculated according to the second uncertainty, as shown in the following formula:

[0143]

[0144] Furthermore, the fine-tuning loss is determined based on the fusion conditional distribution, and the formula of the fine-tuning loss is as follows:

[0145]

[0146] In this way, the correction parameters are iteratively fine-tuned based on the fine-tuning loss until the fine-tuning end condition is met to obtain the fine-tuned correction parameters. In each iterative fine-tuning, it is determined whether the fine-tuning loss meets the fine-tuning end condition. Among them, the fine-tuning end condition can be that the total loss value of the fine-tuning loss is less than a preset value, the total loss value of the fine-tuning loss no longer changes, or the number of training times reaches a preset number of times. Optionally, an optimizer can be used to optimize the fine-tuning loss. Each optimizer has its own specific update rules and hyperparameters, such as learning rate, etc. Specifically, the learning rate and training period can be set based on experimental experience.

[0147] In this way, in the model fusion stage, based on the correction parameters and the confidence of the first known class under the known category in the added class classification result, the second known class confidence under the known category in the known class classification result is classified and optimized, thereby alleviating the task bias of the fused target classification model, that is, the known prediction results output by the added class classification model are used to compensate for the low prediction results (that is, entropy) of the known class classification model for the known category, thereby forming a unified classifier that accurately classifies objects of known categories and newly added categories.

[0148] This embodiment can obtain a first sample object, extract features of the first sample object based on a preset augmented class model, and perform unknown category prediction, newly added category prediction and known category prediction respectively according to the extracted sample features, and obtain corresponding unknown prediction results, newly added prediction results and known prediction results. Further, classification is performed according to the unknown prediction results, newly added prediction results and known prediction results, and the classification loss is determined based on the target classification result obtained by classification and the category label corresponding to the first sample object, and then, the preset augmented class model is trained by the classification loss to obtain a trained augmented class classification model. Therefore, the augmented class classification model is obtained by performing separate training for the newly added category. Compared with the related technology of directly adding a new model to the original known class classification model for mixed training, it can effectively avoid classification conflicts between classification models caused by the coupling of multiple models and improve the accuracy of subsequent classification.

[0149] Furthermore, a second sample object can be obtained and input into the known class classification model and the augmented class classification model for classification, respectively. Based on the augmented class classification result and the correction parameter obtained by classification, the known class classification result obtained by classification is classified and optimized to obtain a known class optimization result. Thus, the known class classification result output by the known class classification model can be optimized to obtain a more accurate known class optimization result. Then, the fine-tuning loss is determined according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and the correction parameter is fine-tuned based on the fine-tuning loss to obtain the fine-tuned correction parameter. Therefore, the target classification model determined based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model can avoid inaccurate classification results caused by classification bias. In this way, the two-stage continuous learning method of first training the incremental classification model separately and then determining the target classification model based on the fine-tuned correction parameters, the known class classification model and the incremental classification model can avoid conflicts between different classification models. In addition, the incremental classification model can learn the classification boundary between the known class classification and the newly added class classification through the known class prediction, so that the classification accuracy of the known class classification model for the known class will not be affected by the incremental classification model. Moreover, the incremental classification model can weaken the influence of the characteristics of the unknown class on the newly added class classification through the unknown class prediction, thereby improving the accuracy of the target classification model corresponding to the final classification result.

[0150] In addition, compared with the related technology of directly coupling multiple classification models together for training, the two-stage training can effectively reduce the training cost and reduce the additional overhead caused by the coupled update of the classification models.

[0151] According to one embodiment of the present application, a classification method is provided. The classification method can be performed after step 150 of the above embodiment, and the classification method can be performed by a computer device (the computer device can be a server, an edge computing device, or other terminal device with certain processing capabilities), and the computer device has at least storage, computing, and communication functions. Figure 6 A schematic flow chart of a classification method provided in this embodiment is shown.

[0152] Figure 7 A target classification model framework diagram is shown below. Figure 6 and Figure 7 The specific steps of the classification method are described.

[0153] Step 210: Obtain the target object to be classified.

[0154] In the embodiment of the present application, the target object refers to an object to be classified in a specific classification scenario, for example, it can be a target image to be classified in an image classification scenario, it can be a target text to be classified in a text classification scenario, or it can be an express package in an item sorting scenario. The following will take the item sorting scenario as an example to explain the various steps in this embodiment. In the item sorting scenario, the item sorting system can classify different express packages so that downstream tasks can be performed according to the package category.

[0155] Exemplarily, multiple types of sensing devices can be used to collect multiple modal information of the target package. For example, a camera can be used to collect image information of the target package, such as the package appearance, and image recognition can be used to obtain text information of the target package, such as package information. A sound sensor can also be used to collect audio information of the material collision of the target package during the sorting process. Furthermore, the image information, text information, and audio information of the target package are used as multi-modal object information, that is, multi-modal object information, for sorting of package types.

[0156] Step 220: Input the object information corresponding to the target object into the target classification model, classify the object information by the known class classification model in the target classification model to obtain a first target classification result, and classify the object information by the augmented class classification model to obtain a second target classification result.

[0157] In some embodiments, the target classification model includes a known class classification model and an augmented class classification model, and the object information includes multimodal object information. The multimodal object information can be classified based on the known class classification model and the augmented class classification model to obtain a classification result corresponding to the target object, that is, the probability that the target object belongs to each category.

[0158] As an implementation method, feature extraction can be performed on object information based on a known class classification model in a target classification model to obtain a first multimodal feature, and classification can be performed based on the first multimodal feature to obtain a first target classification result, and feature extraction can be performed on object information based on an incremental classification model in the target classification model to obtain a second multimodal feature, and classification can be performed based on the second multimodal feature to obtain a second target classification result.

[0159] For example, the multimodal object information of the target package is input into the incremental classification model, and the feature extraction module of the incremental classification model performs feature extraction on the image information, text information and audio information. Specifically, the feature extraction module may include an image feature extractor, a text feature extractor and an audio feature extractor, wherein the image feature extractor performs feature extraction on the image information to obtain image features, the text feature extractor performs feature extraction on the text information to obtain text features, and the audio feature extractor performs feature extraction on the audio information to obtain audio features. Then, the feature vectors corresponding to the image features, text features and audio features are fused to obtain multimodal features, that is, the second multimodal features.

[0160] Further, the object classification module of the incremental classification model performs classification based on the second multimodal feature to obtain a second target classification result. Specifically, the object classification module may be an unknown prediction network, a newly added prediction network, and a known prediction network. Optionally, the unknown prediction network, the newly added prediction network, and the known prediction network may be a fully connected layer or other deep neural network capable of performing classification calculations. The unknown prediction network is used to perform unknown category prediction on the second multimodal feature to obtain an unknown result. The newly added prediction network is used to perform newly added category prediction on the second multimodal feature to obtain a newly added result. The known prediction network is used to perform known category prediction on the second multimodal feature to obtain a known result. Further, the unknown result, the newly added result, and the known result are normalized by the Softmax activation function to obtain the corresponding second unknown confidence, the second newly added confidence, and the second known confidence, and the second target classification result is composed of the second unknown confidence, the second newly added confidence, and the second known confidence.

[0161] In the same method, feature extraction is performed on the object information based on the known class classification model in the target classification model to obtain a first multimodal feature, and classification is performed according to the first multimodal feature to obtain a first target classification result, which may include a first unknown confidence, a first newly added confidence and a first known confidence.

[0162] Step 230: Correct the first target classification result according to the correction parameters in the target classification model to obtain a target correction result.

[0163] As an implementation, the correction parameter may include a scaling parameter and a bias parameter. The scaling parameter and the second known confidence are multiplied to obtain a scaling product term, and the scaling product term is added to the bias parameter to obtain a correction term, and then the correction term and the first newly added confidence are added to obtain a target correction result.

[0164] Step 240: Determine the object category corresponding to the target object based on the target correction result and the second target classification result.

[0165] As an implementation method, based on the target correction result and the second newly added confidence in the second target classification result, the category probabilities of the known categories and newly added categories of the target object can be determined, and further, the category corresponding to the category probability with the largest probability is determined as the object category corresponding to the target object.

[0166] This embodiment can obtain the target object to be classified; input the object information corresponding to the target object into the target classification model, classify the object information by the known class classification model in the target classification model, obtain the first target classification result, and classify the object information by the added class classification model to obtain the second target classification result, and correct the first target classification result according to the correction parameter in the target classification model to obtain the target correction result, and then, based on the target correction result and the second target classification result, determine the object category corresponding to the target object. In this way, the target classification model can be compatible with the classification of known categories and newly added categories, while avoiding the influence of unknown categories on the classification results, thereby effectively improving the accuracy of the final classification results.

[0167] According to one embodiment of the present application, a sorting method is provided. The sorting method can be executed by a computer device (the computer device can be a server, an edge computing device or other terminal device with certain processing capabilities), and the computer device has at least storage, computing and communication functions. Figure 8 A scene diagram of a sorting method provided by this embodiment is shown. Fig. 9 A flow chart of a sorting method provided by this embodiment is shown. Figure 8 and Fig. 9 The specific steps of the sorting method are described.

[0168] In actual logistics sorting scenarios, logistics service providers can develop an item sorting system based on the training method and classification method of the classification model proposed in this application for item sorting. Figure 8 The deployment architecture of the item sorting system shown may include a service backend 310 and a sorting device 320, wherein the service backend 310 may train a target classification model for classifying items during sorting. In addition, considering the efficiency of sorting, the target classification model may be directly deployed on the sorting device 320 to avoid factors such as network delay affecting the efficiency of sorting. The sorting device 320 may collect multimodal information of items for item classification. For example, the classification result may include the probability that the item belongs to n categories {P1, P2, ..., P n For example, P1 can represent the probability that an item belongs to the fragile category, P2 can represent the probability that an item belongs to the perishable category, and P n It can indicate the probability that an item belongs to the category of dangerous goods. The specific steps of the sorting method may include:

[0169] Step 410: Obtain a target classification model based on sample item training.

[0170] As an implementation, the sample item may include sample information and item labels. The sample information may be multimodal information, including image information, text information, and weight information, etc., which is not limited here. The item label indicates the category to which the sample item belongs, such as fragile items, perishable items, dangerous goods, and ordinary items.

[0171] Exemplarily, the service backend 310 can perform two-stage model training based on sample items with newly added item categories and known item categories. Specifically, the service backend 310 can first extract features of the sample items based on a preset category-increasing model, and perform unknown category prediction, newly added category prediction, and known category prediction based on the extracted sample features, to obtain corresponding unknown prediction results, newly added prediction results, and known prediction results. Further, the service backend 310 can classify the unknown prediction results, newly added prediction results, and known prediction results, and determine the classification loss based on the target classification results obtained by classification and the item labels corresponding to the sample items, and then train the preset category-increasing model through the classification loss to obtain a trained category-increasing classification model.

[0172] Furthermore, the service backend 310 can input the sample items into the known class classification model and the augmented class classification model for classification, and optimize the known class classification results obtained by classification based on the correction parameters and the augmented class classification results obtained by classification to obtain the known class optimization results. Furthermore, the service backend 310 can determine the fine-tuning loss based on the known class optimization results, the augmented class classification results, and the category label corresponding to the second sample object, and fine-tune the correction parameters based on the fine-tuning loss to obtain the fine-tuned correction parameters, so as to obtain the target classification model based on the fine-tuned correction parameters, the known class classification model, and the augmented class classification model.

[0173] Step 420: Obtain item information corresponding to the target item to be classified.

[0174] For example, after training the target classification model, the service backend 310 can send the target classification model to the sorting device 320, and the sorting device 320 deploys the target classification model to classify the items on the item sorting line. At the beginning of the classification, the sorting device 320 can collect multimodal information through different types of sensors, such as Figure 8The sorting device 320 shown may be provided with a camera 321, a text recognizer 322, and a pressure sensor 323. On the sorting line, the camera 321 may collect image information of the item, the text recognizer 322 may collect text information of the item, and the pressure sensor 323 may collect weight information of the item. Optionally, in the actual sorting process, as the modality of item information increases, a sensor capable of collecting the newly added modality, such as a sound sensor, may be added to the sorting device 320 to adapt to the sorting of items based on the newly added modality information.

[0175] Step 430: Input the item information into the target classification model, and the target classification model classifies the items based on the item information to obtain the item category corresponding to the target item.

[0176] As an implementation mode, the sorting device 320 may utilize the target classification model of the item sorting system to perform feature extraction on the image information, text information and weight information of the item, and classify the item based on the extracted multimodal features to obtain the item category.

[0177] Specifically, the sorting device 320 can input the image information, text information and weight information into the target classification model, and the known class classification model in the target classification model classifies the image information, text information and weight information to obtain the first target classification result, and the augmented class classification model classifies the image information, text information and weight information to obtain the second target classification result.

[0178] Further, the sorting device 320 can correct the first target classification result according to the correction parameter in the target classification model to obtain the target correction result. Exemplarily, the correction parameter may include a scaling parameter and a bias parameter, so that the sorting device 320 can multiply the scaling parameter and the second known confidence in the second target classification result to obtain a scaling product term, and add the scaling product term to the bias parameter to obtain a correction term, and then add the correction term to the first newly added confidence in the first target classification result to obtain the target correction result.

[0179] Further, the sorting device 320 may determine the classification result corresponding to the target item based on the target correction result and the second target classification result. For example, the classification result may include whether the item belongs to n(n∈N * ) categories {P1,P2,…,P k ,P k+1 ,…,P n}, where {P1,P2,…,P k} is the target correction result, which indicates the probability that the target object belongs to each known category, {P k+1 ,…,P n} is the second target classification result, indicating the probability that the target item belongs to each newly added category. Further, the sorting device 320 can select the category corresponding to the maximum probability from the target correction result and the second target classification result as the item category corresponding to the target item.

[0180] For example, the target correction result in the classification result may include that the target item belongs to a known category: the probability of the fragile item category P1=0.01, the probability of the contraband item category P2=0.12, and the second target classification result in the classification result may include that the target item belongs to a newly added category: the probability of the perishable item category P3=0.18, the probability of the dangerous goods category P4=0.36, and the probability of the ordinary item category P5=0.33. Thus, the sorting device 320 can use the dangerous goods category corresponding to the maximum probability P4=0.36 as the item category of the target item, and complete the classification of the target item in the known categories and the newly added categories.

[0181] In this way, by predicting the known category of the target object through the incremental classification model in the target classification model, the classification boundary between the known category classification and the newly added category classification can be learned, so that the classification accuracy of the known category of the target object by the known category classification model will not be affected by the incremental classification model, and the prediction result of the known category is corrected to the target correction result by correcting the parameters, so that the target classification model can output an accurate item category for the target item, thereby improving the sorting accuracy of the item sorting system.

[0182] See also Fig.10 , which shows a structural block diagram of a classification model training device 500 provided in an embodiment of the present application. The device 500 may include a classification prediction module 510, a loss determination module 520, a model training module 530, a classification optimization module 540 and a parameter fine-tuning module 550.

[0183] The classification prediction module 510 is used to obtain a first sample object, extract features of the first sample object based on a preset class-increasing model, and perform unknown category prediction, newly added category prediction and known category prediction according to the extracted sample features to obtain corresponding unknown prediction results, newly added prediction results and known prediction results;

[0184] A loss determination module 520, configured to perform classification according to the unknown prediction result, the newly added prediction result and the known prediction result, and determine the classification loss based on the target classification result obtained by classification and the category label corresponding to the first sample object;

[0185] A model training module 530 is used to train the preset incremental classification model through the classification loss to obtain a trained incremental classification model;

[0186] The classification optimization module 540 is used to obtain a second sample object, input the second sample object into the known class classification model and the increased class classification model for classification, and optimize the known class classification result obtained by classification based on the increased class classification result and the correction parameter to obtain a known class optimization result;

[0187] The parameter fine-tuning module 550 is used to determine the fine-tuning loss according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and fine-tune the correction parameter based on the fine-tuning loss to obtain the fine-tuned correction parameter, so as to obtain the target classification model based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model.

[0188] In some embodiments, the preset augmented classification model includes a feature extraction module and an object classification module, and the sample features include multimodal sample features; the classification prediction module 510 may include a feature extraction unit and a category prediction unit.

[0189] A feature extraction unit, configured to output the multimodal sample information corresponding to the first sample object to the feature extraction module, and based on the feature extraction module, perform feature extraction on the multimodal sample information to obtain multimodal sample features;

[0190] A category prediction unit is used to input the multimodal sample features into the object classification module, and based on the object classification module, perform unknown category prediction, new category prediction and known category prediction on the multimodal sample features to obtain unknown prediction results, new prediction results and known prediction results corresponding to the first sample object.

[0191] In some embodiments, the feature extraction unit can be specifically used to: when it is detected that new modal information exists in the multimodal information, perform feature extraction on the new modal information to obtain new modal features; perform feature extraction on known modal information in the multimodal information except the new modal information to obtain multiple known modal features; perform feature fusion on the new modal features and the multiple known modal features to obtain multimodal sample features corresponding to the first sample object.

[0192] In some embodiments, the loss determination module 520 may include a combined prediction unit, a result classification unit, an uncertainty calculation unit, and a classification loss determination unit.

[0193] A combined prediction unit, used for performing a prediction combination according to the unknown prediction result, the newly added prediction result and the known prediction result to obtain a combined prediction result;

[0194] A result classification unit, used for classifying the combined prediction result based on a nonlinear activation function to obtain a target classification result corresponding to the first sample object;

[0195] an uncertainty calculation unit, configured to calculate a first uncertainty according to the target classification result and a category label corresponding to the first sample object;

[0196] A classification loss determination unit is used to calculate the conditional distribution and the boundary distribution corresponding to the first sample object based on the first uncertainty, and determine the classification loss by the conditional distribution and the boundary distribution.

[0197] In some embodiments, the uncertainty calculation unit can be specifically used for: when the category label corresponding to the first sample object belongs to a newly added category, a negative logarithm calculation is performed based on the added category confidence under the newly added category in the target classification result to obtain a first uncertainty; when the category label corresponding to the first sample object belongs to a known category, a negative logarithm calculation is performed based on the known category confidence under the known category in the target classification result and the number of known categories to obtain the first uncertainty.

[0198] In some embodiments, the classification loss determination unit may include a condition subunit, a joint subunit, and a loss subunit.

[0199] a conditional subunit, configured to respectively determine a conditional distribution and a boundary distribution corresponding to the first sample object according to the first uncertainty;

[0200] a joint subunit, configured to determine a joint distribution corresponding to the first sample object through the conditional distribution and the boundary distribution;

[0201] The loss subunit is used to obtain an optimization function based on the joint distribution through gradient equivalent optimization to determine the classification loss.

[0202] In some embodiments, the loss subunit can be specifically used to: extract features from the category label corresponding to the first sample object to obtain category label features; extract features from each modal information of the first sample object to obtain modal features corresponding to each modal information; construct a target comparison function based on the modal features corresponding to each modal information and the category label features; perform gradient equivalent optimization on the joint distribution to obtain an optimization function, and determine the classification loss based on the optimization function and the target comparison function.

[0203] In some embodiments, the model training module 530 can be specifically used to iteratively train the feature extraction module and the object classification module in the preset classification model through the classification loss until the training end condition is met, thereby obtaining a trained enhanced classification model.

[0204] In some embodiments, the classification optimization module 540 may include a model classification unit and a result optimization unit.

[0205] A model classification unit, used for inputting the second sample object into a known class classification model for classification to obtain a known class classification result, and inputting the second sample object into the augmented class classification model for classification to obtain an augmented class classification result;

[0206] The result optimization unit is used to classify and optimize the second known class confidence under the known class in the known class classification result based on the correction parameter and the first known class confidence under the known class in the added class classification result to obtain the known class optimization result.

[0207] In some embodiments, the correction parameter includes a scaling parameter and a bias parameter, and the result optimization unit can be specifically used to: multiply the scaling parameter and the first known class confidence under the known category in the incremental classification result to obtain a scaling product term; add the scaling product term and the bias parameter to obtain a correction term; add the correction term and the second known class confidence under the known category in the known class classification result to obtain a known class optimization result.

[0208] In some embodiments, the parameter fine-tuning module 550 may include a known determination unit, a newly added determination unit, and a parameter fine-tuning unit.

[0209] a known determination unit, configured to determine a second uncertainty based on a negative logarithm of an optimization result of the known class when the class label corresponding to the second sample object belongs to a known class;

[0210] A newly added determination unit is configured to determine a second uncertainty based on a negative logarithm of an increased class confidence of the newly added class in the increased class classification result when the class label corresponding to the second sample object belongs to the newly added class;

[0211] A parameter fine-tuning unit is used to perform parameter fine-tuning on the correction parameter based on the fine-tuning loss determined by the second uncertainty to obtain the correction parameter after fine-tuning.

[0212] In some embodiments, the parameter fine-tuning unit can be specifically used to: calculate the fusion condition distribution corresponding to the second sample object according to the second uncertainty, and determine the fine-tuning loss based on the fusion condition distribution; iteratively fine-tune the correction parameter based on the fine-tuning loss until the fine-tuning end condition is met, so as to obtain the fine-tuned correction parameter.

[0213] This embodiment can obtain a first sample object, extract features of the first sample object based on a preset augmented class model, and perform unknown category prediction, newly added category prediction and known category prediction respectively according to the extracted sample features, and obtain corresponding unknown prediction results, newly added prediction results and known prediction results. Further, classification is performed according to the unknown prediction results, newly added prediction results and known prediction results, and the classification loss is determined based on the target classification result obtained by classification and the category label corresponding to the first sample object, and then, the preset augmented class model is trained by the classification loss to obtain a trained augmented class classification model. Therefore, the augmented class classification model is obtained by performing separate training for the newly added category. Compared with the related technology of directly adding a new model to the original known class classification model for mixed training, it can effectively avoid classification conflicts between classification models caused by the coupling of multiple models and improve the accuracy of subsequent classification. Furthermore, a second sample object is obtained, and the second sample object is respectively input into the known class classification model and the augmented class classification model for classification, and based on the augmented class classification result and the correction parameter obtained by classification, the known class classification result obtained by classification is classified and optimized to obtain a known class optimization result. Thus, the known class classification result output by the known class classification model can be optimized to obtain an accurate known class optimization result. Then, the fine-tuning loss is determined according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and the correction parameter is fine-tuned based on the fine-tuning loss to obtain the fine-tuned correction parameter, so that the target classification model determined based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model can avoid inaccurate classification results caused by classification bias. In this way, the two-stage continuous learning method of first training the incremental classification model separately and then determining the target classification model based on the fine-tuned correction parameters, the known class classification model and the incremental classification model can avoid conflicts between different classification models. In addition, the incremental classification model can learn the classification boundary between the known class classification and the newly added class classification through the known class prediction, so that the classification accuracy of the known class classification model for the known class will not be affected by the incremental classification model. Moreover, the incremental classification model can weaken the influence of the characteristics of the unknown class on the newly added class classification through the unknown class prediction, thereby improving the accuracy of the target classification model corresponding to the final classification result.

[0214] See also Fig.11 , which shows a structural block diagram of a classification device 600 provided in an embodiment of the present application. The device 600 may include: an object acquisition module 610, an object classification module 620, a result correction module 630 and a result determination module 640.

[0215] An object acquisition module 610 is used to acquire a target object to be classified;

[0216] The object classification module 620 is used to input the object information corresponding to the target object into the target classification model, the target classification model is obtained by the training method of the above classification model, the known class classification model in the target classification model classifies the object information to obtain a first target classification result, and the augmented class classification model classifies the object information to obtain a second target classification result;

[0217] A result correction module 630, used to correct the first target classification result according to the correction parameters in the target classification model to obtain a target correction result;

[0218] The result determination module 640 is used to determine the object category corresponding to the target object based on the target correction result and the second target classification result.

[0219] In some embodiments, the object information includes multimodal object information; the object classification module 620 can be specifically used to: extract features from the object information based on a known class classification model in the target classification model to obtain a first multimodal feature, and classify according to the first multimodal feature to obtain a first target classification result; extract features from the object information based on an incremental classification model in the target classification model to obtain a second multimodal feature, and classify according to the second multimodal feature to obtain a second target classification result.

[0220] This embodiment can obtain the target object to be classified; input the object information corresponding to the target object into the target classification model, classify the object information by the known class classification model in the target classification model, obtain the first target classification result, and classify the object information by the added class classification model to obtain the second target classification result, and correct the first target classification result according to the correction parameter in the target classification model to obtain the target correction result, and then, based on the target correction result and the second target classification result, determine the object category corresponding to the target object. In this way, the target classification model can be compatible with the classification of known categories and newly added categories, while avoiding the influence of unknown categories on the classification results, thereby effectively improving the accuracy of the final classification results.

[0221] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules, etc. can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0222] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0223] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0224] like Fig.12 As shown, the embodiment of the present application also provides a computer device 700, which includes a processor 710, a memory 720, a power supply 730 and an input unit 740. The memory 720 stores a computer program. When the computer program is called by the processor 710, the various method steps provided in the above embodiment can be implemented. Those skilled in the art can understand that the structure of the computer device shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0225] The processor 710 may include one or more processing cores. The processor 710 uses various interfaces and lines to connect various parts of the entire battery management system, and by running or executing instructions, programs, instruction sets or program sets stored in the memory 720, calls the data stored in the memory 720, executes various functions of the battery management system and processes data, and executes various functions and processes data of the computer device, thereby controlling the computer device as a whole. Optionally, the processor 710 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 710 can integrate one or more combinations of a central processing unit 710 (CPU), a graphics processing unit 710 (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 710, but may be implemented separately through a communication chip.

[0226] The memory 720 may include a random access memory 720 (Random Access Memory, RAM), and may also include a read-only memory 720 (Read-Only Memory). The memory 720 may be used to store instructions, programs, instruction sets or program sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data (such as a phone book and audio and video data) created by the computer device during use. Accordingly, the memory 720 may also include a memory controller to provide the processor 710 with access to the memory 720.

[0227] The power supply 730 can be logically connected to the processor 710 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 730 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0228] The input unit 740 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0229] Although not shown, the computer device 700 may also include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 710 in the computer device will load the executable files corresponding to the processes of one or more computer programs into the memory 720 according to the following instructions, and the processor 710 will run the data stored in the memory 720, such as the phone book and audio and video data, to implement the various method steps provided in the aforementioned embodiments.

[0230] like Fig.13 As shown, the embodiment of the present application further provides a computer-readable storage medium 800, in which a computer program 810 is stored. The computer program 810 can be called by a processor to execute various method steps provided in the embodiment of the present application.

[0231] The computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium includes a non-volatile computer readable storage medium (Non-Transitory Computer-Readable Storage Medium). The computer readable storage medium 800 has storage space for a computer program that executes any of the method steps in the above embodiments. These computer programs can be read from or written to one or more computer program products. The computer program can be compressed in an appropriate form.

[0232] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes various method steps provided in the above embodiments.

[0233] The terms "comprises" and "includes" in the specification of the present application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0234] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0235] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0236] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0238] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. It should also be understood that the various implementation methods provided in the embodiments of the present application can be arbitrarily combined to achieve different technical effects.

[0239] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A classification model training method, characterized in that: The method comprises: Obtain a first sample object, extract features of the first sample object based on a preset class-increasing model, and perform unknown category prediction, newly added category prediction, and known category prediction according to the extracted sample features to obtain corresponding unknown prediction results, newly added prediction results, and known prediction results; Classifying the unknown prediction result, the newly added prediction result, and the known prediction result, and determining the classification loss based on the target classification result obtained by the classification and the category label corresponding to the first sample object; The preset incremental classification model is trained by using the classification loss to obtain a trained incremental classification model; Obtain a second sample object, input the second sample object into the known class classification model and the augmented class classification model for classification, and optimize the known class classification result obtained by classification based on the augmented class classification result and the correction parameter to obtain a known class optimization result; The fine-tuning loss is determined according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and the correction parameter is fine-tuned based on the fine-tuning loss to obtain the fine-tuned correction parameter, so as to obtain the target classification model based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model.

2. The method according to claim 1, characterized in that The preset classification enhancement model includes a feature extraction module and an object classification module, and the sample features include multimodal sample features; The method extracts features of the first sample object based on the preset class-increasing model, and performs unknown category prediction, newly added category prediction and known category prediction according to the extracted sample features to obtain corresponding unknown prediction results, newly added prediction results and known prediction results, including: Outputting the multimodal sample information corresponding to the first sample object to the feature extraction module, and performing feature extraction on the multimodal sample information based on the feature extraction module to obtain multimodal sample features; The multimodal sample features are input into the object classification module, and based on the object classification module, unknown category prediction, new category prediction and known category prediction are performed on the multimodal sample features to obtain unknown prediction results, new prediction results and known prediction results corresponding to the first sample object.

3. The method according to claim 2, characterized in that The extracting features of the multimodal sample information to obtain multimodal sample features includes: When it is detected that there is new modal information in the multimodal information, feature extraction is performed on the new modal information to obtain new modal features; Extracting features of known modal information other than the newly added modal information in the multimodal information to obtain a plurality of known modal features; The newly added modal features and the multiple known modal features are fused to obtain a multimodal sample feature corresponding to the first sample object.

4. The method according to claim 1, characterized in that: The classifying according to the unknown prediction result, the newly added prediction result and the known prediction result, and determining the classification loss based on the target classification result obtained by classification and the category label corresponding to the first sample object, includes: Perform prediction combination according to the unknown prediction result, the newly added prediction result and the known prediction result to obtain a combined prediction result; Classifying the combined prediction result based on a nonlinear activation function to obtain a target classification result corresponding to the first sample object; Calculating a first uncertainty according to the target classification result and the category label corresponding to the first sample object; A conditional distribution and a boundary distribution corresponding to the first sample object are calculated based on the first uncertainty, and a classification loss is determined by the conditional distribution and the boundary distribution.

5. The method according to claim 4, characterized in that The calculating a first uncertainty according to the target classification result and the category label corresponding to the first sample object includes: When the category label corresponding to the first sample object belongs to a newly added category, a negative logarithm calculation is performed based on the added category confidence under the newly added category in the target classification result to obtain a first uncertainty; When the category label corresponding to the first sample object belongs to a known category, a negative logarithm calculation is performed based on the known class confidence and the number of categories of the known category in the target classification result to obtain a first uncertainty.

6. The method according to claim 4, characterized in that The calculating the conditional distribution and the boundary distribution corresponding to the first sample object based on the first uncertainty, and determining the classification loss by the conditional distribution and the boundary distribution, includes: Determine the conditional distribution and the boundary distribution corresponding to the first sample object according to the first uncertainty; Determining a joint distribution corresponding to the first sample object by using the conditional distribution and the boundary distribution; An optimization function is obtained based on the joint distribution through gradient equivalent optimization, and the classification loss is determined.

7. The method according to claim 6, characterized in that The step of obtaining an optimization function based on the joint distribution through gradient equivalent optimization and determining the classification loss includes: Extracting features of the category label corresponding to the first sample object to obtain category label features; Performing feature extraction on each modal information of the first sample object to obtain a modal feature corresponding to each modal information; Constructing a target comparison function according to the modal features corresponding to each modal information and the category label features; The joint distribution is subjected to gradient equivalent optimization to obtain an optimization function, and the classification loss is determined according to the optimization function and the target comparison function.

8. The method according to any one of claims 2 to 3, characterized in that: The step of training the preset augmented classification model by using the classification loss to obtain a trained augmented classification model includes: The feature extraction module and the object classification module in the preset classification model are iteratively trained through the classification loss until the training end condition is met, thereby obtaining a trained increased classification model.

9. The method according to claim 1, characterized in that: The step of inputting the second sample object into the known class classification model and the augmented class classification model for classification, and optimizing the known class classification result obtained by classification based on the augmented class classification result and the correction parameter obtained by classification to obtain the known class optimization result includes: Inputting the second sample object into the known class classification model for classification to obtain a known class classification result, and inputting the second sample object into the augmented class classification model for classification to obtain an augmented class classification result; Based on the correction parameter and the first known class confidence under the known class in the added class classification result, the second known class confidence under the known class in the known class classification result is classified and optimized to obtain a known class optimization result.

10. The method according to claim 9, characterized in that The correction parameters include a scaling parameter and a bias parameter, and based on the correction parameters and the first known class confidence under the known class in the added class classification result, the second known class confidence under the known class in the known class classification result is classified and optimized to obtain the known class optimization result, including: The scaling parameter is multiplied by the first known class confidence under the known class in the incremental classification result to obtain a scaling product term; Adding the scaling product term to the bias parameter to obtain a correction term; The correction term and the second known class confidence under the known category in the known class classification result are added to obtain a known class optimization result.

11. The method according to claim 1, characterized in that: The step of determining the fine-tuning loss according to the known class optimization result, the class-increased classification result, and the class label corresponding to the second sample object, and fine-tuning the correction parameter based on the fine-tuning loss to obtain the fine-tuned correction parameter includes: When the category label corresponding to the second sample object belongs to a known category, determining a second uncertainty based on the negative logarithm of the optimization result of the known class; When the category label corresponding to the second sample object belongs to a newly added category, determining a second uncertainty based on a negative logarithm of the added category confidence under the newly added category in the added category classification result; The correction parameter is fine-tuned based on the fine-tuning loss determined by the second uncertainty to obtain a fine-tuned correction parameter.

12. The method according to claim 11, characterized in that The fine-tuning of the correction parameter based on the fine-tuning loss determined based on the second uncertainty to obtain the fine-tuned correction parameter includes: Calculating a fusion condition distribution corresponding to the second sample object according to the second uncertainty, and determining a fine-tuning loss based on the fusion condition distribution; The correction parameter is iteratively fine-tuned based on the fine-tuning loss until a fine-tuning end condition is met, thereby obtaining a fine-tuned correction parameter.

13. A classification method, characterized in that: The method comprises: Get the target object to be classified; Inputting object information corresponding to the target object into a target classification model, wherein the target classification model is obtained according to the training method of the classification model according to any one of claims 1 to 12, classifying the object information by a known class classification model in the target classification model to obtain a first target classification result, and classifying the object information by the augmented class classification model to obtain a second target classification result; Correcting the first target classification result according to the correction parameters in the target classification model to obtain a target correction result; Based on the target correction result and the second target classification result, an object category corresponding to the target object is determined.

14. The method according to claim 13, characterized in that The object information includes multimodal object information; The method of classifying the object information by a known class classification model in the target classification model to obtain a first target classification result, and classifying the object information by the augmented class classification model to obtain a second target classification result includes: Extracting features of the object information based on a known class classification model in the target classification model to obtain a first multimodal feature, and performing classification according to the first multimodal feature to obtain a first target classification result; Based on the incremental classification model in the target classification model, feature extraction is performed on the object information to obtain a second multimodal feature, and classification is performed according to the second multimodal feature to obtain a second target classification result.

15. A training device for a classification model, characterized in that: The device comprises: A classification prediction module, used to obtain a first sample object, extract features of the first sample object based on a preset class-increasing model, and perform unknown category prediction, newly added category prediction and known category prediction according to the extracted sample features, to obtain corresponding unknown prediction results, newly added prediction results and known prediction results; a loss determination module, configured to classify the unknown prediction result, the newly added prediction result, and the known prediction result, and determine the classification loss based on the target classification result obtained by the classification and the category label corresponding to the first sample object; A model training module, used for training the preset incremental classification model through the classification loss to obtain a trained incremental classification model; a classification optimization module, used for obtaining a second sample object, inputting the second sample object into the known class classification model and the increased class classification model for classification, and optimizing the known class classification result obtained by classification based on the increased class classification result and the correction parameter, to obtain a known class optimization result; A parameter fine-tuning module is used to determine the fine-tuning loss according to the known class optimization result, the augmented class classification result and the category label corresponding to the second sample object, and fine-tune the correction parameter based on the fine-tuning loss to obtain the fine-tuned correction parameter, so as to obtain the target classification model based on the fine-tuned correction parameter, the known class classification model and the augmented class classification model.

16. A classification device, characterized in that: The device comprises: An object acquisition module is used to acquire a target object to be classified; An object classification module, used for inputting object information corresponding to the target object into a target classification model, wherein the target classification model is obtained according to the training method of the classification model according to any one of claims 1 to 12, classifying the object information by a known class classification model in the target classification model to obtain a first target classification result, and classifying the object information by the augmented class classification model to obtain a second target classification result; A result correction module, used to correct the first target classification result according to the correction parameters in the target classification model to obtain a target correction result; A result determination module is used to determine the object category corresponding to the target object based on the target correction result and the second target classification result.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the training method of the classification model as described in any one of claims 1 to 12, or the classification method as described in any one of claims 13 to 14.

18. A computer device, characterized in that: include: Memory; A processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the training method of the classification model as described in any one of claims 1 to 12, or the classification method as described in any one of claims 13 to 14 is implemented.

19. A computer program product comprising a computer program or computer instructions, characterized in that When the computer program or computer instruction is executed by a processor, it implements the training method of the classification model as described in any one of claims 1 to 12, or the classification method as described in any one of claims 13 to 14.

Images