Model training method, device, equipment and storage medium

Abstract

The present application discloses a model training method, apparatus, device and storage medium, which belongs to the field of computer technology. The method includes: obtaining input information for a desired classification; and obtaining sample information to be classified, the input information being information associated with the information belonging to the desired classification; respectively determining recall information matching the input information in the sample information to be classified by at least two matching methods; obtaining confirmation information for confirming the recall information, the confirmation information being used to reflect whether the recall information belongs to the desired classification; training a machine learning model based on the recall information confirmed by the confirmation information, the machine learning model being used to determine the information belonging to the desired classification in the information to be classified based on the information belonging to the desired classification. In the process of training the machine learning model, the user does not need to collect a large amount of information and label it, and only needs to confirm the recall information, which can improve the efficiency of training the machine learning model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, apparatus, device and storage medium. Background Art

[0002] When a server provides information for users to browse, it usually categorizes the information so that users can quickly find the information they want to browse. For example, the categories include food, travel, entertainment, pets, etc.

[0003] Typically, server operators add classification labels to sample information to indicate its classification. The server trains a machine learning model based on the labeled sample information, allowing the model to learn the characteristics of different types of information. The machine learning model is then used to classify the information waiting to be classified on the server.

[0004] In the process of training machine learning models, in order to ensure the accuracy of classification results, operators need to collect a large amount of sample information and manually label each collected sample information, resulting in low efficiency. Summary of the Invention

[0005] This application provides a model training method, apparatus, device, and storage medium that can improve the efficiency of information classification. The technical solution is as follows:

[0006] According to one aspect of the present application, a model training method is provided, the method comprising:

[0007] Acquiring input information for a desired classification; and acquiring sample information to be classified, wherein the input information is information associated with information belonging to the desired classification;

[0008] Determining, in the sample information to be classified, recall information that matches the input information by using at least two matching methods;

[0009] Acquiring confirmation information for confirming the recall information, wherein the confirmation information is used to reflect whether the recall information belongs to the expected category;

[0010] Based on the recall information confirmed by the confirmation information, a machine learning model is trained, and the machine learning model is used to determine the information to be classified that belongs to the expected category based on the information that belongs to the expected category.

[0011] According to another aspect of the present application, a model training device is provided, comprising:

[0012] an acquisition module for acquiring input information for a desired classification and acquiring sample information to be classified, wherein the input information is information associated with information belonging to the desired classification;

[0013] a determination module, configured to determine, in the sample information to be classified, recall information that matches the input information by using at least two matching methods;

[0014] The acquisition module is further configured to acquire confirmation information for confirming the recall information, wherein the confirmation information is used to reflect whether the recall information belongs to the expected category;

[0015] A training module is used to train a machine learning model based on the recall information confirmed by the confirmation information, and the machine learning model is used to determine the information to be classified that belongs to the expected category based on the information that belongs to the expected category.

[0016] In an optional design, the input information includes at least one of the following information:

[0017] Positive sample information belonging to the desired classification;

[0018] Negative sample information that does not belong to the expected classification;

[0019] Description information used to describe the desired classification.

[0020] In an optional design, the acquisition module is used to:

[0021] When the input information includes the positive sample information, obtaining first confirmation information for confirming the first recall information, the first confirmation information being used to reflect whether the first recall information belongs to the expected category, the first recall information being obtained by matching based on the positive sample information;

[0022] The training module is used to:

[0023] training the machine learning model based on an error between a first classification label and the first confirmation information, the first classification label being used to reflect that the first recalled information belongs to the expected classification;

[0024] Based on the positive sample information and the first recall information, continue training the machine learning model in the same manner as training the machine learning model using the input information.

[0025] In an optional design, the acquisition module is used to:

[0026] When the input information includes the negative sample information, obtaining second confirmation information for confirming the second recall information, the second confirmation information being used to reflect whether the second recall information belongs to the expected classification, the second recall information being obtained by matching based on the negative sample information;

[0027] The training module is used to:

[0028] training the machine learning model based on an error between a second classification label and the second confirmation information, the second classification label being used to reflect that the second recalled information does not belong to the expected classification;

[0029] Based on the negative sample information and the second recall information, continue training the machine learning model in the same manner as training the machine learning model using the input information.

[0030] In an optional design, the acquisition module is used to:

[0031] When the input information includes the description information, obtaining third confirmation information for confirming the third recall information, the third confirmation information being used to reflect whether the third recall information belongs to the expected category, the third recall information being obtained by matching based on the description information;

[0032] The training module is used to:

[0033] The machine learning model is trained based on the error between the third classification label and the third confirmation information, and the third classification label is used to reflect that the third recall information belongs to the expected classification.

[0034] In an optional design, the determining module is used to:

[0035] In response to the accuracy of the machine learning model satisfying the accuracy threshold, determining that training of the machine learning model is complete;

[0036] In which, the accuracy is determined based on the ratio of the number of accurate output information to the total number of output information in the output information output by the machine learning model for the i-th time, and the accurate output information is determined based on the fourth confirmation information confirmed for the output information, and the fourth confirmation information is used to reflect whether the output information belongs to the expected classification.

[0037] In an optional design, the determining module is used to:

[0038] determining, according to a weight corresponding to each of the at least two matching modes, the number of information recalls corresponding to each matching mode, wherein the weight is positively correlated with the number of information recalls;

[0039] Based on the information recall quantity, the recalled information is determined in the sample information to be classified respectively through the at least two matching methods.

[0040] In an optional design, the device further includes:

[0041] The determination module is configured to determine the recall accuracy of each matching method based on the confirmation information;

[0042] An adjustment module, configured to adjust the weight corresponding to each matching method according to the recall accuracy;

[0043] The training module is configured to continue training the machine learning model based on the adjusted weights, the input information, and the recall information in the same manner as the machine learning model was trained using the input information;

[0044] Among them, the recall accuracy rate is determined by the ratio of accurate recall information to all recall information in the recall information determined according to each matching method. The accurate recall information is the predicted correct recall information based on the confirmation information, and the recall accuracy rate is positively correlated with the adjusted weight.

[0045] In an optional design, the matching method includes at least two of the following:

[0046] Matching based on collaborative filtering algorithm;

[0047] Matching based on text similarity;

[0048] Matching based on image similarity;

[0049] Classification label matching based on the sample information to be classified;

[0050] Based on multimodal feature vector matching.

[0051] In an optional design, the device further includes:

[0052] a screening module, configured to sort the recalled information according to a matching degree, and screen out the top n recalled information, wherein the matching degree is used to reflect the degree of similarity between the features of the recalled information and the input information;

[0053] The acquisition module is used to:

[0054] Acquire confirmation information for the first n recall information.

[0055] In an optional design, the screening module is used to:

[0056] The recall information is sorted according to a preset rule to filter out the first n recall information. The preset rule includes sorting the recall information according to the number of times the recall information appears in the recall information determined by different matching methods.

[0057] In an optional design, the screening module is used to:

[0058] The recalled information is sorted based on a sorting algorithm to filter out the top n recalled information, wherein the sorting algorithm is an algorithm for sorting the recalled information based on the similarity between the recalled information and the input information.

[0059] In an optional design, the acquisition module is used to:

[0060] Acquire first classification information and the information to be classified, wherein the first classification information belongs to the expected classification;

[0061] The determination module is used to use the trained machine learning model to determine the second classification information that matches the first classification information in the information to be classified through the at least two matching methods.

[0062] In an optional design, the determining module is used to:

[0063] Based on the first classification information and the second classification information, continue to determine the third classification information in the same manner as determining the second classification information;

[0064] The third classification information is information predicted to belong to the desired classification, the amount of the second classification information is greater than the amount of the first classification information, and the amount of the third classification information is greater than the amount of the second classification information.

[0065] According to another aspect of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the model training method described above.

[0066] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, code set or instruction set is stored. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the model training method described above.

[0067] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model training method provided in various optional implementations of the above aspects.

[0068] The beneficial effects of the technical solution provided by this application include at least:

[0069] During the training of a machine learning model, recall information can be determined from the sample information to be classified using at least two matching methods. The machine learning model can then be trained based on the confirmation of the recall information. This eliminates the need for users to collect and label large amounts of information, requiring only confirmation of the recall information, which improves the efficiency of training the machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0071] Figure 1 is a schematic diagram of a process for implementing information classification provided by an exemplary embodiment of the present application;

[0072] Figure 2 is a flowchart of a model training method provided by an exemplary embodiment of the present application;

[0073] Figure 3 is a flowchart of a model training method provided by an exemplary embodiment of the present application;

[0074] Figure 4 is a schematic diagram of a user interface provided by an exemplary embodiment of the present application;

[0075] Figure 5 is a structural diagram of a model training device provided by an exemplary embodiment of the present application;

[0076] Figure 6 is a structural diagram of a model training device provided by an exemplary embodiment of the present application;

[0077] Figure 7 is a structural diagram of a model training device provided by an exemplary embodiment of the present application;

[0078] Figure 8 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0079] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0081] Figure 1 FIG. 1 is a schematic diagram of a process for implementing information classification provided by an exemplary embodiment of the present application. Figure 1 As shown, in step S1, the computer device obtains positive sample information and negative sample information labeled by the administrator's user account for the desired classification. Optionally, the computer device also obtains description information. The positive sample information belongs to the desired classification, and the negative sample information does not belong to the desired classification. The description information is used to describe the desired classification.

[0082] In step S2, the computer device determines, through at least two matching methods, the first recall information that matches the positive sample information and the second recall information that matches the negative sample information from the sample information to be classified in the information pool of the computer device. Optionally, the computer device will also determine the third recall information that matches the description information, and the third recall information belongs to the first recall information. Optionally, the above-mentioned matching methods include at least two of the following: matching based on collaborative filtering algorithm, matching based on text similarity, matching based on image similarity, matching based on classification labels of information in the information pool, and matching based on multimodal feature vectors. Each matching method has a corresponding weight, and the number of recall information determined by different matching methods is positively correlated with the weight corresponding to the matching method.

[0083] In step S3, the computer device sorts the first and second recall information using a preset rule or sorting algorithm, selecting the top n pieces of first recall information and the top m pieces of second recall information. Optionally, the preset rule includes sorting the recall information based on the number of times the recall information appears in the recall information determined using different matching methods. The sorting algorithm sorts the recall information based on the similarity between the recall information and the information used to determine the recall information.

[0084] In step S4, the computer device sends the filtered first and second recall information to the administrator's user account and obtains first confirmation information from the user account confirming the filtered first recall information and second confirmation information from the user account confirming the filtered second recall information. The first confirmation information indicates whether the first recall information belongs to the desired category. The second confirmation information indicates whether the second recall information belongs to the desired category. The machine learning model used in steps S2 and S3 is then trained based on the error between the first classification label and the first confirmation information, as well as the error between the second classification label and the second confirmation information. The first classification label indicates that the first recall information belongs to the desired category. The second classification label indicates that the second recall information does not belong to the desired category. The machine learning model can include multiple relevance measurement algorithms and ranking algorithms. During this process, the computer device also adjusts the weight of each matching method based on the recall accuracy of each matching method determined based on the confirmation information. The computer device can then iterate through the above steps, continuing to train the machine learning model based on the positive sample information, negative sample information, description information, recall and ranking results, and confirmation information from the user account until the accuracy of the information predicted by the machine learning model reaches a threshold set by the user account.

[0085] After the training is completed, the computer device can filter out the second classification information belonging to the desired category from the information to be classified in the information pool in the above method based on the first classification information belonging to the desired category circled by the user account, and can continue to filter out more information belonging to the desired category from the information pool based on the filtered information.

[0086] In the process of training a machine learning model, recall information can be determined through at least two matching methods, and the machine learning model can be trained based on the confirmation information of the recall information. In the process of training a machine learning model, the user does not need to collect a large amount of information and label it, only the recall information needs to be confirmed. In addition, the recall information can be used as a training sample to continue training the machine learning model to improve the accuracy of the machine learning model, which can improve the efficiency of training the machine learning model. When determining information that belongs to the desired category, it is only necessary to filter out a small amount of information that belongs to the desired category and input it into the trained machine learning model to determine a large amount of information that belongs to the desired category, which can improve the efficiency of classifying information.

[0087] Figure 2 This is a flow chart of a model training method provided by an exemplary embodiment of the present application. This method can be used in computer devices. Figure 2 As shown, the method includes:

[0088] Step 202: Obtain input information for the desired classification and obtain information of samples to be classified.

[0089] The desired classification is set by a user account, for example, the desired classification is a classification newly created by the user account for information stored in a computer device. The user account can be an account of an operations staff, a developer, a product staff, or an account of another user.

[0090] The input information is information associated with information belonging to the desired classification and is determined by the user account. Optionally, the input information includes at least one of positive sample information belonging to the desired classification, negative sample information not belonging to the desired classification, and descriptive information describing the desired classification. The positive sample information and negative sample information can include text, images, and videos.

[0091] Sample information to be classified includes information stored in a computer device, excluding input information. Sample information to be classified is information that needs to be classified. Classification involves determining whether the sample information to be classified belongs to the desired category. Sample information to be classified includes text, images, and videos. The format of the sample information to be classified may be the same as or different from the format of the input information.

[0092] Optionally, the computer device can be a server, which is a single server, or a server cluster consisting of several servers, or a virtual server in a cloud computing service center, etc.

[0093] Step 204: Determine the recall information that matches the input information in the sample information to be classified by using at least two matching methods.

[0094] Recall information refers to information in the sample information to be classified that matches the input information, specifically, information with similar characteristics to the input information. Based on different input information, the computer device can determine the corresponding recall information. For example, first recall information is determined based on positive sample information. First recall information is information predicted to belong to the desired classification. Second recall information is determined based on negative sample information. Second recall information is information predicted not to belong to the desired classification. Third recall information is determined based on descriptive information. Third recall information is information predicted to belong to the desired classification.

[0095] Optionally, the matching methods include at least two of the following: matching based on collaborative filtering algorithms, matching based on text similarity, matching based on image similarity, matching based on classification labels of sample information to be classified, and matching based on multimodal feature vectors. For different input information, the computer uses the same or different matching methods when determining recall information. Optionally, each matching method is also assigned a weight, and the weight of the matching method is positively correlated with the amount of recall information determined using that matching method. The weight is determined by the computer device.

[0096] Step 206: Acquire confirmation information for confirming the recall information.

[0097] The confirmation information is used to reflect whether the recalled information belongs to the expected category. Optionally, the confirmation information is obtained by the user account confirming the recalled information. After confirming the recalled information, the computer device will send the recalled information to the user account for confirmation, thereby obtaining the confirmation information sent by the user account.

[0098] Optionally, the computer device also sorts the recall information by matching degree, thereby selecting the top n recall information and sending them to the user account. Confirmation information is then obtained from the user account confirming these top n recall information. Here, n is determined by the computer device. The matching degree reflects the degree of feature similarity between the recall information and the input information. This approach prevents less relevant information from being sent to the user account for confirmation, which could reduce model training efficiency.

[0099] Step 208: Train the machine learning model based on the recall information confirmed by the confirmation information.

[0100] The machine learning model is used to determine information belonging to the expected category in the input information based on the information belonging to the expected category. The computer device can train the machine learning model based on the classification label of the recalled information and the error between the confirmation information. When the recalled information is determined by positive sample information, the classification label is used to reflect that the recalled information belongs to the expected category. When the recalled information is determined by negative sample information, the classification label is used to reflect that the recalled information does not belong to the expected category. When the recalled information is determined by descriptive information, the classification label is used to reflect that the recalled information belongs to the expected category.

[0101] Optionally, the machine learning model is used in the process of determining recall information through different matching methods and sorting the recall information. For example, the machine learning model can be used to implement text similarity matching, image similarity matching, extract multimodal feature vectors of information for matching, and implement sorting based on a sorting algorithm. By training the machine learning model, it is possible to accurately determine a large amount of information belonging to a desired category based on a small amount of information belonging to the desired category.

[0102] In summary, the method provided in this embodiment enables, during the training of a machine learning model, the determination of recall information from sample information to be classified through at least two matching methods, and the training of the machine learning model based on the confirmation of the recall information. This eliminates the need for users to collect and annotate large amounts of information during the training process, requiring only the confirmation of the recall information, thereby improving the efficiency of training the machine learning model.

[0103] Figure 3This is a flow chart of a model training method provided by an exemplary embodiment of the present application. This method can be used in computer devices. Figure 3 As shown, the method includes:

[0104] Step 302: Obtain input information for the desired classification and obtain information of samples to be classified.

[0105] The desired classification is set by the user account. Input information is information associated with information belonging to the desired classification, and is determined by the user account. Sample information to be classified includes information stored in the computer device, excluding the input information. Sample information to be classified is information that needs to be classified, and classification involves determining whether the sample information to be classified belongs to the desired classification.

[0106] Optionally, the input information includes at least one of the following information:

[0107] Positive sample information belonging to the expected classification;

[0108] Negative sample information that does not belong to the expected classification;

[0109] Descriptive information used to describe the expected classification.

[0110] Positive and negative sample information can include text, images, and videos. The description information is text. The sample information to be classified includes text, images, and videos. The format of the sample information to be classified can be the same as or different from the format of the input information.

[0111] Step 304: Determine the recall information that matches the input information in the sample information to be classified by using at least two matching methods.

[0112] Recall information is information in the sample information to be classified that matches the input information, specifically information with similar characteristics to the input information. Based on the positive sample information, the computer device can determine first recall information, which is information that matches the positive sample information. Based on the negative sample information, the computer device can determine second recall information, which is information that matches the negative sample information. Based on the description information, the computer device can determine third recall information, which is information that matches the description information.

[0113] Optionally, the matching method includes at least two of the following:

[0114] Matching based on collaborative filtering algorithm;

[0115] Matching based on text similarity;

[0116] Matching based on image similarity;

[0117] Classification label matching based on the sample information to be classified;

[0118] Based on multimodal feature vector matching.

[0119] Collaborative filtering algorithms include the Item-based Collaboration Filter (ItemCF) algorithm. When using ItemCF to determine recall information, the computer first determines the similarity between the input information and the sample information to be classified. It then filters the sample information based on the similarity between the input information and the sample information to be classified, as well as the historical behavior information of the sample user account, to determine the recall information.

[0120] Exemplarily, the computer device determines the first number of sample user accounts that have interacted with both the input information and the sample information to be classified, and the second number of sample user accounts that have interacted with the input information based on historical behavior information. The interactive behaviors include clicks, likes, favorites, comments, and forwarding. The similarity is then determined based on the ratio of the first number to the second number. The sample information to be classified is then sorted based on the degree of interest of the sample user accounts that are interested in the input information and the similarity between the input information and the sample information to be classified, thereby filtering out the recalled information. Whether the sample user account is interested in a certain information and the degree of interest are determined based on historical behavior information. For example, if a click behavior has occurred, it is considered to be interested, and if a comment behavior has occurred, the degree of interest is higher than that of a click behavior.

[0121] When a computer device determines recall information from sample information to be classified using a method based on text similarity matching, it can use a machine learning model to implement it. For example, a computer device can implement text similarity matching through a Bidirectional Encoder Representations from Transformer (BERT) model based on a transformer. In this process, the BERT model extracts the semantic feature vectors of the sample information to be classified and the semantic feature vectors of the input information, and then implements text similarity matching based on the determined semantic vectors to determine the recall information. Optionally, when the input information includes text, the computer device determines the recall information through a method based on text similarity matching, and the recall information determined at this time is also text.

[0122] When the computer device determines the recall information from the sample information to be classified using a method based on image similarity matching, it can use a machine learning model to implement it. For example, the computer device can obtain the recall information based on image similarity matching through a machine learning model based on a neural network (NN). The input of the machine learning model is a picture pair, and the output is the similarity between the pictures in the picture pair. Through the machine learning model, the computer device can determine the image similarity between the sample information to be classified and the input information, so as to filter out the recall information. Optionally, when the input information includes pictures or videos, the computer device will determine the recall information based on image similarity matching, and the recall information determined at this time is also a picture or video. In this process, the video is processed based on the video frames (i.e., pictures) in the video.

[0123] When determining recall information from the sample information, the computer device uses a classification label matching method based on the sample information to be classified. The classification label of the sample information to be classified is matched with the classification label of the input information to determine the recall information. The classification label of the sample information to be classified is a label already stored in the computer device, while the classification label of the input information is determined by the user account.

[0124] When a computer device uses a method based on multimodal feature vector matching to determine recall information from sample information to be classified, it can use a machine learning model to achieve this. A multimodal feature vector refers to a vector that represents the characteristics of information in multiple modalities such as text and images in the same feature space. For example, a computer device can use a machine learning model to map input information in different formats to the same feature space to obtain a feature vector of the input information, and map the sample information to be classified to the feature space to obtain a feature vector of the sample information to be classified. Then, based on the similarity between the feature vectors, the recall information can be determined.

[0125] Optionally, during the process of determining recall information, the computer device determines the amount of information recall corresponding to each of the at least two matching methods based on the weight corresponding to each matching method. Then, based on the amount of information recall, the computer device determines recall information from the sample information to be classified using the at least two matching methods. The weight of the matching method is positively correlated with the amount of information recall corresponding to the matching method. That is, the higher the weight of the matching method, the greater the amount of recall information determined by the computer device using that matching method. The weight is set by the computer device.

[0126] Step 306: Sort the recalled information according to the matching degree, and filter out the top n recalled information.

[0127] The matching degree is used to reflect the degree of similarity between the features of the recalled information and the input information.

[0128] Optionally, the computer device can sort the recall information according to preset rules, selecting the top n recall information. The preset rules include sorting the recall information according to the number of times the recall information appears in the recall information determined by different matching methods. When recall information is determined using different input information and different matching methods, the same recall information may appear multiple times. Therefore, the computer device can sort the recall information according to the number of times the recall information appears in the recall information determined by different matching methods.

[0129] Optionally, the computer device can further sort the recalled information based on a sorting algorithm to select the top n recalled information. The sorting algorithm is an algorithm that sorts the recalled information based on the similarity between the recalled information and the input information. The computer device can implement sorting of the recalled information based on the sorting algorithm using a machine learning model. The machine learning model is composed of a neural network and can determine the similarity between the recalled information and the input information, thereby sorting the recalled information based on the similarity.

[0130] If the input information does not include descriptive information, the computer device can also sort the recalled information based on the descriptive information. The computer device uses a machine learning model to extract feature vectors of the descriptive information and the recall information. Then, based on the similarity between the feature vectors, the computer device can sort and filter the recalled information. The computer device can filter the same or different number of recalled information determined based on different input information. This number is set by the computer device.

[0131] Optionally, before the collection of recall information converges, the computer device performs sorting based on the direction of collecting the information that can best help the computer device confirm the scope of the collection of recall information. After the collection of recall information converges, the computer device performs sorting based on the direction of most relevance.

[0132] Step 308: Acquire confirmation information for the first n pieces of recall information.

[0133] The confirmation information is used to reflect whether the recalled information belongs to the expected category. Optionally, the confirmation information is obtained by the user account confirming the recalled information. After confirming the recalled information, the computer device will send the recalled information to the user account for confirmation, thereby obtaining the confirmation information sent by the user account.

[0134] When the input information includes positive sample information, the computer device obtains first confirmation information for confirming the first recall information, wherein the first confirmation information is used to reflect whether the first recall information belongs to the expected category, and the first recall information is obtained by matching based on the positive sample information.

[0135] When the input information includes negative sample information, the computer device obtains second confirmation information for confirming the second recall information, wherein the second confirmation information is used to reflect whether the second recall information belongs to the expected category, and the second recall information is obtained by matching the negative sample information.

[0136] When the input information includes the description information, the computer device will obtain third confirmation information for confirming the third recall information, wherein the third confirmation information is used to reflect whether the third recall information belongs to the expected category, and the third recall information is obtained by matching based on the description information.

[0137] Step 310: Train the machine learning model based on the recall information confirmed by the confirmation information.

[0138] This machine learning model is used to determine which information in the to-be-classified information belongs to the desired category based on the information that belongs to the desired category. This machine learning model is used during the process of determining and sorting the recalled information using different matching methods. By training this machine learning model, it is possible to accurately determine a large amount of information that belongs to the desired category based on a small amount of information that belongs to the desired category.

[0139] When the input information includes positive sample information, the computer device can train the machine learning model based on the error between the first classification label and the first confirmation information, wherein the first classification label is used to reflect that the first recall information belongs to the expected classification.

[0140] When the input information includes negative sample information, the computer device can train the machine learning model based on the error between the second classification label and the second confirmation information, wherein the second classification label is used to reflect that the second recall information does not belong to the expected classification.

[0141] When the input information includes the descriptive information, the computer device can train the machine learning model based on the error between the third classification label and the third confirmation information, wherein the third classification label is used to reflect that the third recall information belongs to the expected classification.

[0142] Optionally, after training the machine learning model using the above information, the computer device can also continue to train the machine learning model in a cycle based on the determined recall information and input information, using the same method as training the machine learning model using the input information, so as to continuously improve the accuracy of the machine learning model.

[0143] For example, the computer device may continue to train the machine learning model based on the positive sample information and the first recall information in the same manner as the input information is used to train the machine learning model. The computer device may continue to train the machine learning model based on the negative sample information and the second recall information in the same manner as the input information is used to train the machine learning model. When the input information includes multiple types of information, the computer device may continue to train the machine learning model based on the input information and the recall information determined from the multiple types of input information.

[0144] During the cyclic training of the machine learning model, the training of the machine learning model is determined to be complete in response to the accuracy of the machine learning model satisfying an accuracy threshold. The accuracy is determined based on the ratio of the number of accurate output information to the total number of output information in the i-th output of the machine learning model. The accurate output information is determined based on fourth confirmation information confirming the output information, which indicates whether the output information belongs to the expected classification. The accuracy threshold is set by the user account.

[0145] For example, the output information for the i-th output includes information 1, which is determined based on the positive sample information. If the fourth confirmation information indicates that information 1 belongs to the expected classification, then information 1 is accurate output information. The output information for the i-th output includes information 2, which is determined based on the negative sample information. If the fourth confirmation information indicates that information 2 belongs to the expected classification, then information 2 is accurate output information.

[0146] Optionally, during the process of cyclic training of the machine learning model, the computer device can also adjust the weight of each matching method. The computer device determines the recall accuracy of each matching method based on the confirmation information. The weight corresponding to each matching method can be adjusted according to the recall accuracy. Then, based on the adjusted weights, input information and recall information, the machine learning model is continued to be trained in the same manner as the input information is used to train the machine learning model, that is, the cyclic training of the machine learning model is continued. Among them, the recall accuracy is determined by the ratio of accurate recall information in the recall information determined by each matching method to all recall information. The accurate recall information is the recall information that is predicted to be correct based on the confirmation information, and the recall accuracy is positively correlated with the adjusted weight.

[0147] Step 312: Obtain first classification information and information to be classified.

[0148] The first category information is information belonging to a desired category. The first category information is determined by the user account. When a large amount of similar information needs to be circled based on a small amount of information belonging to a desired category, the user account will determine the first category information.

[0149] Step 314: Using the trained machine learning model, determine second classification information that matches the first classification information in the information to be classified through at least two matching methods.

[0150] Optionally, after determining the second classification information, the computer device can further determine third classification information based on the first and second classification information in the same manner as for determining the second classification information. The third classification information is information predicted to belong to the desired classification, the amount of second classification information is greater than the amount of first classification information, and the amount of third classification information is greater than the amount of second classification information. In other words, the computer device can continue to identify more similar information based on the identified similar information.

[0151] When a specific category of information needs to be identified, the computer device trains a machine learning model for that category, enabling it to identify a large amount of similar information using a small amount of information. Furthermore, because the information stored in the computer device is frequently updated, the computer device regularly retrains the machine learning model with updated data to ensure its accuracy.

[0152] It should be noted that the above steps 302 to 310 can be independently implemented as a model training method on the model training side, and the above steps 312 to 314 can be independently implemented as an information classification method on the model application side.

[0153] In summary, the method provided in this embodiment enables, during the training of a machine learning model, the determination of recall information from sample information to be classified through at least two matching methods, and the training of the machine learning model based on the confirmation of the recall information. This eliminates the need for users to collect and annotate large amounts of information during the training process, requiring only the confirmation of the recall information, thereby improving the efficiency of training the machine learning model.

[0154] In addition, training a machine learning model based on different input information can enhance the learning ability of the machine learning model, thereby improving the accuracy of the machine learning model. Continuing to train the machine learning model based on recall information and input information can continuously improve the accuracy of the machine learning model. Judging whether the machine learning model training is complete based on the accuracy threshold can achieve flexible control of the desired machine learning model accuracy and improve training efficiency. By sorting and screening the recalled information, it is possible to avoid confirming information with low relevance, which leads to a decrease in model training efficiency. Setting weights for different matching methods can enable methods with higher accuracy to recall more recalled information, thereby improving the efficiency of information recall. Determining recalled information through multiple matching methods can improve the richness and accuracy of the determined recalled information.

[0155] It should be noted that the order of the method steps provided in the embodiments of the present application can be appropriately adjusted, and the steps can be increased or decreased accordingly according to the circumstances. Any technical personnel familiar with this technical field can easily think of the changed methods within the technical scope disclosed in this application, and they should be covered within the scope of protection of this application, so they will not be repeated here.

[0156] In a specific example, when the management personnel need to circle the relevant information of the merchants that can provide services for interacting with pet dogs, the expected classification will be determined as pet dog interaction (stroking the dog), and the input information will be determined for the expected classification. The input information includes positive sample information, negative sample information and descriptive information. The positive sample information includes the merchant name of the first merchant, the merchant's introduction information, the merchant's store introduction photos, the merchant's store introduction video, the merchant's store introduction information, the merchant's product introduction information, the first merchant's comment information, the comment photos and the comment video, etc. The first merchant is a merchant that can provide services for interacting with pet dogs. The negative sample information includes the above information of the second merchant, and the second merchant is a merchant that cannot provide services for interacting with pet dogs, such as restaurants that only provide dining, hotels that only provide accommodation, scenic spots and bathing merchants, etc. The descriptive information can be, for example, "restaurants that provide services for playing with pet dogs, bars that provide services for playing with pet dogs, and rest places that provide services for playing with pet dogs."

[0157] Based on the input information, the server uses collaborative filtering, text similarity, image similarity, classification label matching, and multimodal feature vector matching to retrieve first recall information matching the positive sample information, second recall information matching the negative sample information, and third recall information matching the description information from the sample information to be classified. The server then sorts the first, second, and third recall information based on their matching scores, filtering each of the first, second, and third recall information. The filtered first, second, and third recall information are then sent to a management staff member for confirmation, resulting in a first confirmation message confirming the first recall information, a second confirmation message confirming the second recall information, and a third confirmation message confirming the third recall information. A machine learning model is trained based on the error between the first confirmation information and the first classification label, the error between the second confirmation information and the second classification label, and the error between the third confirmation information and the third classification label. Furthermore, the weights of each matching method are adjusted based on the first, second, and third confirmation information. Afterwards, the machine learning model is trained in a cycle using the above method according to the first recall information corresponding to the first confirmation information, the second recall information corresponding to the second confirmation information, the third recall information corresponding to the third confirmation information, the positive sample information, the negative sample information and the description information until the accuracy of the machine learning model meets the accuracy threshold set by the administrator.

[0158] After training the machine learning model, the administrator will identify a small amount of information that falls into the desired category and upload it to the server. This can be done by directly using the aforementioned positive sample information, as well as the information that falls into the desired category identified during the machine learning model training process. Using the trained machine learning model and the uploaded information, the server can identify a large amount of information that falls into the desired category within the information to be classified, thereby effectively identifying the information that falls into the desired category within the server's information to be classified.

[0159] Figure 4 FIG. 1 is a schematic diagram of a user interface provided by an exemplary embodiment of the present application. Figure 4As shown, after the server has circled the information belonging to the pet dog interaction category, it can display the information belonging to this category to the user. When the user opens the information recommendation interface 401 through the client corresponding to the server, the client will send an information recommendation request to the server. The server determines that the user is interested in the information belonging to the above-mentioned expected category, and will determine the recommended information from the information belonging to the expected category determined in the above manner, and send it to the client. The information recommendation interface 401 includes an information category filtering button 402, and the information category includes the expected category. After receiving the recommended information, the client will display the recommended information 403 on the information recommendation interface 401. Thereby, the information belonging to the expected category is circled, and then the information is recommended to the user according to the expected category. The recommended information recommended to the user includes an introduction to merchants (restaurants, bars) that can provide "dog petting" services, an introduction to the products of such merchants, and comments (text, pictures, videos) on the experience of such merchants, etc.

[0160] Figure 5 This is a schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application. The device can be used in computer equipment. Figure 5 As shown, the device includes:

[0161] The acquisition module 501 is used to acquire input information for a desired classification and to acquire sample information to be classified. The input information is information associated with information belonging to the desired classification.

[0162] The determination module 502 is configured to determine the recall information that matches the input information in the sample information to be classified by using at least two matching methods.

[0163] The acquisition module 501 is further configured to acquire confirmation information for confirming the recalled information, where the confirmation information is used to reflect whether the recalled information belongs to an expected category.

[0164] The training module 503 is used to train a machine learning model based on the recall information confirmed by the confirmation information, and the machine learning model is used to determine the information to be classified that belongs to the expected category based on the information that belongs to the expected category.

[0165] In an optional design, the input information includes at least one of the following information:

[0166] Positive sample information belonging to the expected classification;

[0167] Negative sample information that does not belong to the expected classification;

[0168] Descriptive information used to describe the expected classification.

[0169] In an optional design, the acquisition module 501 is configured to:

[0170] When the input information includes positive sample information, first confirmation information for confirming the first recall information is obtained. The first confirmation information is used to reflect whether the first recall information belongs to the expected category. The first recall information is obtained by matching based on the positive sample information.

[0171] The training module 503 is used to:

[0172] The machine learning model is trained based on the error between the first classification label and the first confirmation information, where the first classification label indicates that the first recall information belongs to the desired classification. The machine learning model is trained based on the positive sample information and the first recall information in the same manner as the input information.

[0173] In an optional design, the acquisition module 501 is configured to:

[0174] When the input information includes negative sample information, second confirmation information is obtained for confirming the second recall information. The second confirmation information is used to reflect whether the second recall information belongs to the expected classification. The second recall information is obtained by matching based on the negative sample information.

[0175] The training module 503 is used to:

[0176] The machine learning model is trained based on the error between the second classification label and the second confirmation information, where the second classification label is used to reflect that the second recall information does not belong to the expected classification. The machine learning model is further trained based on the negative sample information and the second recall information in the same manner as the input information.

[0177] In an optional design, the acquisition module 501 is configured to:

[0178] In the case where the input information includes description information, third confirmation information for confirming the third recall information is obtained, the third confirmation information is used to reflect whether the third recall information belongs to the expected category, and the third recall information is obtained by matching based on the description information.

[0179] The training module 503 is used to:

[0180] The machine learning model is trained based on the error between the third classification label and the third confirmation information, and the third classification label is used to reflect that the third recall information belongs to the expected classification.

[0181] In an optional design, the determination module 502 is configured to:

[0182] In response to the accuracy of the machine learning model satisfying the accuracy threshold, it is determined that the training of the machine learning model is completed. The accuracy is determined based on a ratio of the amount of accurate output information to the total amount of output information in the i-th output information output by the machine learning model, and the accurate output information is determined based on fourth confirmation information confirming the output information, the fourth confirmation information being used to reflect whether the output information belongs to the expected classification.

[0183] In an optional design, the determination module 502 is configured to:

[0184] The number of information recalls corresponding to each of the at least two matching methods is determined based on a weight corresponding to each matching method, where the weight is positively correlated with the number of information recalls. Based on the number of information recalls, recall information is determined in the sample information to be classified using the at least two matching methods.

[0185] In an alternative design, such as Figure 6 As shown, the device also includes:

[0186] The determination module 502 is used to determine the recall accuracy of each matching method according to the confirmation information.

[0187] The adjustment module 504 is configured to adjust the weight corresponding to each matching method according to the recall accuracy.

[0188] The training module 503 is used to continue training the machine learning model based on the adjusted weights, input information and recall information in the same manner as the input information is used to train the machine learning model.

[0189] Among them, the recall accuracy is determined by the ratio of accurate recall information to all recall information in the recall information determined by each matching method. Accurate recall information is the predicted correct recall information based on the confirmation information. The recall accuracy is positively correlated with the adjusted weight.

[0190] In an optional design, the matching methods include at least two of the following:

[0191] Matching based on collaborative filtering algorithm;

[0192] Matching based on text similarity;

[0193] Matching based on image similarity;

[0194] Classification label matching based on the sample information to be classified;

[0195] Based on multimodal feature vector matching.

[0196] In an alternative design, such as Figure 7 As shown, the device also includes:

[0197] The screening module 505 is used to sort the recalled information according to the matching degree and screen out the first n recalled information. The matching degree is used to reflect the degree of similarity between the features of the recalled information and the input information.

[0198] The acquisition module 501 is used to:

[0199] Get confirmation information for the first n recall information.

[0200] In an optional design, the screening module 505 is used to:

[0201] The recall information is sorted according to preset rules to filter out the top n recall information. The preset rules include sorting the recall information according to the number of times the recall information appears in the recall information determined by different matching methods.

[0202] In an optional design, the screening module 505 is used to:

[0203] The recalled information is sorted based on a sorting algorithm to filter out the top n recalled information. The sorting algorithm is an algorithm that sorts the recalled information based on the similarity between the recalled information and the input information.

[0204] In an optional design, the acquisition module 501 is configured to:

[0205] First classification information and information to be classified are obtained, where the first classification information belongs to the expected classification.

[0206] The determination module 502 is configured to use the trained machine learning model to determine, in the information to be classified, second classification information that matches the first classification information through at least two matching methods.

[0207] In an optional design, the determination module 502 is configured to:

[0208] Based on the first and second classification information, the third classification information is determined in the same manner as the second classification information, wherein the third classification information is information predicted to belong to the desired classification, the amount of the second classification information is greater than the first classification information, and the amount of the third classification information is greater than the second classification information.

[0209] It should be noted that the model training device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model training device provided in the above embodiment and the model training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0210] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the model training method provided by the above-mentioned method embodiments.

[0211] Optionally, the computer device is a server. For example, Figure 8 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0212] The computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including a random access memory (RAM) 802 and a read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. The computer device 800 also includes a basic input / output system (I / O system) 806 that facilitates information transmission between various components within the computer device, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

[0213] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 such as a mouse and keyboard for user input. The display 808 and the input device 809 are both connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include an input / output controller 810 for receiving and processing input from a variety of other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0214] The mass storage device 807 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable storage medium provide non-volatile storage for the computer device 800. In other words, the mass storage device 807 may include a computer-readable storage medium (not shown) such as a hard disk or a CD-ROM drive.

[0215] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 804 and mass storage device 807 can be collectively referred to as memory.

[0216] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 801. The one or more programs contain instructions for implementing the above-mentioned method embodiments. The central processing unit 801 executes the one or more programs to implement the methods provided by the above-mentioned method embodiments.

[0217] According to various embodiments of the present application, the computer device 800 may also be connected to a remote computer device on a network such as the Internet for operation. That is, the computer device 800 may be connected to a network 812 via a network interface unit 811 connected to the system bus 805, or the network interface unit 811 may be used to connect to other types of networks or remote computer device systems (not shown).

[0218] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0219] A computer-readable storage medium is also provided in an embodiment of the present application, which stores at least one instruction, at least one program, code set or instruction set. When the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor of a computer device, the model training method provided by the above-mentioned method embodiments is implemented.

[0220] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model training methods provided in the above-mentioned method embodiments.

[0221] Those skilled in the art will understand that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0222] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switches, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquiring input information for a desired classification; and acquiring sample information to be classified, wherein the input information is information associated with information belonging to the desired classification; Determining, in the sample information to be classified, recall information that matches the input information by using at least two matching methods; Acquiring confirmation information for confirming the recall information, wherein the confirmation information is used to reflect whether the recall information belongs to the expected category; Training a machine learning model based on the recall information confirmed by the confirmation information, wherein the machine learning model is used to determine information belonging to the expected category in the information to be classified based on the information belonging to the expected category; Determining the recall information matching the input information in the sample information to be classified by at least two matching methods includes: determining, according to a weight corresponding to each of the at least two matching modes, the number of information recalls corresponding to each matching mode, wherein the weight is positively correlated with the number of information recalls; Based on the information recall quantity, the recalled information is determined in the sample information to be classified respectively through the at least two matching methods.

2. The method according to claim 1, characterized in that The input information includes at least one of the following information: Positive sample information belonging to the desired classification; Negative sample information that does not belong to the expected classification; Description information used to describe the desired classification.

3. The method according to claim 2, characterized in that The obtaining of confirmation information for confirming the recall information includes: When the input information includes the positive sample information, obtaining first confirmation information for confirming the first recall information, the first confirmation information being used to reflect whether the first recall information belongs to the expected category, the first recall information being obtained by matching based on the positive sample information; The training of the machine learning model based on the recall information confirmed by the confirmation information includes: training the machine learning model based on an error between a first classification label and the first confirmation information, the first classification label being used to reflect that the first recalled information belongs to the expected classification; Based on the positive sample information and the first recall information, continue training the machine learning model in the same manner as training the machine learning model using the input information.

4. The method according to claim 2, characterized in that The obtaining of confirmation information for confirming the recall information includes: When the input information includes the negative sample information, obtaining second confirmation information for confirming the second recall information, the second confirmation information being used to reflect whether the second recall information belongs to the expected classification, the second recall information being obtained by matching based on the negative sample information; The training of the machine learning model based on the recall information confirmed by the confirmation information includes: training the machine learning model based on an error between a second classification label and the second confirmation information, the second classification label being used to reflect that the second recalled information does not belong to the expected classification; Based on the negative sample information and the second recall information, continue training the machine learning model in the same manner as training the machine learning model using the input information.

5. The method according to claim 2, characterized in that The obtaining of confirmation information for confirming the recall information includes: When the input information includes the description information, obtaining third confirmation information for confirming the third recall information, the third confirmation information being used to reflect whether the third recall information belongs to the expected category, the third recall information being obtained by matching based on the description information; The training of the machine learning model based on the recall information confirmed by the confirmation information includes: The machine learning model is trained based on the error between the third classification label and the third confirmation information, and the third classification label is used to reflect that the third recall information belongs to the expected classification.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: In response to the accuracy of the machine learning model satisfying the accuracy threshold, determining that training of the machine learning model is complete; In which, the accuracy is determined based on the ratio of the number of accurate output information to the total number of output information in the output information output by the machine learning model for the i-th time, and the accurate output information is determined based on the fourth confirmation information confirmed for the output information, and the fourth confirmation information is used to reflect whether the output information belongs to the expected classification.

7. The method according to claim 1, characterized in that The method further comprises: Determining the recall accuracy of each matching method based on the confirmation information; Adjusting the weight corresponding to each matching method according to the recall accuracy; continuing to train the machine learning model based on the adjusted weights, the input information, and the recall information in the same manner as the machine learning model was trained using the input information; Among them, the recall accuracy rate is determined by the ratio of accurate recall information to all recall information in the recall information determined according to each matching method. The accurate recall information is the predicted correct recall information based on the confirmation information, and the recall accuracy rate is positively correlated with the adjusted weight.

8. The method according to any one of claims 1 to 5, characterized in that: The matching methods include at least two of the following: Matching based on collaborative filtering algorithm; Matching based on text similarity; Matching based on image similarity; Classification label matching based on the sample information to be classified; Based on multimodal feature vector matching.

9. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Sorting the recalled information according to the matching degree, and selecting the top n recalled information, wherein the matching degree is used to reflect the degree of similarity between the features of the recalled information and the input information; The obtaining of confirmation information for confirming the recall information includes: Acquire confirmation information for the first n recall information.

10. The method according to claim 9, characterized in that The step of sorting the recalled information according to the matching degree and selecting the top n recalled information includes: The recall information is sorted according to a preset rule to filter out the first n recall information. The preset rule includes sorting the recall information according to the number of times the recall information appears in the recall information determined by different matching methods.

11. The method according to claim 9, characterized in that The step of sorting the recalled information according to the matching degree and selecting the top n recalled information includes: The recalled information is sorted based on a sorting algorithm to filter out the top n recalled information, wherein the sorting algorithm is an algorithm for sorting the recalled information based on the similarity between the recalled information and the input information.

12. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire first classification information and the information to be classified, wherein the first classification information belongs to the expected classification; The trained machine learning model is used to determine second classification information matching the first classification information in the information to be classified through the at least two matching methods.

13. The method according to claim 12, characterized in that The method further comprises: Based on the first classification information and the second classification information, continue to determine the third classification information in the same manner as determining the second classification information; The third classification information is information predicted to belong to the desired classification, the amount of the second classification information is greater than the amount of the first classification information, and the amount of the third classification information is greater than the amount of the second classification information.

14. A model training device, characterized in that: The device comprises: an acquisition module for acquiring input information for a desired classification and acquiring sample information to be classified, wherein the input information is information associated with information belonging to the desired classification; A determination module is configured to determine, in the sample information to be classified, the recall information that matches the input information, respectively, by using at least two matching methods. The determination module comprises: determining, according to a weight corresponding to each of the at least two matching modes, the number of information recalls corresponding to each matching mode, wherein the weight is positively correlated with the number of information recalls; Based on the information recall quantity, determining the recall information in the sample information to be classified respectively by using the at least two matching methods; The acquisition module is further configured to acquire confirmation information for confirming the recall information, wherein the confirmation information is used to reflect whether the recall information belongs to the expected category; A training module is used to train a machine learning model based on the recall information confirmed by the confirmation information, and the machine learning model is used to determine the information to be classified that belongs to the expected category based on the information that belongs to the expected category.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the model training method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the model training method as described in any one of claims 1 to 13.

Images