Counterfeit labeling methods, devices, electronic equipment, media and program products

Abstract

This application discloses a method, apparatus, electronic device, medium, and program product for pseudo-labeling. The pseudo-labeling method includes: obtaining a real label sample set; labeling an unlabeled sample set as an initial pseudo-label sample set based on a first labeling model constructed from the real label sample set; dividing the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set; re-labeling the second pseudo-label sample set using a second labeling model jointly constructed from the real label sample set and the first pseudo-label sample set to obtain a third pseudo-label sample set; calculating the label difference value between the pseudo-labels labeled in the second and third pseudo-label sample sets; and filtering the target pseudo-label sample set by removing samples from the second and third pseudo-label sample sets whose label difference value is greater than a preset label difference threshold. This application solves the technical problem of low accuracy in pseudo-labeling in the prior art.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology in financial technology (Fintech), and more particularly to a pseudo-labeling method, device, electronic device, medium, and program product. Background Technology

[0002] With the continuous development of fintech, especially internet fintech, more and more technologies (such as distributed systems and artificial intelligence) are being applied in the financial field. However, the financial industry is also placing higher demands on technology, such as on the distribution of tasks to be completed.

[0003] When training a supervised model, a sufficient number of labeled samples are usually required to ensure the accuracy of the training model. However, the model trainer usually has a large number of unlabeled samples and a small number of labeled samples. The number of labeled samples does not meet the requirements for the number of labeled samples for model training. Therefore, the model trainer needs to perform pseudo-labeling on the unlabeled samples to meet the requirements for the number of labeled samples for model training. However, if the labeling model built with a small number of labeled samples is used to perform pseudo-labeling on the unlabeled samples, the accuracy of the pseudo-labeling cannot be guaranteed because the accuracy of the labeling model is not high. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, electronic device, medium and program product for labeling fake labels, which aims to solve the technical problem of low accuracy of fake label labeling in the prior art.

[0005] To achieve the above objectives, this application provides a pseudo-labeling method, which includes:

[0006] Obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo label sample set;

[0007] The initial pseudo-label sample set is divided into a first pseudo-label sample set and a second pseudo-label sample set;

[0008] Based on the second labeling model jointly constructed from the real label sample set and the first pseudo label sample set, the second pseudo label sample set is re-labeled to obtain the third pseudo label sample set;

[0009] Calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set;

[0010] The target pseudo-label sample set is filtered by removing samples whose label difference values ​​are greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0011] This application also provides a joint modeling optimization method applied to a first model builder, the joint modeling optimization method comprising:

[0012] Obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo label sample set;

[0013] The initial pseudo-label sample set is divided into a first pseudo-label sample set and a second pseudo-label sample set;

[0014] Based on the second labeling model jointly constructed from the real label sample set and the first pseudo label sample set, the second pseudo label sample set is re-labeled to obtain the third pseudo label sample set;

[0015] Calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set;

[0016] The target pseudo-label sample set is filtered by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0017] By aligning samples with the second model builder, each aligned sample is selected from the target pseudo-label sample set;

[0018] The sample identifiers and pseudo-labels corresponding to each aligned sample are sent to the second model builder so that the second model builder can build the corresponding user qualification assessment model.

[0019] This application also provides a pseudo-label marking device, which is a virtual device, and the pseudo-label marking device includes:

[0020] The first labeling module is used to obtain a real label sample set, and to label the unlabeled sample set according to the first labeling model constructed based on the real label sample set, so as to obtain an initial pseudo label sample set.

[0021] The segmentation module is used to segment the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set.

[0022] The second labeling module is used to re-label the second pseudo-label sample set based on the second labeling model jointly constructed by the real label sample set and the first pseudo-label sample set, to obtain the third pseudo-label sample set.

[0023] The label difference calculation module is used to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set.

[0024] The pseudo-label sample screening module is used to screen the target pseudo-label sample set by removing samples whose label difference value is greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0025] This application also provides a joint modeling optimization device, applied to a first model builder, the joint modeling optimization device comprising:

[0026] The first labeling module is used to obtain a real label sample set, and to label the unlabeled sample set according to the first labeling model constructed based on the real label sample set, so as to obtain an initial pseudo label sample set.

[0027] The segmentation module is used to segment the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set.

[0028] The second labeling module is used to re-label the second pseudo-label sample set based on the second labeling model jointly constructed by the real label sample set and the first pseudo-label sample set, to obtain the third pseudo-label sample set.

[0029] The label difference calculation module is used to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set.

[0030] The pseudo-label sample filtering module is used to filter the target pseudo-label sample set by removing samples whose label difference value is greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0031] The sample alignment module is used to filter aligned samples in the target pseudo-label sample set by aligning samples with the second model builder.

[0032] The joint modeling module is used to send the sample identifiers and pseudo-labels corresponding to each of the aligned samples to the second model builder, so that the second model builder can build the corresponding user qualification assessment model.

[0033] This application also provides an electronic device, the electronic device comprising: a memory, a processor, and a program of the pseudo-labeling method stored in the memory and executable on the processor, wherein when the program of the pseudo-labeling method is executed by the processor, it can implement the steps of the pseudo-labeling method as described above.

[0034] This application also provides an electronic device, the electronic device comprising: a memory, a processor, and a program of the joint modeling optimization method stored in the memory and executable on the processor, wherein when the program of the joint modeling optimization method is executed by the processor, it can implement the steps of the joint modeling optimization method as described above.

[0035] This application also provides a computer-readable storage medium storing a program that implements a pseudo-labeling method, wherein when the program is executed by a processor, it implements the steps of the pseudo-labeling method as described above.

[0036] This application also provides a computer-readable storage medium storing a program implementing a joint modeling optimization method, wherein when the program is executed by a processor, it implements the steps of the joint modeling optimization method as described above.

[0037] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the pseudo-labeling method described above.

[0038] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the joint modeling optimization method described above.

[0039] This application provides a pseudo-labeling method, apparatus, electronic device, medium, and program product. Compared to existing technologies that directly use a labeling model constructed from a small number of labeled samples to pseudo-label unlabeled samples, this application first labels the unlabeled sample set using a first labeling model constructed from labeled samples to obtain an initial pseudo-labeled sample set. Then, the initial pseudo-labeled sample set is divided into a first pseudo-labeled sample set and a second pseudo-labeled sample set. A second labeling model is constructed using a local full sample set that mixes the real labeled sample set and the first pseudo-labeled sample set. Then, based on the second labeling model, the second pseudo-labeled sample set is re-labeled, thereby utilizing the local distribution information of the full sample set to correct the labels on the second pseudo-labeled sample set, resulting in a third pseudo-labeled sample set. The calculation of the first pseudo-labeled sample set... If the label difference value between the pseudo-labels in the second pseudo-label sample set and the third pseudo-label sample set is greater than a preset label difference threshold, it proves that there is a discrepancy between the labeling of the sample by the first labeling model and the second labeling model. Therefore, it is determined that the pseudo-label for the sample is not accurate enough. As a result, samples with label difference values ​​greater than the preset label difference threshold are removed from the second pseudo-label sample set and the third pseudo-label sample set. Finally, a target pseudo-label sample set with more accurate pseudo-labeling is obtained. This achieves accurate pseudo-labeling of unlabeled samples using a small number of labeled samples. It overcomes the technical defect in the prior art that if a labeling model built with a small number of labeled samples is used to pseudo-label unlabeled samples, the accuracy of pseudo-labeling cannot be guaranteed due to the low accuracy of the labeling model. This improves the accuracy of pseudo-labeling. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the first embodiment of the pseudo-labeling method of this application;

[0043] Figure 2 This is a flowchart illustrating the joint modeling method based on pseudo-label learning in the pseudo-label annotation method of this application;

[0044] Figure 3This is a flowchart illustrating the second embodiment of the pseudo-labeling method of this application;

[0045] Figure 4 This is a schematic diagram of an embodiment of the pseudo-label marking device of this application;

[0046] Figure 5 This is a schematic diagram of an embodiment of the joint modeling and optimization device of this application;

[0047] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the pseudo-labeling method in this application embodiment.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Example 1

[0051] This application provides a pseudo-labeling method. In the first embodiment of the pseudo-labeling method of this application, refer to... Figure 1 The pseudo-labeling method includes:

[0052] Step S10: Obtain the real label sample set, and label the unlabeled sample set according to the first label annotation model constructed based on the real label sample set to obtain the initial pseudo label sample set;

[0053] Step S20: Divide the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set;

[0054] Step S30: Based on the second labeling model jointly constructed by the real label sample set and the first pseudo label sample set, the second pseudo label sample set is re-labeled to obtain the third pseudo label sample set;

[0055] Step S40: Calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set;

[0056] Step S50: Filter the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0057] In this embodiment, it should be noted that, to ensure the accuracy of message recommendations, the message recommendation side typically trains a qualification assessment model to evaluate user qualifications, thereby achieving accurate message recommendations for users with good qualifications. However, training the user qualification assessment model requires a massive amount of real customer samples. These real customer samples have real sample labels, so they are also called real-label samples. These real sample labels are used to identify the level of a user's real qualifications. For most message recommendation sides, there are usually only a few real sample labels, and most samples are unlabeled samples, that is, non-customer samples. If the qualification assessment model is built based on only a few real sample labels, the prediction accuracy of the qualification assessment model cannot be guaranteed. On the other hand, although the unlabeled samples are later manually labeled with pseudo-labels, and the unlabeled samples with manually labeled pseudo-labels are used to train the qualification assessment model, the efficiency of manual labeling is low.

[0058] As an example, steps S10 to S50 include: obtaining a real label sample set; using the real label sample set as a training set to iteratively train and construct a first label annotation model; inputting samples from the unlabeled sample set into the first label annotation model to label the unlabeled sample set, obtaining an initial pseudo-label sample set; randomly dividing the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set; using the real label sample set and the first pseudo-label sample set together as a training set to iteratively train and construct a second label annotation model with local preservation characteristics, wherein the second label annotation model can be a KNN (K-Nearest Neighbor) model or an inverse distance weighted reconstruction model, etc.; inputting each sample from the second pseudo-label sample set into the second label annotation model to re-label each sample from the second pseudo-label sample set, obtaining a third pseudo-label sample set; calculating the pseudo-labels corresponding to the same samples in the second pseudo-label sample set and the third pseudo-label sample set. The label differences between labels are used to obtain label difference values. Abnormal samples with label difference values ​​greater than a preset label difference threshold are removed from the second and third pseudo-label sample sets to obtain a target pseudo-label sample set. The target pseudo-label sample set includes the remaining normal samples (excluding abnormal samples) from the third pseudo-label sample set, the pseudo-labels of the remaining normal samples in the second pseudo-label sample set, and the pseudo-labels of the remaining normal samples in the third pseudo-label sample set. The samples in the second and third pseudo-label sample sets are identical, but the pseudo-labels corresponding to the same samples are different. The pseudo-labels corresponding to samples in the second pseudo-label sample set are labeled using a first label labeling model, and the pseudo-labels corresponding to samples in the third pseudo-label sample set are labeled using a second label labeling model. The pseudo-labels can be label vectors, which represent the quality of a user's qualifications. These label vectors can be normalized into user qualification scores, or they can be the user qualification scores themselves.

[0059] It should be noted that a sample can be a vector or matrix composed of multiple sample feature values. These sample features can construct a high-dimensional space (the number of dimensions of the space equals the number of sample features). The sample label corresponding to a sample is a low-dimensional differentiable manifold embedded in this feature space. Here, the low dimension can be a single dimension of user qualification. Differentiability means that the low-dimensional differentiable manifold is smooth, that is, the user qualification indicator has continuity. Thus, the sample labels corresponding to each sample in the true sample label set and the initial pseudo-label sample set can be considered as samples on this low-dimensional differentiable manifold. The sample labels corresponding to each sample in the true sample label set and the initial pseudo-label sample set should have the same data form in the feature space. Since the number of samples in the initial pseudo-label sample set is usually much larger than the number of samples in the true sample label set, the sampling density of the initial pseudo-label sample set on this low-dimensional differentiable manifold is greater than that of the true sample label set. Therefore, the data distribution of the sample labels corresponding to each sample in the initial pseudo-label sample set in the feature space is smoother than the data distribution of the sample labels corresponding to each sample in the true sample label set in the feature space.

[0060] Based on the above, in this embodiment, after labeling unlabeled samples using the first labeling model to obtain an initial pseudo-sample set, the initial pseudo-label sample set is divided into a first pseudo-label sample set and a second pseudo-label sample set. Then, the real label sample set and the first pseudo-label sample set are mixed into a full sample set. The second labeling model with local preservation characteristics is retrained using this full sample set. By using the second labeling model, the second pseudo-label sample set is re-labeled to obtain a third pseudo-label sample set. Thus, by utilizing the label difference values ​​between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed from both sets. That is, samples that do not conform to the local structure of the labeled pseudo-labels in the feature space are removed. This makes the local structure of the labeled pseudo-labels corresponding to each sample in the second pseudo-label sample set and the labeled pseudo-labels corresponding to each sample in the third pseudo-label sample set as consistent as possible in the feature space. Therefore, the purpose of correcting the labeled pseudo-labels using the local structure of the full sample set is achieved, improving the accuracy of pseudo-label labeling.

[0061] It should be noted that the local structure of data refers to the structural form formed by the data and its neighborhood in a specific geometric space (usually multidimensional Euclidean space or manifold). The significance of this structural form is that when performing data compression or other affine transformations (such as dimensionality reduction or feature extraction), specific structure-preserving functions (such as inverse distance weighted reconstruction) can be used as constraints to ensure that the transformed data retains effective structural information and avoids the loss of effective information caused by compression.

[0062] As an example, the step of randomly dividing the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set includes:

[0063] According to a preset segmentation ratio, the initial pseudo-label sample set is randomly segmented into a first pseudo-label sample set and a second pseudo-label sample set. The preset segmentation ratio can be determined based on the ratio of the number of samples in the real label sample set to the number of samples in the unlabeled sample set.

[0064] The step of re-labeling the second pseudo-label sample set using the second label annotation model jointly constructed from the real label sample set and the first pseudo-label sample set includes:

[0065] Step S31: Mix the real label sample set and the first pseudo label sample set into a full sample set;

[0066] Step S32: Iteratively train a second labeling model with local preservation properties using the full sample set;

[0067] Step S33: Re-label the second pseudo-label sample set according to the second label annotation model.

[0068] As an example, steps S31 to S33 include: mixing the real label sample set and the first pseudo-label sample set into a full sample set; using the full sample set as a training sample set to iteratively train a second label annotation model with local preservation characteristics, wherein the second label annotation model can be a KNN (K-Nearest Neighbor) model or an inverse distance weighted reconstruction model, etc.; by inputting each sample in the second pseudo-label sample set into the second label annotation model, re-labeling each sample in the second pseudo-label sample set to obtain a third pseudo-label sample set. Thus, this embodiment of the application utilizes the label difference values ​​between the pseudo-labels annotated in the second pseudo-label sample set and the third pseudo-label sample set to remove samples in the second and third pseudo-label sample sets that do not conform to the local structure of the labeled pseudo-labels in the feature space, making the local structure of the labeled pseudo-labels corresponding to each sample in the second and third pseudo-label sample sets as consistent as possible in the feature space, achieving the purpose of correcting the labeled pseudo-labels using the local structure of the full sample set, thereby improving the accuracy of pseudo-label annotation.

[0069] In the first model builder that uses the target pseudo-label sample set for joint modeling, after the step of filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second and third pseudo-label sample sets, the pseudo-label annotation method further includes:

[0070] Step S60: Align samples with the second model builder and filter the aligned samples in the target pseudo-label sample set.

[0071] Step S70: Send the sample identifier and pseudo-label corresponding to each aligned sample to the second model builder so that the second model builder can build the corresponding user qualification assessment model.

[0072] In this implementation, it should be noted that the first model builder can be the message recommendation side, the second model builder can be the media side, and the samples in the real tag sample set can be samples composed of user credit data feature values, such as loan information or deposit information. Currently, in order to improve the accuracy of the credit assessment model by leveraging samples from the media side, the message recommendation side typically uses federated learning to build the credit assessment model. To protect the data privacy of all parties involved in federated learning, it is usually based on encryption methods such as homomorphic encryption, secret sharing, or differential privacy. In this way, while protecting the data privacy of both the message recommendation side and the media side, the credit data from the message recommendation side and the rich user profiles from the media side can be combined to build a more accurate user credit assessment model. However, in federated learning systems, firstly, federated learning environments need to be deployed separately on the data partners (i.e., the message recommendation side and the media side), which presents certain technical hurdles and resource limitations, and also requires certain maintenance costs; secondly, the model training algorithm needs to be customized in a distributed version, which limits the flexibility of model training; and thirdly, since the data transmitted between data partners is encrypted data, the volume of encrypted data is much larger than that of plaintext data, which will affect the iterative efficiency of model training.

[0073] As an example, steps S60 to S70 include: aligning samples with the media side, filtering the intersection samples between the message recommendation side and the media side in the target pseudo-label sample set, and using these intersection samples as alignment samples; sending the sample identifiers and pseudo-labels corresponding to each alignment sample to the media side model builder, so that the media side model builder can use the local samples corresponding to each identifier and the pseudo-labels corresponding to the local samples as a training set to build the corresponding user qualification assessment model. In this way, after the message recommendation side filters out the target pseudo-label sample set, since the pseudo-labels are not the real labels of the samples, they can be directly sent to the media side in plaintext form. Therefore, the media side can directly train the user qualification assessment model locally based on the pseudo-labels output by the message recommendation side. Compared to federated learning, this embodiment of the application has the following advantages: firstly, it eliminates the need for deployment and maintenance of the underlying technical framework of federated learning; secondly, it also protects the data privacy of the message recommendation side and the media side; and thirdly, the message recommendation side can directly send the pseudo-labels as plaintext data without sending encrypted data, thus improving the iterative efficiency of model training.

[0074] As an example, refer to Figure 2 , Figure 2 This is a flowchart illustrating the joint modeling method based on pseudo-label learning in this application embodiment. Specifically, the message recommendation side first trains a first label annotation model using a real label sample set. Then, it uses the first label annotation model to label unlabeled data (non-customer data) to obtain an initial pseudo-label sample set. The initial pseudo-label sample set is then split into a first pseudo-label sample set and a second pseudo-label sample set. The real label sample set and the first pseudo-label sample set are mixed to form a full sample set (labeled data). A second label annotation model is trained using the full sample set, and then the second label annotation model is used to apply labels to the second pseudo-labels. The sample set is re-labeled to obtain the third pseudo-label sample set; based on the label difference value between the pseudo-labels labeled in the second and third pseudo-label sample sets, the target pseudo-label sample set is selected from the labeled data; the message recommendation side can then encrypt and transmit the target pseudo-label sample set to the media side for modeling. Here, the labels are used to identify the level of user qualifications. In this embodiment, user qualifications can be divided into three levels: high qualification, medium qualification, and low qualification. The labeled data is a full sample set that mixes the real label sample set and the first pseudo-label sample set, and the unlabeled data is a sample set without labels.

[0075] This application provides a pseudo-labeling method. Compared to existing techniques that directly use a labeling model built from a small number of labeled samples to pseudo-label unlabeled samples, this application first labels the unlabeled sample set using a first labeling model built from labeled samples to obtain an initial pseudo-labeled sample set. Then, the initial pseudo-labeled sample set is divided into a first pseudo-labeled sample set and a second pseudo-labeled sample set. A second labeling model is constructed using a local full sample set that mixes the real labeled sample set and the first pseudo-labeled sample set. Then, the second pseudo-labeled sample set is re-labeled according to the second labeling model, thereby utilizing the local distribution information of the full sample set to correct the labels of the second pseudo-labeled sample set, resulting in a third pseudo-labeled sample set. The second pseudo-label is then calculated. If the label difference value between the pseudo-labels in the sample set and the third pseudo-label sample set is greater than a preset label difference threshold, it proves that there is a discrepancy between the labeling of the sample by the first labeling model and the second labeling model. Therefore, it is determined that the pseudo-label for the sample is not accurate enough. As a result, samples with label difference values ​​greater than the preset label difference threshold are removed from both the second pseudo-label sample set and the third pseudo-label sample set. Finally, a target pseudo-label sample set with more accurate pseudo-labeling is obtained. This achieves accurate pseudo-labeling of unlabeled samples using a small number of labeled samples. It overcomes the technical defect in the prior art that if a labeling model built with a small number of labeled samples is used to pseudo-label unlabeled samples, the accuracy of pseudo-labeling cannot be guaranteed due to the low accuracy of the labeling model. This improves the accuracy of pseudo-labeling.

[0076] Example 2

[0077] Furthermore, referring to Figure 3 Based on the above embodiments, in another embodiment of the pseudo-labeling method of this application, the step of filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set includes:

[0078] Step S51: Remove samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set to obtain each output pseudo-label sample.

[0079] Step S52: Obtain each verification sample, and classify each verification sample into user qualification categories according to the verification label pseudo-labels corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-labels are obtained by labeling the verification samples using the first label labeling model or the second label labeling model.

[0080] Step S53: Obtain the real business indicator values ​​corresponding to each of the verification samples. Based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

[0081] In this embodiment, it should be noted that the verification sample can be a reserved real sample. The reserved real sample can be a sample with real labels that is reserved in advance before the first label annotation model is constructed. Therefore, after the first label annotation model and the second label annotation model are constructed, the reserved real sample can be labeled using the first label annotation model and the second label annotation model respectively to obtain the verification label pseudo label corresponding to the reserved real sample.

[0082] As an example, steps S51 to S53 include: removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set to obtain each output pseudo-label sample; obtaining reserved real samples, and labeling the reserved real samples according to the first label labeling model and the second label labeling model to obtain each verification label pseudo-label; determining the qualification score corresponding to each reserved real sample according to the verification label pseudo-label corresponding to each reserved real sample, wherein if the verification label pseudo-label is the qualification score itself, the verification label pseudo-label can be directly used as the qualification score, and if the... If the pseudo-labels are verified to be label vectors, then the label vectors can be normalized to qualification scores. Based on the qualification scores corresponding to each reserved real sample, the user qualification tier corresponding to each reserved real sample is determined. Based on the real business indicator values ​​corresponding to the reserved real samples under each user qualification tier, the tier business indicator statistics under each user qualification tier are calculated. Based on the tier business indicator statistics, it is verified whether the user qualification tiering results meet the preset order preservation requirements, and the order preservation verification results are obtained. Based on the order preservation verification results, a target pseudo-label sample set is selected from the second pseudo-label sample set and the third pseudo-label sample set.

[0083] Additionally, it should be noted that the verification sample can be a converted sample, which can be a customer sample converted from the unlabeled sample set over time, that is, a converted real labeled sample. The second pseudo-label sample set and the third pseudo-label sample set have the same samples, but the pseudo-labels corresponding to the samples are different.

[0084] As an example, steps S51 to S53 include: removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set to obtain each output pseudo-label sample; after a sample conversion from the unlabeled sample set to the real label sample set occurs, determining each converted sample from the unlabeled sample set to the real label sample set in each of the output pseudo-label samples, wherein the converted sample is a customer sample converted from a non-customer sample; determining the qualification score corresponding to each converted sample based on the verification label pseudo-label corresponding to each converted sample, wherein if the verification label pseudo-label is the qualification score itself, then it can be directly... The verification label pseudo-label serves as the qualification score. If the verification label pseudo-label is a label vector, the label vector can be normalized to obtain the qualification score. Based on the qualification score corresponding to each conversion sample, the user qualification tier corresponding to each conversion sample is determined. Based on the actual business indicator value corresponding to the conversion sample under each user qualification tier, the tier business indicator statistical value under each user qualification tier is calculated. Based on the tier business indicator statistical value, the user qualification tier result is verified to meet the preset order preservation requirement, and the order preservation verification result is obtained. Based on the order preservation verification result, a target pseudo-label sample set is selected from the second pseudo-label sample set and the third pseudo-label sample set.

[0085] It should be noted that the higher the user qualification tier corresponding to the sample, the higher the corresponding user qualification. The preset order preservation requirement means that the higher the user qualification tier, the higher the statistical value of the tier business indicator under the user qualification tier. In other words, the statistical value of the tier business indicator under the higher user qualification tier should be higher than the statistical value of the tier business indicator under the lower user qualification tier. The statistical value of the tier business indicator can be the average, proportion, standard deviation or variance of the actual business indicator values ​​under the user qualification tier.

[0086] As an example, the real business indicator value can be one or more of the following: an indicator value representing whether a financial loan has been approved, an indicator value representing the amount of a financial loan, and an indicator value representing whether a financial loan application has been generated. Correspondingly, the statistical value of the tiered business indicator can be the approval rate of financial loans, the average amount of financial loans, and the number of financial loan applications generated under the user's qualification tier.

[0087] The step of filtering the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements based on the actual business indicator values ​​includes:

[0088] Step S531: Based on the actual business indicator values, verify whether the user qualification classification results meet the preset order preservation requirements.

[0089] Step S532: If the condition is met, then all the output pseudo-label samples are used together as the target pseudo-label sample set.

[0090] If step S533 is not satisfied, return to the following steps: obtain the real label sample set, and label the unlabeled sample set according to the first label annotation model constructed based on the real label sample set to obtain the initial pseudo label sample set.

[0091] As an example, steps S531 to S533 include: calculating the statistical value of the business indicator for each user qualification tier based on the real business indicator value corresponding to the verification sample under each user qualification tier; calculating the difference between the statistical values ​​of the business indicator corresponding to each adjacent higher user qualification tier and lower user qualification tier, obtaining the difference of each statistical value; if all the differences of each statistical value are positive, it is determined that each user qualification tier result meets the preset order preservation requirement, and all the output pseudo-label samples are used together as the target pseudo-label sample set; if all the differences of each statistical value are not positive, it is determined that each user qualification tier result does not meet the preset order preservation requirement, and the process returns to the execution step: obtaining the real label sample set, and labeling the unlabeled sample set according to the first label labeling model constructed from the real label sample set, to obtain the initial pseudo-label sample set. This application realizes that by detecting whether the results of the above pseudo-label labeling meet the preset order preservation requirement, iteratively eliminating samples with inaccurate pseudo-label labeling can be further improved.

[0092] The user qualification classification results include a first user qualification classification result and a second user qualification classification result.

[0093] The step of classifying each verification sample into user qualification categories based on the verification annotation pseudo-labels corresponding to each verification sample, and obtaining the user qualification classification results, includes:

[0094] Step S521: Obtain the verification annotation pseudo-labels of the first label annotation model for the verification sample annotation as initial pseudo-labels, and the verification annotation pseudo-labels of the second label annotation model for the verification sample annotation as correction pseudo-labels;

[0095] Step S522: Based on the first tag score corresponding to each initial pseudo tag, perform user qualification classification on each verification sample to obtain the first user qualification classification result;

[0096] Step S523: Based on the second tag score corresponding to each of the modified pseudo-tags, perform user qualification classification on each of the verification samples to obtain the second user qualification classification result.

[0097] In this embodiment, it should be noted that the first user qualification classification result includes at least a first user qualification classification level, and the second user qualification classification result includes at least a second user qualification classification level.

[0098] As an example, steps S521 to S523 include: obtaining the verification annotation pseudo-labels of the verification samples labeled by the first label annotation model as initial pseudo-labels, and the verification annotation pseudo-labels of the verification samples labeled by the second label annotation model as corrected pseudo-labels; classifying each verification sample into a corresponding first user qualification tier based on the first label score corresponding to each initial pseudo-label, wherein the higher the first label score, the higher the corresponding first user qualification tier; and classifying each verification sample into a corresponding second user qualification tier based on the second label score corresponding to each initial pseudo-label, wherein the higher the second label score, the higher the corresponding second user qualification tier.

[0099] The first user qualification classification result includes at least one first user qualification classification level, and the second user qualification classification result includes at least one second user qualification classification level.

[0100] The step of verifying whether the user qualification classification result meets the preset order preservation requirements based on each of the real business indicator values ​​includes:

[0101] Step A10: Calculate the first-tier business indicator statistical value corresponding to the actual business indicator value under each of the first user qualification tiers, and the second-tier business indicator statistical value corresponding to the actual business indicator value under each of the second user qualification tiers.

[0102] Step A20: Based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators, verify whether the user qualification classification results meet the preset order preservation requirements.

[0103] In this embodiment, it should be noted that the statistical value of the first tier business indicator can be the average, percentage, standard deviation, or variance of the actual business indicator values ​​under the first user qualification tier, and the statistical value of the second tier business indicator can be the average, percentage, standard deviation, or variance of the actual business indicator values ​​under the second user qualification tier.

[0104] As an example, steps A10 to A20 include: calculating the first-tier business indicator statistical value corresponding to the actual business indicator value under each first user qualification tier, and the second-tier business indicator statistical value corresponding to the actual business indicator value under each second user qualification tier; detecting whether the value of each first-tier business indicator statistical value matches the corresponding first user qualification tier, and whether the value of each second-tier business indicator statistical value matches the corresponding second user qualification tier; if the value of each first-tier business indicator statistical value matches the corresponding first user qualification tier, and the value of each second-tier business indicator statistical value matches the corresponding second user qualification tier, then the user qualification tiering result is determined to meet the preset order preservation requirement; otherwise, the user qualification tiering result is determined to not meet the preset order preservation requirement, wherein, the higher the first user qualification tier, the larger the value of the corresponding first-tier business indicator statistical value, and the higher the second user qualification tier, the larger the value of the corresponding second-tier business indicator statistical value.

[0105] The step of verifying whether the user qualification classification result meets the preset order preservation requirements based on the difference values ​​of the indicators of each first tier and the difference values ​​of the indicators of each second tier includes:

[0106] Step A31: Detect whether the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase as the user's qualification level increases;

[0107] Step A32: If yes, then determine that the user qualification classification result meets the preset order preservation requirement;

[0108] Step A33: If not, then it is determined that the user qualification classification result does not meet the preset order preservation requirements.

[0109] As an example, steps A31 to A32 include: detecting whether the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase with the increase of the user qualification level; if the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators do not both increase with the increase of the user qualification level, then it is determined that the user qualification classification result does not meet the preset order preservation requirement; if the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase with the increase of the user qualification level, then the adjacent higher first user qualification level and lower first... The differences between the statistical values ​​of business indicators in the first tier of user qualification tiers are used to obtain the differences of each first statistical value. The differences between the statistical values ​​of business indicators in the first tier of adjacent higher and lower second user qualification tiers are also calculated to obtain the differences of each second statistical value. If both the differences of each first statistical value and each difference of each second statistical value are greater than a preset difference threshold, the user qualification tiering result is determined to meet the preset order preservation requirement. If neither the differences of each first statistical value nor each difference of each second statistical value are greater than the preset difference threshold, the user qualification tiering result is determined to not meet the preset order preservation requirement. Thus, the final target pseudo-label sample set meets the preset order preservation requirement, meaning the pseudo-labels assigned to the samples meet the preset order preservation requirement. This achieves the goal of assigning higher-scoring pseudo-labels to samples with higher user qualifications and lower-scoring pseudo-labels to samples with lower user qualifications, thereby improving the accuracy of pseudo-label annotation.

[0110] This application provides a pseudo-label annotation method. Specifically, after calculating the label difference value, samples with label difference values ​​greater than a preset label difference threshold are removed from the second and third pseudo-label sample sets to obtain output pseudo-label samples. Each output pseudo-label sample is then used to identify verification samples that have been transformed from the unlabeled sample set to the real label sample set. Based on the verification annotation pseudo-labels corresponding to each verification sample, user qualification classification is performed on each verification sample to obtain user qualification classification results. The corresponding real business indicator values ​​are obtained, and based on these real business indicator values, the user qualification classification results are verified to meet a preset order preservation requirement to filter the target pseudo-label sample set. This ensures that the annotated pseudo-labels are more closely matched with the corresponding real business indicator values, avoiding situations where the real business performance of higher user qualification classification levels is better than that of lower user qualification classification levels. This achieves improved accuracy of pseudo-label annotation based on real business conditions.

[0111] Example 3

[0112] Furthermore, based on the above embodiments, in another embodiment of the pseudo-labeling method of this application, the step of verifying whether the user qualification classification result meets the preset order preservation requirements based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators includes:

[0113] Step B10: If the statistical values ​​of the first-level business indicators and the statistical values ​​of the second-level business indicators do not increase with the increase of the user qualification level, then it is determined that the user qualification classification result does not meet the preset order preservation requirement.

[0114] Step B20: If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase with the increase of the user qualification level, then detect whether the significance of the difference between the statistical values ​​of the first-tier business indicators is higher than the significance of the difference between the statistical values ​​of the second-tier business indicators.

[0115] Step B30: If yes, then determine that the user qualification classification result meets the preset order preservation requirement;

[0116] Step B40: If not, it is determined that the user qualification classification result does not meet the preset order preservation requirements.

[0117] In this embodiment, it should be noted that the significance of the difference can be the significance of the difference between the statistical values ​​of the business indicators of adjacent user qualification tiers. The greater the difference between the statistical values ​​of the business indicators of adjacent user qualification tiers, the higher the significance of the difference.

[0118] As an example, steps B10 to B40 include: if the statistical values ​​of each first-tier business indicator and each second-tier business indicator do not all increase with the increase of the user qualification tier, then it is determined that the user qualification grading result does not meet the preset order preservation requirement; if the statistical values ​​of each first-tier business indicator and each second-tier business indicator all increase with the increase of the user qualification tier, then the difference between the statistical values ​​of the first-tier business indicators under adjacent higher first user qualification tiers and lower first user qualification tiers is calculated to obtain the difference of each first statistical value, and the difference between the statistical values ​​of the first-tier business indicators under adjacent higher second user qualification tiers and lower second user qualification tiers is calculated. The differences between statistical values ​​are used to obtain the differences between each second statistical value. Based on the differences between each first statistical value and each second statistical value, it is determined whether the significance of the differences between the statistical values ​​of each first-tier business indicator is higher than the significance of the differences between the statistical values ​​of each second-tier business indicator. If the significance of the differences between the statistical values ​​of each first-tier business indicator is higher than the significance of the differences between the statistical values ​​of each second-tier business indicator, then it is determined that the user qualification classification result meets the preset order preservation requirement. If the significance of the differences between the statistical values ​​of each first-tier business indicator is not higher than the significance of the differences between the statistical values ​​of each second-tier business indicator, then it is determined that the user qualification classification result does not meet the preset order preservation requirement.

[0119] As an example, the step of detecting whether the significance of the difference between the statistical values ​​of the first-tier business indicators is higher than the significance of the difference between the statistical values ​​of the second-tier business indicators based on the difference between the first statistical values ​​and the difference between the second statistical values ​​includes:

[0120] If the difference in the second statistical value under the same user qualification level is greater than the difference in the first statistical value, then the difference between the statistical values ​​of the business indicators of each first level is determined to be more significant than the difference between the statistical values ​​of the business indicators of each second level; if the difference in the second statistical value under the same user qualification level is not greater than the difference in the first statistical value, then the difference between the statistical values ​​of the business indicators of each first level is determined to be less significant than the difference between the statistical values ​​of the business indicators of each second level.

[0121] As an example, the step of detecting whether the significance of the difference between the statistical values ​​of the first-tier business indicators is higher than the significance of the difference between the statistical values ​​of the second-tier business indicators based on the difference between the first statistical values ​​and the difference between the second statistical values ​​includes:

[0122] Obtain the first value range of the difference between each first statistical value and the second value range of the difference between each second statistical value; if the lower limit of the second value range is greater than the upper limit of the first value range, then determine that the difference between the statistical values ​​of each first-level business indicator is more significant than the difference between the statistical values ​​of each second-level business indicator; if the lower limit of the second value range is not greater than the upper limit of the first value range, then determine that the difference between the statistical values ​​of each first-level business indicator is not more significant than the difference between the statistical values ​​of each second-level business indicator.

[0123] This application provides an order-preservation verification method. Specifically, if the statistical values ​​of each first-tier business indicator and each second-tier business indicator do not increase with the increase of the user qualification tier, then the user qualification grading result is determined to not meet the preset order-preservation requirement. If the statistical values ​​of each first-tier business indicator and each second-tier business indicator both increase with the increase of the user qualification tier, then it is detected whether the significance of the difference between the statistical values ​​of each first-tier business indicator is higher than the significance of the difference between the statistical values ​​of each second-tier business indicator. If yes, then the user qualification grading result is determined to meet the preset order-preservation requirement; otherwise, the user qualification grading result is determined to not meet the preset order-preservation requirement. This makes the pseudo-labels obtained from the annotation more closely match the corresponding real business metric values, which can improve the accuracy of pseudo-label annotation. Furthermore, the accuracy of the corrected pseudo-labels is higher than that of the initial pseudo-labels. Based on this, the first label annotation model and the second label annotation model are retrained iteratively. This ensures that the label annotation accuracy of the second label annotation model is higher than that of the first label annotation model. The second label annotation model will correct the pseudo-labels annotated by the first label annotation model in a more accurate direction, thus further improving the accuracy of pseudo-label annotation.

[0124] Example 4

[0125] This application provides a joint modeling optimization method. In a first embodiment of the joint modeling optimization method, it is applied to a first model builder. The joint modeling optimization method includes:

[0126] Step C10: Obtain the real label sample set, and label the unlabeled sample set according to the first label annotation model constructed based on the real label sample set to obtain the initial pseudo label sample set;

[0127] Step C20: Divide the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set;

[0128] Step C30: Based on the second labeling model jointly constructed by the real label sample set and the first pseudo label sample set, the second pseudo label sample set is re-labeled to obtain the third pseudo label sample set;

[0129] Step C40: Calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set;

[0130] Step C50: Filter the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set;

[0131] Step C60: Align samples with the second model builder and filter the aligned samples in the target pseudo-label sample set.

[0132] Step C70: Send the sample identifier and pseudo-label corresponding to each aligned sample to the second model builder so that the second model builder can build the corresponding user qualification assessment model.

[0133] In this embodiment, it should be noted that the first model builder can be the message recommendation side, and the second model builder can be the media side. Currently, in order to improve the accuracy of the qualification assessment model by leveraging samples from the media side, the message recommendation side typically uses federated learning to build the qualification assessment model. To protect the data privacy of all parties involved in federated learning, it is usually based on encryption methods such as homomorphic encryption, secret sharing, or differential privacy. This way, while protecting the data privacy of both the message recommendation side and the media side, it can combine the credit data from the message recommendation side and the rich user profiles from the media side to build a more accurate user qualification assessment model. However, in a federated learning system, firstly, it requires deploying federated learning environments separately on the data partners (i.e., the message recommendation side and the media side), which presents certain technical hurdles and resource limitations, and also incurs certain maintenance costs; secondly, the model training algorithm needs to be customized in a distributed version, limiting the flexibility of model training; and thirdly, since the data transmitted between data partners is encrypted data, the volume of encrypted data is much larger than that of plaintext data, thus affecting the iterative efficiency of model training.

[0134] As an example, the specific implementation steps of steps C10 to C70 in this embodiment can refer to the specific implementation process of steps S10 to S70 above, and will not be repeated here.

[0135] This application provides a joint modeling optimization method. A first model builder uses a first labeling model constructed from labeled samples to label an unlabeled sample set, obtaining an initial pseudo-label sample set. Then, the initial pseudo-label sample set is divided into a first pseudo-label sample set and a second pseudo-label sample set. A second labeling model is constructed using a local full sample set composed of the real label sample set and the first pseudo-label sample set. Then, based on the second labeling model, the second pseudo-label sample set is re-labeled, thereby utilizing the local distribution information of the full sample set to correct the labels of the second pseudo-label sample set, resulting in a third pseudo-label sample set. The label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set is calculated. If the label difference value is greater than a preset label difference threshold, it indicates a discrepancy between the first and second labeling models in the labeling of the sample, thus determining that the pseudo-label for that sample is inaccurate. Therefore, the method is used to refine the labeling of the sample in the second pseudo-label sample set and the third pseudo-label sample set. The third set of pseudo-label samples removes samples with label difference values ​​greater than a preset label difference threshold, and finally selects a target pseudo-label sample set with more accurate pseudo-label annotation. This achieves accurate pseudo-label annotation of unlabeled samples using a small number of labeled samples. After the first model builder selects the target pseudo-label sample set, since the pseudo-labels are not the real labels of the samples, they can be sent directly to the second model builder in plaintext. The second model builder can then directly train the user qualification assessment model locally based on the pseudo-labels output by the first model builder. Compared with federated learning, in this embodiment, the first aspect is that there is no need to deploy and maintain the underlying technical framework of federated learning; the second aspect is that it can also protect the data privacy of the message recommendation side and the media side; and the third aspect is that the message recommendation side can directly send the pseudo-labels as plaintext data without sending encrypted data, which reduces the amount of communication data between the message recommendation side and the media side when iteratively training the user qualification assessment model, improves communication efficiency, and thus improves the iterative training efficiency of the user qualification assessment model.

[0136] Example 5

[0137] Reference Figure 4 This application also provides a pseudo-label marking device, which includes:

[0138] The first labeling module 10A is used to obtain a real label sample set, and to label the unlabeled sample set according to the first labeling model constructed based on the real label sample set, so as to obtain an initial pseudo label sample set.

[0139] The segmentation module 20A is used to segment the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set.

[0140] The second labeling module 30A is used to re-label the second pseudo-label sample set based on the second labeling model jointly constructed by the real label sample set and the first pseudo-label sample set to obtain the third pseudo-label sample set.

[0141] The label difference calculation module 40A is used to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set.

[0142] The pseudo-label sample screening module 50A is used to screen the target pseudo-label sample set by removing samples whose label difference value is greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0143] Optionally, the pseudo-label sample screening module 50A is further used for:

[0144] In the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed to obtain each output pseudo-label sample;

[0145] Each verification sample is obtained, and user qualification classification is performed on each verification sample according to the verification label pseudo-label corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-label is obtained by labeling the verification sample by the first label labeling model or the second label labeling model.

[0146] Obtain the real business indicator values ​​corresponding to each of the verification samples, and based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

[0147] Optionally, the pseudo-label sample screening module 50A is further used for:

[0148] Based on the actual business indicator values, verify whether the user qualification classification results meet the preset order preservation requirements;

[0149] If the conditions are met, then all the output pseudo-label samples will be used together as the target pseudo-label sample set;

[0150] If the conditions are not met, return to the following steps: obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo-label sample set.

[0151] Optionally, the user qualification classification results include a first user qualification classification result and a second user qualification classification result, and the pseudo-label sample screening module 50A is further used for:

[0152] The first label annotation model obtains the verification annotation pseudo-labels of the verification sample annotation as initial pseudo-labels, and the second label annotation model obtains the verification annotation pseudo-labels of the verification sample annotation as correction pseudo-labels;

[0153] Based on the first tag score corresponding to each initial pseudo tag, the user qualification of each verification sample is classified to obtain the first user qualification classification result.

[0154] Based on the second tag score corresponding to each of the modified pseudo-tags, the user qualification of each of the verification samples is classified to obtain the second user qualification classification result.

[0155] Optionally, the first user qualification classification result includes at least one first user qualification classification level, and the second user qualification classification result includes at least one second user qualification classification level. The pseudo-label sample screening module 50A is further used for:

[0156] Calculate the statistical value of the first-tier business indicator corresponding to the actual business indicator value under each of the first user qualification tiers, and the statistical value of the second-tier business indicator corresponding to the actual business indicator value under each of the second user qualification tiers.

[0157] Based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators, verify whether the user qualification classification results meet the preset order preservation requirements.

[0158] Optionally, the pseudo-label sample screening module 50A is further used for:

[0159] If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators do not increase with the increase of the user qualification tier, then it is determined that the user qualification tiering result does not meet the preset order preservation requirement.

[0160] If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase with the increase of the user qualification level, then it is detected whether the significance of the difference between the statistical values ​​of the first-tier business indicators is higher than the significance of the difference between the statistical values ​​of the second-tier business indicators.

[0161] If so, the user qualification classification result is determined to meet the preset order preservation requirement;

[0162] If not, the user qualification classification result is determined to not meet the preset order preservation requirements.

[0163] Optionally, the pseudo-label sample screening module 50A is further used for:

[0164] Check whether the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase as the user's qualification level increases;

[0165] If so, the user qualification classification result is determined to meet the preset order preservation requirement;

[0166] If not, the user qualification classification result is determined to not meet the preset order preservation requirement.

[0167] Optionally, the second labeling module 30A is further used for:

[0168] The real label sample set and the first pseudo label sample set are mixed into a full sample set;

[0169] The second labeling model with local preservation properties is iteratively trained using the full sample set.

[0170] Based on the second labeling model, the second pseudo-label sample set is re-labeled.

[0171] Optionally, the pseudo-label annotation device is applied to a first model builder that performs joint modeling using the target pseudo-label sample set, and the pseudo-label annotation device is further used for:

[0172] By aligning samples with the second model builder, each aligned sample is selected from the target pseudo-label sample set;

[0173] The sample identifiers and pseudo-labels corresponding to each aligned sample are sent to the second model builder so that the second model builder can build the corresponding user qualification assessment model.

[0174] The pseudo-label marking device provided in this application, employing the pseudo-label marking method in the above embodiments, solves the technical problem of low accuracy in pseudo-label marking. Compared with the prior art, the beneficial effects of the pseudo-label marking device provided in this application are the same as those of the pseudo-label marking method provided in the above embodiments, and other technical features in this pseudo-label marking device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0175] Example 6

[0176] Reference Figure 5 This application also provides a joint modeling optimization device, applied to a first model builder, the joint modeling optimization device comprising:

[0177] The first labeling module 10B is used to obtain a real label sample set, and to label the unlabeled sample set according to the first labeling model constructed based on the real label sample set, so as to obtain an initial pseudo label sample set.

[0178] The segmentation module 20B is used to segment the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set.

[0179] The second labeling module 30B is used to re-label the second pseudo-label sample set based on the second labeling model jointly constructed by the real label sample set and the first pseudo-label sample set to obtain the third pseudo-label sample set.

[0180] The label difference calculation module 40B is used to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set.

[0181] The pseudo-label sample screening module 50B is used to screen the target pseudo-label sample set by removing samples whose label difference value is greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set.

[0182] The sample alignment module 60B is used to filter each aligned sample in the target pseudo-label sample set by aligning the samples with the second model builder.

[0183] The joint modeling module 70B is used to send the sample identifiers and pseudo-labels corresponding to each of the aligned samples to the second model builder, so that the second model builder can build the corresponding user qualification assessment model.

[0184] Optionally, the pseudo-label sample screening module 50B is further used for:

[0185] In the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed to obtain each output pseudo-label sample;

[0186] Each verification sample is obtained, and user qualification classification is performed on each verification sample according to the verification label pseudo-label corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-label is obtained by labeling the verification sample by the first label labeling model or the second label labeling model.

[0187] Obtain the real business indicator values ​​corresponding to each of the verification samples, and based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

[0188] Optionally, the pseudo-label sample screening module 50B is further used for:

[0189] Based on the actual business indicator values, verify whether the user qualification classification results meet the preset order preservation requirements;

[0190] If the conditions are met, then all the output pseudo-label samples will be used together as the target pseudo-label sample set;

[0191] If the conditions are not met, return to the following steps: obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo-label sample set.

[0192] Optionally, the user qualification classification results include a first user qualification classification result and a second user qualification classification result, and the pseudo-label sample screening module 50B is further used for:

[0193] The first label annotation model obtains the verification annotation pseudo-labels of the verification sample annotation as initial pseudo-labels, and the second label annotation model obtains the verification annotation pseudo-labels of the verification sample annotation as correction pseudo-labels;

[0194] Based on the first tag score corresponding to each initial pseudo tag, the user qualification of each verification sample is classified to obtain the first user qualification classification result.

[0195] Based on the second tag score corresponding to each of the modified pseudo-tags, the user qualification of each of the verification samples is classified to obtain the second user qualification classification result.

[0196] Optionally, the first user qualification classification result includes at least one first user qualification classification level, and the second user qualification classification result includes at least one second user qualification classification level. The pseudo-label sample screening module 50B is further used for:

[0197] Calculate the statistical value of the first-tier business indicator corresponding to the actual business indicator value under each of the first user qualification tiers, and the statistical value of the second-tier business indicator corresponding to the actual business indicator value under each of the second user qualification tiers.

[0198] Based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators, verify whether the user qualification classification results meet the preset order preservation requirements.

[0199] Optionally, the pseudo-label sample screening module 50B is further used for:

[0200] If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators do not increase with the increase of the user qualification tier, then it is determined that the user qualification tiering result does not meet the preset order preservation requirement.

[0201] If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase with the increase of the user qualification level, then it is detected whether the significance of the difference between the statistical values ​​of the first-tier business indicators is higher than the significance of the difference between the statistical values ​​of the second-tier business indicators.

[0202] If so, the user qualification classification result is determined to meet the preset order preservation requirement;

[0203] If not, the user qualification classification result is determined to not meet the preset order preservation requirements.

[0204] Optionally, the pseudo-label sample screening module 50B is further used for:

[0205] Check whether the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase as the user's qualification level increases;

[0206] If so, the user qualification classification result is determined to meet the preset order preservation requirement;

[0207] If not, the user qualification classification result is determined to not meet the preset order preservation requirement.

[0208] Optionally, the second labeling module 30B is also used for:

[0209] The real label sample set and the first pseudo label sample set are mixed into a full sample set;

[0210] The second labeling model with local preservation properties is iteratively trained using the full sample set.

[0211] Based on the second labeling model, the second pseudo-label sample set is re-labeled.

[0212] Optionally, the joint modeling optimization device is applied to a first model builder that performs joint modeling using the target pseudo-label sample set, and the joint modeling optimization device is further used for:

[0213] By aligning samples with the second model builder, each aligned sample is selected from the target pseudo-label sample set;

[0214] The sample identifiers and pseudo-labels corresponding to each aligned sample are sent to the second model builder so that the second model builder can build the corresponding user qualification assessment model.

[0215] The joint modeling optimization apparatus provided in this application, employing the joint modeling optimization method in the above embodiments, solves the technical problem of low iterative training efficiency of the user qualification assessment model due to excessively high communication data volume between the message recommendation side and the media side. Compared with the prior art, the beneficial effects of the joint modeling optimization apparatus provided in this application are the same as those of the joint modeling optimization method provided in the above embodiments, and other technical features in this joint modeling optimization apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0216] Example 7

[0217] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the pseudo-labeling method or the joint modeling optimization method in the above embodiments.

[0218] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0219] like Figure 6 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0220] Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0221] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined above in the methods of embodiments of this disclosure.

[0222] The electronic device provided in this application employs the pseudo-labeling method or joint modeling optimization method described in the above embodiments, solving the technical problem of low accuracy in pseudo-labeling or the technical problem of low iterative training efficiency of the user qualification assessment model due to excessive communication data volume between the message recommendation side and the media side. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the pseudo-labeling method or joint modeling optimization method provided in the above embodiments, and other technical features in this electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0223] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0224] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0225] Example 8

[0226] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the pseudo-labeling method or the joint modeling optimization method in the above embodiment.

[0227] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0228] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0229] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following actions: to acquire a real label sample set; to label an unlabeled sample set according to a first labeling model constructed from the real label sample set, thereby obtaining an initial pseudo-label sample set; to divide the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set; to re-label the second pseudo-label sample set according to a second labeling model jointly constructed from the real label sample set and the first pseudo-label sample set, thereby obtaining a third pseudo-label sample set; to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set; and to filter the target pseudo-label sample set by removing samples from the second pseudo-label sample set and the third pseudo-label sample set whose label difference value is greater than a preset label difference threshold.

[0230] Alternatively, obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo-label sample set; divide the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set; based on the second label annotation model jointly constructed from the real label sample set and the first pseudo-label sample set, re-label the second pseudo-label sample set to obtain a third pseudo-label sample set; calculate the label difference value between the pseudo-labels annotated in the second pseudo-label sample set and the third pseudo-label sample set; filter the target pseudo-label sample set by removing samples from the second pseudo-label sample set and the third pseudo-label sample set whose label difference value is greater than a preset label difference threshold; align the samples with the second model builder, and filter each aligned sample in the target pseudo-label sample set; send the sample identifier and labeled pseudo-labels corresponding to each aligned sample to the second model builder so that the second model builder can construct the corresponding user qualification assessment model.

[0231] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0232] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0233] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0234] The computer-readable storage medium provided in this application stores computer-readable program instructions for executing the aforementioned pseudo-labeling method or joint modeling optimization method, thus solving the technical problem of low accuracy in pseudo-labeling or the technical problem of low iterative training efficiency of the user qualification assessment model due to excessive communication data volume between the message recommendation side and the media side. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the pseudo-labeling method or joint modeling optimization method provided in the above embodiments, and will not be repeated here.

[0235] Example 9

[0236] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the pseudo-labeling method or the joint modeling optimization method described above.

[0237] The computer program product provided in this application solves the technical problem of low accuracy in pseudo-labeling or the technical problem of low iterative training efficiency of user qualification assessment models due to excessive communication data between the message recommendation side and the media side. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the pseudo-labeling method or joint modeling optimization method provided in the above embodiments, and will not be repeated here.

[0238] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A pseudo-labeling method, characterized in that, The pseudo-labeling method is applied in the financial field, including: Obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo label sample set; The initial pseudo-label sample set is divided into a first pseudo-label sample set and a second pseudo-label sample set; Based on the second labeling model jointly constructed from the real label sample set and the first pseudo label sample set, the second pseudo label sample set is re-labeled to obtain the third pseudo label sample set; Calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set; The target pseudo-label sample set is filtered by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set. The step of filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set includes: In the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed to obtain each output pseudo-label sample; Each verification sample is obtained, and user qualification classification is performed on each verification sample according to the verification label pseudo-label corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-label is obtained by labeling the verification sample by the first label labeling model or the second label labeling model. Obtain the real business indicator values ​​corresponding to each of the verification samples, and based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

2. The pseudo-labeling method as described in claim 1, characterized in that, The step of filtering the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements based on each of the real business indicator values ​​includes: Based on the actual business indicator values, verify whether the user qualification classification results meet the preset order preservation requirements; If the conditions are met, then all the output pseudo-label samples will be used together as the target pseudo-label sample set. If the conditions are not met, return to the following steps: obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo-label sample set.

3. The pseudo-labeling method as described in claim 2, characterized in that, The user qualification classification results include the first user qualification classification result and the second user qualification classification result. The step of classifying each verification sample into user qualification categories based on the verification annotation pseudo-labels corresponding to each verification sample, and obtaining the user qualification classification results, includes: The pseudo-labels of the verification annotations of the first label annotation model for the verification samples are obtained as initial pseudo-labels, and the pseudo-labels of the verification annotations of the second label annotation model for the verification samples are obtained as correction pseudo-labels; Based on the first tag score corresponding to each initial pseudo tag, the user qualification of each verification sample is classified to obtain the first user qualification classification result. Based on the second tag score corresponding to each of the modified pseudo-tags, the user qualification of each of the verification samples is classified to obtain the second user qualification classification result.

4. The pseudo-labeling method as described in claim 3, characterized in that, The first user qualification classification result includes at least one first user qualification classification level, and the second user qualification classification result includes at least one second user qualification classification level. The step of verifying whether the user qualification classification result meets the preset order preservation requirements based on each of the real business indicator values ​​includes: Calculate the statistical value of the first-tier business indicator corresponding to the actual business indicator value under each of the first user qualification tiers, and the statistical value of the second-tier business indicator corresponding to the actual business indicator value under each of the second user qualification tiers. Based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators, verify whether the user qualification classification results meet the preset order preservation requirements.

5. The pseudo-labeling method as described in claim 4, characterized in that, The step of verifying whether the user qualification classification result meets the preset order preservation requirements based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators includes: If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators do not increase with the increase of the user qualification tier, then it is determined that the user qualification tiering result does not meet the preset order preservation requirement. If the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase with the increase of the user qualification level, then it is detected whether the significance of the difference between the statistical values ​​of the first-tier business indicators is higher than the significance of the difference between the statistical values ​​of the second-tier business indicators. If so, the user qualification classification result is determined to meet the preset order preservation requirement; If not, the user qualification classification result is determined to not meet the preset order preservation requirements.

6. The pseudo-labeling method as described in claim 4, characterized in that, The step of verifying whether the user qualification classification result meets the preset order preservation requirements based on the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators includes: Check whether the statistical values ​​of the first-tier business indicators and the statistical values ​​of the second-tier business indicators both increase as the user's qualification level increases; If so, the user qualification classification result is determined to meet the preset order preservation requirement; If not, the user qualification classification result is determined to not meet the preset order preservation requirements.

7. The pseudo-labeling method as described in claim 1, characterized in that, The step of re-labeling the second pseudo-label sample set using the second labeling model constructed jointly based on the real label sample set and the first pseudo-label sample set includes: The real label sample set and the first pseudo label sample set are mixed into a full sample set; The second labeling model with local preservation properties is iteratively trained using the full sample set. Based on the second labeling model, the second pseudo-label sample set is re-labeled.

8. The pseudo-labeling method as described in claim 1, characterized in that, The pseudo-labeling method, applied to a first model builder performing joint modeling using the target pseudo-label sample set, further includes the following steps after filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second and third pseudo-label sample sets: By aligning samples with the second model builder, each aligned sample is selected from the target pseudo-label sample set; The sample identifiers and pseudo-labels corresponding to each aligned sample are sent to the second model builder so that the second model builder can build the corresponding user qualification assessment model.

9. A joint modeling optimization method, characterized in that, Applied to the first model builder, the joint modeling optimization method includes: Obtain a real label sample set, and based on the first label annotation model constructed from the real label sample set, label the unlabeled sample set to obtain an initial pseudo label sample set; The initial pseudo-label sample set is divided into a first pseudo-label sample set and a second pseudo-label sample set; Based on the second labeling model jointly constructed from the real label sample set and the first pseudo label sample set, the second pseudo label sample set is re-labeled to obtain the third pseudo label sample set; Calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set; The target pseudo-label sample set is filtered by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set. By aligning samples with the second model builder, each aligned sample is selected from the target pseudo-label sample set; The sample identifier and pseudo-label corresponding to each of the alignment samples are sent to the second model builder so that the second model builder can build the corresponding user qualification assessment model; The step of filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set includes: In the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed to obtain each output pseudo-label sample; Each verification sample is obtained, and user qualification classification is performed on each verification sample according to the verification label pseudo-label corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-label is obtained by labeling the verification sample by the first label labeling model or the second label labeling model. Obtain the real business indicator values ​​corresponding to each of the verification samples, and based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

10. A pseudo-label marking device, characterized in that, The pseudo-label marking device includes: The first labeling module is used to obtain a real label sample set, and to label the unlabeled sample set according to the first labeling model constructed based on the real label sample set, so as to obtain an initial pseudo label sample set. The segmentation module is used to segment the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set. The second labeling module is used to re-label the second pseudo-label sample set based on the second labeling model jointly constructed by the real label sample set and the first pseudo-label sample set, to obtain the third pseudo-label sample set. The label difference calculation module is used to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set. The pseudo-label sample filtering module is used to filter the target pseudo-label sample set by removing samples whose label difference value is greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set. The step of filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set includes: In the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed to obtain each output pseudo-label sample; Each verification sample is obtained, and user qualification classification is performed on each verification sample according to the verification label pseudo-label corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-label is obtained by labeling the verification sample by the first label labeling model or the second label labeling model. Obtain the real business indicator values ​​corresponding to each of the verification samples, and based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

11. A joint modeling and optimization device, characterized in that, Applied to the first model builder, the joint modeling optimization device includes: The first labeling module is used to obtain a real label sample set, and to label the unlabeled sample set according to the first labeling model constructed based on the real label sample set, so as to obtain an initial pseudo label sample set. The segmentation module is used to segment the initial pseudo-label sample set into a first pseudo-label sample set and a second pseudo-label sample set. The second labeling module is used to re-label the second pseudo-label sample set based on the second labeling model jointly constructed by the real label sample set and the first pseudo-label sample set, to obtain the third pseudo-label sample set. The label difference calculation module is used to calculate the label difference value between the pseudo-labels labeled in the second pseudo-label sample set and the third pseudo-label sample set. The pseudo-label sample filtering module is used to filter the target pseudo-label sample set by removing samples whose label difference value is greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set. The sample alignment module is used to filter aligned samples in the target pseudo-label sample set by aligning samples with the second model builder. The joint modeling module is used to send the sample identifiers and pseudo-labels corresponding to each of the aligned samples to the second model builder, so that the second model builder can build the corresponding user qualification assessment model. The step of filtering the target pseudo-label sample set by removing samples with label difference values ​​greater than a preset label difference threshold from the second pseudo-label sample set and the third pseudo-label sample set includes: In the second pseudo-label sample set and the third pseudo-label sample set, samples with label difference values ​​greater than a preset label difference threshold are removed to obtain each output pseudo-label sample; Each verification sample is obtained, and user qualification classification is performed on each verification sample according to the verification label pseudo-label corresponding to each verification sample to obtain user qualification classification results. The verification label pseudo-label is obtained by labeling the verification sample by the first label labeling model or the second label labeling model. Obtain the real business indicator values ​​corresponding to each of the verification samples, and based on each of the real business indicator values, filter the target pseudo-label sample set by verifying whether the user qualification classification results meet the preset order preservation requirements.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the pseudo-labeling method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for implementing the pseudo-labeling method, which is executed by a processor to implement the steps of the pseudo-labeling method as described in any one of claims 1 to 9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the pseudo-labeling method as described in any one of claims 1 to 9.

Images