A user label missing data filling method and system

Abstract

The application provides a user label missing data filling method and system, the method comprises the following steps: collecting user historical labels; preprocessing the user historical labels to obtain a plurality of features; calculating the similarity of different features to obtain a feature weight file; obtaining a label query request; obtaining a plurality of to-be-queried user labels according to the label query request; sorting the to-be-queried user labels; and filling the sorted to-be-queried user labels by using the feature weight file. The method combines offline calculation and real-time filling technology, improves the saturation of the user label by filling the missing values in the user label, and improves the sparsity of the user label.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, and particularly relates to a user label missing data filling method and system. BACKGROUND

[0002] At present, user labels are widely used in portrait population analysis, modeling and other methods. User labels are standardized definitions of user attributes, behaviors, preferences and the like, and are used to describe user characteristics. However, the current user label saturation is low and cannot meet the requirements of portrait population analysis, modeling and other methods. SUMMARY

[0003] In view of the defects in the prior art, the present application provides a user label missing data filling method and system to improve the saturation of user labels.

[0004] In a first aspect, a user label missing data filling method comprises:

[0005] Collecting user historical labels;

[0006] Preprocessing the user historical labels to obtain a plurality of features;

[0007] Calculating the similarity of different features to obtain a feature weight file;

[0008] Obtaining a label query request;

[0009] Obtaining a plurality of to-be-queried user labels according to the label query request;

[0010] Sorting the to-be-queried user labels;

[0011] Filling the sorted to-be-queried user labels using the feature weight file.

[0012] Further, preprocessing the user historical labels to obtain a plurality of features specifically comprises:

[0013] Determining a sample size;

[0014] Sampling and filtering the user historical labels according to the sample size to obtain features;

[0015] Constructing a sample set according to all the features.

[0016] Further, sampling and filtering the user historical labels to obtain features specifically comprises:

[0017] Layering the user historical labels, and extracting the user historical labels layer by layer to obtain column sampling results;

[0018] Match each user ID with the column sampling result respectively to obtain the number of user historical labels matched by each user ID;

[0019] Arrange the user IDs in descending order according to the number of user historical labels;

[0020] Extract the user historical labels of each user ID in the column sampling result in turn according to the arrangement order to obtain the row sampling result;

[0021] Filter the row sampling result according to the filtering rule to obtain the features.

[0022] Further, the similarity of different features is calculated to obtain a feature weight file, which specifically includes:

[0023] The cosine similarity between any two features is calculated respectively;

[0024] The feature similarity matrix is constructed according to all cosine similarities;

[0025] The feature similarity matrix is normalized to obtain the feature weight file.

[0026] Further, the sorting of the user label to be queried specifically includes:

[0027] Obtain the similarity of the user label to be queried in the feature weight file;

[0028] Arrange the user label to be queried in descending order according to the similarity.

[0029] Further, the user label to be queried after sorting is filled in by using the feature weight file, which specifically includes:

[0030] Extract the label value of the user label to be queried respectively, and construct a label value matrix according to all label values;

[0031] Multiply the similarity of the user label to be queried with the label value matrix to obtain a label filling value;

[0032] Fill in the user label to be queried by using the label filling value.

[0033] The second aspect is a user label missing data filling system, which includes:

[0034] A pre-training module is used to collect user historical labels, pre-process the user historical labels to obtain a plurality of features, and calculate the similarity of different features to obtain a feature weight file;

[0035] A real-time filling module is used to obtain a label query request, obtain a plurality of user labels to be queried according to the label query request, sort the user labels to be queried, and fill in the user labels to be queried after sorting by using the feature weight file.

[0036] Further, the pre-training module is specifically configured to:

[0037] determine a sample size;

[0038] layer the user historical labels, and extract the user historical labels layer by layer to obtain column sampling results;

[0039] match each user ID with the column sampling results respectively to obtain the number of user historical labels matched by each user ID;

[0040] arrange the user IDs in descending order according to the number of user historical labels;

[0041] extract the user historical labels of each user ID in the column sampling results in turn according to the arrangement order to obtain row sampling results;

[0042] filter the row sampling results according to a filtering rule to obtain features; the number of features is equal to the sample size;

[0043] construct a sample set according to all the features.

[0044] Further, the pre-training module is specifically configured to:

[0045] calculate the cosine similarity between any two features respectively;

[0046] construct a feature similarity matrix according to all the cosine similarities;

[0047] normalize the feature similarity matrix to obtain a feature weight file.

[0048] Further, the real-time filling module is specifically configured to:

[0049] obtain the similarity of a to-be-queried user label in the feature weight file;

[0050] arrange the to-be-queried user labels in descending order according to the similarity;

[0051] extract the label values of the to-be-queried user labels respectively, and construct a label value matrix according to all the label values;

[0052] multiply the similarity of the to-be-queried user label with the label value matrix to obtain a label filling value;

[0053] fill the to-be-queried user label by using the label filling value.

[0054] As can be seen from the above technical solutions, the user label missing data filling method and system provided by the application combine offline calculation and real-time filling technology, improve the saturation of user labels by filling the missing values in the user labels, and improve the sparsity of user labels. Attached Figure Description

[0055] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0056] Figure 1 A flowchart of a method for filling in missing user tag data provided in an embodiment.

[0057] Figure 2 A flowchart of a user history tag preprocessing method provided for an embodiment.

[0058] Figure 3 A flowchart of a user history tag sampling and filtering method provided for an embodiment.

[0059] Figure 4 A flowchart of a feature weight file generation method provided for an embodiment.

[0060] Figure 5 This is a flowchart of a method for sorting user tags to be queried, provided as an example.

[0061] Figure 6 A flowchart of a method for filling in user tags to be queried, provided for an embodiment.

[0062] Figure 7 A block diagram of a user tag missing data filling system provided in this embodiment. Detailed Implementation

[0063] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore merely examples and should not be used to limit the scope of protection of the present invention. It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0064] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0065] It should also be understood that the terms used herein are for the purpose of describing particular embodiments and are not intended to limit the application. As used in the specification and the appended claims, the singular forms "a," "an" and "the" are intended to include plural forms unless the context clearly dictates otherwise.

[0066] As used in the specification and the appended claims, the term "if' can be construed to mean "when" or "once," or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be construed to mean "once it is determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]," depending on the context.

[0067] Embodiments:

[0068] A user tag missing data filling method, see Figure 1 , comprising:

[0069] S1: collecting user historical tags;

[0070] S2: preprocessing the user historical tags to obtain a plurality of features;

[0071] S3: calculating the similarity of different features to obtain a feature weight file;

[0072] S4: obtaining a label query request;

[0073] S5: obtaining a plurality of to-be-queried user tags according to the label query request;

[0074] S6: sorting the to-be-queried user tags;

[0075] S7: filling the sorted to-be-queried user tags using the feature weight file.

[0076] In the embodiment, the method first acquires user historical labels of all user IDs to obtain full data, and the full data includes more than 1000-dimensional features corresponding to 20 billion user IDs. Then the method pre-processes the user historical labels to obtain a plurality of features, wherein the features are extracted user historical labels. The pre-processing step is mainly data extraction. Since the full data is large in quantity, the calculation of the full data will consume a lot of resources and time. Therefore, the method extracts features from the full data before calculation, thereby reducing the calculation amount and saving resource consumption and time consumption. Then the method calculates the similarity of different features to obtain a feature weight file. The feature weight file can be calculated in an offline state, and the calculation process can be implemented by using a time window technology, for example, a time window is established to read a fixed number of features for calculation, and when a round of calculation is completed, the time window is moved to read the next round of features for calculation. The method can store the feature weight file in redis for subsequent services that need to fill in labels to call.

[0077] In the embodiment, the method receives a label query request in real time, and acquires a plurality of to-be-queried user labels according to the label query request, wherein the label query request can include a user ID, and the method queries the corresponding user label according to the user ID. Then the method sorts the to-be-queried user labels, fills in the missing values in the to-be-queried user labels according to the feature weight file, for example, the method calls the feature weight file, and fills in the missing values in the to-be-queried user labels according to the filling logic.

[0078] The user label missing data filling method combines offline calculation and real-time filling technology, and improves the saturation of the user label and the sparsity of the user label by filling in the missing values in the user label.

[0079] Further, in some embodiments, referring to Figure 2 The pre-processing of the user historical labels to obtain a plurality of features specifically includes:

[0080] S11: determining a sample size;

[0081] S12: sampling and filtering the user historical labels according to the sample size to obtain features;

[0082] S13: constructing a sample set according to all the features.

[0083] In this embodiment, when sampling the user historical labels, in order to balance the size of the storage space, the minimum sample size can be calculated according to the full data, for example, it is calculated that the minimum sample size is 5 million, at this time, the sample size is defined as 5 million, so that the number of user historical labels in the final obtained sample set is 5 million. The method can first randomly extract 2 times the sample size of user historical labels, and then filter the user historical labels according to the filtering rule to obtain a sample set consistent with the sample size. For example, first extract 10 million user historical labels, and then filter out 5 million user historical labels according to the filtering rule, and construct a sample set according to the remaining 5 million user historical labels. The filtering rule can be pre-set or modified during use.

[0084] Further, in some embodiments, referring to Figure 3 sampling and filtering the user historical labels to obtain the feature specifically includes:

[0085] S21: stratifying the user historical labels, and extracting the user historical labels layer by layer to obtain column sampling results;

[0086] S22: respectively matching each user ID with the column sampling results to obtain the number of user historical labels matched by each user ID;

[0087] S23: arranging the user IDs in descending order according to the number of user historical labels;

[0088] S24: extracting the user historical labels of each user ID in the column sampling results in turn according to the arrangement order to obtain row sampling results;

[0089] S25: filtering the row sampling results according to the filtering rule to obtain the feature.

[0090] In the embodiment, the sampling of the user historical labels in the method mainly includes column sampling and row sampling. Since the coverage rates of each user historical label are quite different (from 90% to 1%), if only random sampling is performed on the user historical labels, it will be difficult to sample the user historical labels with low coverage rates, resulting in that many user historical labels cannot be subjected to subsequent filling processes. Therefore, when sampling, the user historical labels can be stratified according to the coverage rates, and then the user historical labels are extracted layer by layer to obtain column sampling results, so that user historical labels with any coverage rate can be sampled.

[0091] In the embodiment, the method needs to perform row sampling after column sampling is completed. In order to ensure that the user history labels of different coverage rates can be extracted in limited samples, sampling can be performed according to the user ID, so that the user history label of each user ID can be extracted, and each user ID can have rich features for learning. The row extraction process can be arranged in descending order according to the number of user history labels matched by the user ID, and the user ID is extracted.

[0092] In the embodiment, when the user history label is output, some sensitive labels may be shielded, which may result in that some user history labels with high coverage rates cannot be output. If such user history labels are directly deleted, the user corresponding to the user history labels cannot be filled with labels. Therefore, when the similarity of the features is calculated, the sensitive labels can be included in the similarity calculation.

[0093] Further, in some embodiments, referring to Figure 4 , the similarity of different features is calculated to obtain a feature weight file, and specifically includes:

[0094] S31: the cosine similarity between any two features is calculated respectively;

[0095] S32: a feature similarity matrix is constructed according to all the cosine similarities;

[0096] S33: the feature similarity matrix is normalized to obtain a feature weight file.

[0097] In the embodiment, because the number of user IDs is huge, the features are stable, and the feature magnitude is small, the similarity of the features is calculated by the method. When the similarity is calculated, the cosine similarity can be selected. After the cosine similarity between any two features is obtained, a feature similarity matrix is constructed according to all the cosine similarities, and the feature similarity matrix is normalized to output a feature weight file. The feature weight file can be output in a row-column format, and is a row-column symmetric csv format document. The method can also output after excluding the similarity of sensitive labels in the feature weight file. The method can also set the label that does not need to be filled to 0, limit the number of decimal points, and avoid that the extremely small value in the feature weight file appears in scientific notation.

[0098] Further, in some embodiments, referring to Figure 5 , the query user label is sorted, and specifically includes:

[0099] S41: the similarity of the query user label in the feature weight file is obtained;

[0100] S42: the query user label is arranged in descending order according to the similarity.

[0101] In the embodiment, the method first reads the similarity of the to-be-queried user label from the feature weight file, and arranges the similarity in descending order. The method can return the first N to-be-queried user labels in the arrangement order.

[0102] Further, in some embodiments, referring to Figure 6 , the filling of the to-be-queried user label after the sorting processing by using the feature weight file specifically includes:

[0103] S51: Extract the label value of the to-be-queried user label respectively, and construct a label value matrix according to all the label values;

[0104] S52: Multiply the similarity of the to-be-queried user label with the label value matrix to obtain a label filling value;

[0105] S53: Fill the to-be-queried user label by using the label filling value.

[0106] In the embodiment, the method first obtains the label value matrix of the to-be-queried user label, then multiplies the similarity with the label value matrix to obtain a label filling value, and fills the to-be-queried user label by using the label filling value.

[0107] A user label missing data filling system, referring to Figure 7 , includes:

[0108] A pre-training module 1: used for collecting user historical labels; pre-processing the user historical labels to obtain a plurality of features; calculating the similarity of different features to obtain a feature weight file;

[0109] A real-time filling module 2: used for obtaining a label query request; obtaining a plurality of to-be-queried user labels according to the label query request; sorting the to-be-queried user labels; filling the sorted to-be-queried user labels by using the feature weight file.

[0110] Further, the pre-training module 1 is specifically used for:

[0111] Determining the sample size;

[0112] Layering the user historical labels, and extracting the user historical labels layer by layer to obtain a column sampling result;

[0113] Matching each user ID with the column sampling result respectively to obtain the number of user historical labels matched by each user ID;

[0114] Arranging the user IDs in descending order according to the number of user historical labels;

[0115] Extracting the user historical labels of each user ID in the column sampling result in turn according to the arrangement order to obtain a row sampling result;

[0116] Filtering the row sampling result according to the filtering rule to obtain features; the number of features is equal to the sample size;

[0117] According to all the features, a sample set is constructed.

[0118] Further, the pre-training module 1 is specifically used for:

[0119] The cosine similarity between any two features is calculated respectively;

[0120] According to all the cosine similarities, a feature similarity matrix is constructed.

[0121] The feature similarity matrix is normalized to obtain a feature weight file.

[0122] Further, the real-time filling module 2 is specifically used for:

[0123] Obtaining the similarity of the to-be-queried user label in the feature weight file;

[0124] The to-be-queried user label is arranged in descending order according to the similarity;

[0125] The label value of the to-be-queried user label is extracted respectively, and a label value matrix is constructed according to all the label values;

[0126] The similarity of the to-be-queried user label is multiplied by the label value matrix to obtain a label filling value;

[0127] The to-be-queried user label is filled by using the label filling value.

[0128] The system provided by the embodiments of the present application is briefly described, and the part not mentioned in the embodiments can refer to the corresponding content in the foregoing embodiments.

[0129] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the description of the present application.

Claims

1. A method for imputing missing user tag data, characterized in that, include: Collect user historical tags; The user's historical tags are preprocessed to obtain multiple features; Calculate the similarity of different features to obtain the feature weight file; Get the tag query request; Multiple user tags to be queried are obtained according to the tag query request. The user tags are used to define the characteristics of user attributes, user behavior and user preferences. Sort the user tags to be queried; The sorted user tags are filled in using the aforementioned feature weight file; The preprocessing of the user's historical tags to obtain multiple features specifically includes: Determine the sample size; The user's historical tags are sampled and filtered based on the sample size to obtain the features; Construct a sample set based on all the described features; The sampling and filtering of the user's historical tags to obtain the features specifically includes: The user history tags are layered, and the user history tags are extracted layer by layer to obtain column sampling results; Each user ID is matched with the column sampling results to obtain the number of historical tags that match each user ID; Sort user IDs in descending order based on the number of historical tags associated with each user. The user history tags for each user ID are extracted sequentially from the column sampling results according to the sorting order to obtain the row sampling results; The row sampling results are filtered according to the filtering rules to obtain the features; The sorting of the user tags to be queried specifically includes: Obtain the similarity of the user tags to be queried in the feature weight file; Sort the user tags to be queried in descending order of similarity; The step of filling in the sorted user tags using the feature weight file specifically includes: Extract the tag values ​​of the tags of the user to be queried, and construct a tag value matrix based on all tag values; The similarity of the user's tags to be queried is multiplied by the tag value matrix to obtain the tag imputation value; The tag filling values ​​are used to fill in the tags of the user to be queried.

2. The user tag missing data imputation method according to claim 1, characterized in that, The calculation of the similarity of different features to obtain the feature weight file specifically includes: Calculate the cosine similarity between any two of the features; Construct a feature similarity matrix based on all cosine similarities; The feature similarity matrix is ​​normalized to obtain the feature weight file.

3. A system for filling in missing user tag data, characterized in that, include: Pre-training module: used to collect users' historical tags; The user's historical tags are preprocessed to obtain multiple features; Calculate the similarity of different features to obtain the feature weight file; Real-time filling module: used to obtain tag query requests; Multiple user tags to be queried are obtained according to the tag query request. The user tags are used to define the features of user attributes, user behavior and user preferences. The user tags to be queried are sorted. The sorted user tags to be queried are filled in using the feature weight file. The pre-training module is specifically used for: Determine the sample size; The user history tags are layered, and the user history tags are extracted layer by layer to obtain column sampling results; Each user ID is matched with the column sampling results to obtain the number of historical tags that match each user ID; Sort user IDs in descending order based on the number of historical tags associated with each user. The user history tags for each user ID are extracted sequentially from the column sampling results according to the sorting order to obtain the row sampling results; The row sampling results are filtered according to the filtering rules to obtain the features; the number of features is equal to the sample size. Construct a sample set based on all the described features; The real-time filling module is specifically used for: Obtain the similarity of the user tags to be queried in the feature weight file; Sort the user tags to be queried in descending order of similarity; Extract the tag values ​​of the tags of the user to be queried, and construct a tag value matrix based on all tag values; The similarity of the user's tags to be queried is multiplied by the tag value matrix to obtain the tag imputation value; The tag filling values ​​are used to fill in the tags of the user to be queried.

4. The user tag missing data imputation system according to claim 3, characterized in that, The pre-training module is specifically used for: Calculate the cosine similarity between any two of the features; Construct a feature similarity matrix based on all cosine similarities; The feature similarity matrix is ​​normalized to obtain the feature weight file.

Images