Data screening method and device

Abstract

The present invention discloses a data screening method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a table to be filtered, determining the attributes to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label; for each item, using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model, determine the first probability of the item under each first label; calculate the count matrix according to the first label and the first probability; determine the wrong data in the table to be filtered according to the first probability and the count matrix. This implementation can automatically find out the wrong data in the table, thereby improving the efficiency and accuracy of data screening.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data screening method, device, electronic device and computer-readable medium. Background Art

[0002] A table consists of one or more rows of cells, which are used to display numbers and other items for quick reference and analysis. Items in a table are organized into rows and columns. The header generally refers to the first row of the table, indicating the content and meaning of each column of the table. Nowadays, tables are needed to perform statistical processing on data in various business management tasks. With the development of various businesses, the amount of data will also increase, and the information recorded in the table is manually marked and modified, which inevitably leads to omissions. In the process of realizing the present invention, the inventors found that in the prior art, these erroneous data can only be screened manually. Manual screening is time-consuming and labor-intensive, and for large-scale tables, the efficiency of manual screening is very low. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data screening method, device, electronic device and computer-readable medium, which can automatically find erroneous data in a table and improve the efficiency and accuracy of data screening.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a data screening method is provided, comprising:

[0005] Acquire a table to be filtered, and determine an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as a first label;

[0006] For each item, using attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model, determine a first probability of the item under each first label;

[0007] Calculate a count matrix according to the first label and the first probability;

[0008] According to the first probability and the counting matrix, erroneous data in the table to be filtered is determined.

[0009] Optionally, after determining the erroneous data in the table to be filtered, the method further includes: correcting the erroneous data in the table to be filtered.

[0010] Optionally, the number of the preset models is multiple;

[0011] Determining the first probability of the item under each first label by using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model includes:

[0012] Inputting attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered into the multiple preset models to obtain multiple second probabilities of the item under each of the first labels;

[0013] An average value of the plurality of second probabilities is calculated, and the average value is used as the first probability.

[0014] Optionally, calculating a count matrix according to the first label and the first probability includes:

[0015] For each first tag, taking the first tag as a target tag, and taking the first probability of the object under the target tag as a target probability;

[0016] Calculate the average of the target probabilities of all items, and use the average of the target probabilities of all items as the target confidence threshold corresponding to the target label;

[0017] Comparing the target probability with the target confidence threshold, if the target probability is greater than or equal to the target confidence threshold, taking the target label as the second label;

[0018] A count matrix is ​​generated according to the first label and the second label; wherein the element C of the count matrix y′＝i,y＝j Denotes the number of items whose first label y′ is i and whose second label y is j.

[0019] Optionally, after generating the count matrix, the method further comprises:

[0020] The elements in the counting matrix are modified so that the sum of the elements in the counting matrix is ​​equal to the number of all items in the table to be filtered.

[0021] Optionally, modifying the elements in the count matrix includes:

[0022] For the elements C in the counting matrix y′＝i,y＝j , determine the number C of items with the first label i in the table to be filtered y′＝i , determine the sum of the number of items with the first label i and the second label k ∑ k C y′＝i,y＝k , where k is the enumeration value of the second label;

[0023] Calculate C y′＝i,y＝j Divide by ∑ k C y′＝i,y＝k The quotient of C y′＝iThe product C′ y′＝i,y＝j As the corrected C y′＝i,y＝j .

[0024] Optionally, determining erroneous data in the table to be filtered according to the first probability and the counting matrix;

[0025] Normalize the elements in the corrected count matrix;

[0026] According to the number of all items in the table to be filtered and the normalized count matrix, the number N of erroneous data corresponding to each manual label is calculated; wherein N is an integer greater than or equal to 0;

[0027] For each manual label, all items are sorted in descending order according to the first probability under the manual label;

[0028] The data of the N items with the lowest first probability are regarded as erroneous data.

[0029] Optionally, the method further comprises obtaining the preset model according to the following process:

[0030] Acquire training sample data, the training sample data including a history table, the history table being used to record multiple attributes of multiple items, each attribute including multiple attribute values, taking one of the multiple attributes as a feature to be filtered, and taking the attribute value of the feature to be filtered as a third label;

[0031] Determine an erroneous third label in the plurality of history tables and a fourth label corresponding to the erroneous third label;

[0032] Constructing a logistic regression model, and using a cross entropy loss function as a target loss function of the logistic regression model;

[0033] The logistic regression model is trained using the training sample data, the fourth label and the gradient descent strategy to obtain the preset model.

[0034] To achieve the above object, according to another aspect of an embodiment of the present invention, a data screening device is provided, comprising:

[0035] an acquisition module, used to acquire a table to be filtered, and determine an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label;

[0036] a probability calculation module, configured to determine, for each item, a first probability of the item under each of the first tags by using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model;

[0037] A matrix generation module, used for calculating a count matrix according to the first label and the first probability;

[0038] A data screening module is used to determine erroneous data in the table to be filtered according to the first probability and the counting matrix.

[0039] Optionally, the device further comprises a correction module for correcting erroneous data in the table to be filtered.

[0040] Optionally, the number of the preset models is multiple;

[0041] The probability calculation module is also used for:

[0042] Inputting attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered into the multiple preset models to obtain multiple second probabilities of the item under each of the first labels;

[0043] An average value of the plurality of second probabilities is calculated, and the average value is used as the first probability.

[0044] Optionally, the matrix generation module is further used for:

[0045] For each first tag, taking the first tag as a target tag, and taking the first probability of the object under the target tag as a target probability;

[0046] Calculate the average of the target probabilities of all items, and use the average of the target probabilities of all items as the target confidence threshold corresponding to the target label;

[0047] Comparing the target probability with the target confidence threshold, if the target probability is greater than or equal to the target confidence threshold, taking the target label as the second label;

[0048] A count matrix is ​​generated according to the first label and the second label; wherein the element C of the count matrix y′＝i,y＝j Denotes the number of items whose first label y′ is i and whose second label y is j.

[0049] Optionally, the matrix generation module is further used to: modify the elements in the counting matrix so that the sum of the elements in the counting matrix is ​​equal to the number of all items in the table to be filtered.

[0050] Optionally, the matrix generation module is further used for:

[0051] For the elements C in the counting matrix y′＝i,y＝j , determine the number C of items with the first label i in the table to be filtered y′＝i , determine the sum of the number of items with the first label i and the second label k ∑ k C y′＝i,y＝k , where k is the enumeration value of the second label;

[0052] Calculate C y′＝i,y＝j Divide by ∑ k C y′＝i,y＝k The quotient of C y′＝i The product C′ y′＝i,y＝j As the corrected C y′＝i,y＝j .

[0053] Optionally, the data screening module is further used for:

[0054] Normalize the elements in the corrected count matrix;

[0055] According to the number of all items in the table to be filtered and the normalized count matrix, the number N of erroneous data corresponding to each manual label is calculated; wherein N is an integer greater than or equal to 0;

[0056] For each manual label, all items are sorted in descending order according to the first probability under the manual label;

[0057] The data of the N items with the lowest first probability are regarded as erroneous data.

[0058] Optionally, the device further comprises a model training module, which is used to:

[0059] Acquire training sample data, the training sample data including a history table, the history table being used to record multiple attributes of multiple items, each attribute including multiple attribute values, taking one of the multiple attributes as a feature to be filtered, and taking the attribute value of the feature to be filtered as a third label;

[0060] Determine an erroneous third label in the plurality of history tables and a fourth label corresponding to the erroneous third label;

[0061] Constructing a logistic regression model, and using a cross entropy loss function as a target loss function of the logistic regression model;

[0062] The logistic regression model is trained using the training sample data, the fourth label and the gradient descent strategy to obtain the preset model.

[0063] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data screening method of an embodiment of the present invention.

[0064] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data screening method of an embodiment of the present invention is implemented.

[0065] An embodiment of the above invention has the following advantages or beneficial effects: because a table to be filtered is obtained and the attributes to be filtered in the table to be filtered are determined, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label; for each item, the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model are used to determine the first probability of the item under each first label; based on the first label and the first probability, a counting matrix is ​​calculated; based on the first probability and the counting matrix, the technical means of determining the erroneous data in the table to be filtered can automatically find out the erroneous data in the table and improve the efficiency and accuracy of data screening.

[0066] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0068] Figure 1 is a schematic diagram of the main process of the data screening method according to an embodiment of the present invention;

[0069] Figure 2 is a schematic diagram of a sub-process of a data screening method according to another embodiment of the present invention;

[0070] Figure 3 is a schematic diagram of a sub-process of a data screening method according to an embodiment of the present invention;

[0071] Figure 4 is a schematic diagram of the main modules of the data screening device according to an embodiment of the present invention;

[0072] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0073] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0075] Figure 1 is a schematic diagram of the main steps of the data screening method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0076] Step S101: obtaining a table to be filtered, and determining an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as a first label;

[0077] Step S102: for each item, using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model, determine the first probability of the item under each first label;

[0078] Step S103: Calculating a count matrix according to the first label and the first probability;

[0079] Step S104: Determine erroneous data in the table to be filtered according to the first probability and the counting matrix.

[0080] For step S101, as an example, the table to be filtered may be an item attribute table, each row of data in the table is a record, and each column is a different attribute of the record, which may represent the category (such as the primary category) and volume attribute (such as large item) to which the item belongs. For example, as shown in Table 1 below, the attribute to be filtered may be a volume attribute, and large item, medium item, and small item are the attribute values ​​of the attribute to be filtered, i.e., the first label.

[0081] Table 1:

[0082] No. volume weight price Product categories Volume properties 0 510 10 22 food Small items 1 200 2 59 cell phone Small items 2 2411 0.1 1203 computer Middleware 3 300 0.4 24 food Middleware 4 3411 22 55 food Small items 5 2323 1 68 computer Middleware 6 5000 60 998 computer Large items 7 3452 230 2563 Home Appliances Large items 8 89000 13 1223 Kitchenware Large items 9 4500 5.5 364 books Middleware 10 2300 1.2 342 food Middleware

[0083] For step S102, Figure 2 As shown, the preset model can be obtained according to the following process:

[0084] Step S201: Acquire training sample data, the training sample data includes a history table, the history table is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, one of the multiple attributes is used as a feature to be filtered, and the attribute value of the feature to be filtered is used as a third label;

[0085] Step S202: determining an erroneous third label and a fourth label corresponding to the erroneous third label in the plurality of history tables;

[0086] Step S203: constructing a logistic regression model, and using the cross entropy loss function as the target loss function of the logistic regression model;

[0087] Step S204: using the training sample data, the fourth label and the gradient descent strategy, training the logistic regression model to obtain the preset model.

[0088] For step S201, the number of history tables can be one or more, and the present invention does not limit this. Each row of data in the history table is a sample. As an example, the history table is shown in Table 2 below:

[0089] Table 2:

[0090] serial number volume weight price category Volume properties 1 510 10 22 food Small items 2 200 2 59 cell phone Small items 3 2411 0.1 1203 computer Middleware 4 300 0.4 24 food Middleware 5 3411 22 55 food Small items 6 2323 1 68 computer Middleware 7 5000 60 998 computer Large items 8 3452 230 2563 Home Appliances Large items 9 8900 13 1223 Kitchenware Large items 10 4500 5.5 364 books Middleware

[0091] The volume attribute in the history table is used as the feature to be filtered, and large pieces, medium pieces and small pieces are the attribute values ​​of the feature to be filtered, that is, the third label.

[0092] As for step S202, this step can be implemented by manual marking. According to the volume of each item in the history table, it can be determined that the wrong record is number 4 and number 5, and the corresponding third label is the wrong third label. According to the volume recorded in the history table, it can be determined that the fourth label corresponding to number 5 is a medium piece, and the fourth label corresponding to number 6 is a small piece.

[0093] For step S203, a supervised machine learning model can be used as a model for predicting the fourth label. In this embodiment, a logistic regression model is used as a model for predicting the fourth label. The input of the logistic regression model is the first four attributes in the history table, namely volume, weight, price and commodity category, and the output of the logistic regression model is the fourth label predicted by the model. In addition, if the attribute value of the commodity category is character type, it is necessary to convert the attribute value of the commodity category into a numeric type. The expression of the logistic regression model is shown in the following formula (1):

[0094]

[0095] Among them, x ij represents the attribute value of the jth attribute of the i-th sample in the history table except the feature to be filtered, I represents the number of samples, J represents the number of features to be filtered in the history table, for Table 2, J = 4, w j Represents x j The weight of , b is the bias term. σ(z) represents the logistic regression function, and let represents the fourth label, which is the output of the model, e is a natural constant, K is the number of attribute values ​​of the feature to be filtered, and for Table 2 above, K=3. The loss function of the logistic regression model is the cross entropy loss function: y′ i Indicates the third label of the i-th sample. The logistic regression model of this embodiment is a three-classification model, so the output of the logistic regression model is a one-dimensional vector containing three elements. As an example, if the second label of a sample is small, it is represented by a vector [1, 0, 0] (if it is medium, it is represented by [0, 1, 0], and if it is large, it is represented by [0, 0, 1]). This means that the probability that the sample is a small item is 0.5, the probability that it is a medium item is 0.4, and the probability that it is a large item is 0.1. The loss is calculated as loss = -(1×log0.5+0×log0.4+0×log0.1)≈-0.3.

[0096] For step S204, the purpose of training the model is to find a suitable set of w and b to minimize the loss, for example, to reduce it to a certain range (converge to a certain value and then remain stable). The initial values ​​of the weights can all be set to 0, and then updated through the gradient descent strategy during the model training process. The gradient descent strategy refers to the model in each round of iterative training. The gradient is obtained by the chain rule of derivation So as to seek Finally, through w j :=w j -ηdw j , update the weights. For example, in the last round w j The result of the last update is 0.7, then η is 0.1, dw j is 0.5, then in this round w j The update is w j :=0.7-0.1×0.5=0.65. The initial value of the bias term b can also be set to 0 and then updated using the gradient descent strategy.

[0097] In an optional embodiment, there are multiple preset models.

[0098] In this embodiment, the purpose of training multiple preset models is to improve the robustness of the model. Because only one prediction does not necessarily represent the true accuracy of the model, multiple predictions are performed and reliable model output is obtained through cross-validation of the results of multiple predictions.

[0099] As an example, the training sample data is divided into 10 parts, 9 of which are used for training each time, and the remaining 1 part can be used for verification, thereby obtaining 10 preset models. The attribute values ​​of other attributes of an item in the table to be filtered except the attribute to be filtered are input into the 10 preset models respectively, and the results shown in Table 3 below are obtained, which indicates that the probability of the item being a small item is 0.7, the probability of being a medium item is 0.2, and the probability of being a large item is 0.1.

[0100] Table 3:

[0101]

[0102] After obtaining multiple preset models, the preset models can be used to calculate the first probability. The step of determining the first probability of the item under each first label by using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and the preset models includes:

[0103] Determining the first probability of the item under each first label by using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model includes:

[0104] Inputting attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered into the multiple preset models to obtain multiple second probabilities of the item under each of the first labels;

[0105] An average value of the plurality of second probabilities is calculated, and the average value is used as the first probability.

[0106] For the table to be filtered shown in Table 1, its first probability is shown in Table 4 below. Wherein, p[t][j] represents the first probability of the tth item in the first label j, j=1 represents a small item, j=2 represents a medium item, and j=3 represents a large item.

[0107] Table 4:

[0108] p[t][j] j＝1 j＝2 j＝3 t＝0 0.8 0.1 0.1 t＝1 0.7 0.2 0.1 t＝2 0.7 0.1 0.2 t＝3 0.2 0.6 0.2 t＝4 0.1 0.8 0.1 t＝5 0.07 0.9 0.03 t＝6 0.05 0.05 0.9 t＝7 0.1 0.1 0.8 t＝8 0.2 0.1 0.7 t＝9 0.3 0.2 0.5 t＝10 0.18 0.7 0.12

[0109] For step S103, the element C in the counting matrix y′＝i,y＝j represents the number of items whose first label y′ is i and whose second label y is j. Therefore, before obtaining the counting matrix, it is necessary to determine the second label of each item. Specifically, the second label of each item can be determined according to the following process:

[0110] For each first tag, taking the first tag as a target tag, and taking the first probability of the object under the target tag as a target probability;

[0111] Calculate the average of the target probabilities of all items, and use the average of the target probabilities of all items as the target confidence threshold corresponding to the target label;

[0112] The target probability is compared with the target confidence threshold, and if the target probability is greater than or equal to the target confidence threshold, the target label is used as the second label.

[0113] Continuing with the above Table 4 as an example, the confidence thresholds corresponding to the first labels are shown in the following Table 5, where a[j] represents the confidence threshold corresponding to the first label j.

[0114] Table 5:

[0115] j＝1 j＝2 j＝3 a[j] 0.31 0.35 0.34

[0116] Among them, a[j=1]=(0.8+0.7+0.7+…+0.18) / 11=0.31, a[j=2]=(0.2+0.1+0.1+…+0.7) / 11=0.35, a[j=1]=(0.1+0.1+0.2+…+0.12) / 11=0.34.

[0117] After obtaining the confidence threshold corresponding to each first label, determine the size of the first probability and confidence threshold of each item, and determine the second label of the item based on the judgment result. If the first probability is greater than or equal to the confidence threshold, the first label is used as the second label. For example, for the item numbered t=0, according to Table 4, its first probability under small items is 0.8, and the corresponding confidence threshold is 0.31, so the second label of the item is small items. Add the number of items with the first label j and the second label j as the value of the corresponding element of y′=j and y=j. Similarly, calculate the values ​​of the corresponding elements of y′∈(1,2,…,j) under y∈(1,2,…,j) to obtain the counting matrix.

[0118] Continuing with the above Table 5 as an example, the calculated second labels of each item are shown in the following Table 6:

[0119] Table 6:

[0120] serial number First Label Second label t＝0 Small items Small items t＝1 Small items Small items t＝2 Middleware Small items t＝3 Middleware Middleware t＝4 Small items Middleware t＝5 Middleware Middleware t＝6 Large items Large items t＝7 Large items Large items t＝8 Large items Large items t＝9 Middleware Large items t＝10 Small items Small items

[0121] In this embodiment, after obtaining the second label, it is not possible to determine the wrong data by comparing whether the first label is the same as the second label, that is, the data of the corresponding item is not eliminated because the first label is different from the second label (the data of the item corresponding to the first probability is not eliminated because the first probability is less than the confidence threshold), because in the case of left-skewed distribution, the overall average value is pulled up by the first probability of individual items, resulting in most records needing to be eliminated. Among them, left-skewed distribution refers to a distribution in which the peak of the frequency distribution is located on one side and the tail extends to the other side, which is opposite to the "normal distribution".

[0122] The count matrix obtained according to Table 6 is shown in Table 7 below. Among them, y'=1 means that the first label is a small piece, y'=2 means that the first label is a medium piece, and y'=3 means that the first label is a large piece. y=1 means that the second label is a small piece, y=2 means that the second label is a medium piece, and y=3 means that the second label is a large piece.

[0123] Table 7:

[0124] First label\Second label y＝1 y＝2 y＝3 y'＝1 2 1 0 y'＝2 1 3 0 y'＝3 0 1 3

[0125] In some cases, the first probability under each first label is equal to the corresponding confidence threshold, then the item will have multiple second labels, and the sum of the elements in the count matrix will be greater than 1, but in fact the item will only have one true label. Therefore, after obtaining the count matrix, it is necessary to correct the count matrix so that the sum of the elements in the count matrix is ​​equal to the number of all items in the table to be filtered. Specifically, the count matrix is ​​corrected according to the following process:

[0126] For the elements C in the counting matrix y′＝i,y＝j , determine the number C of items with the first label i in the table to be filtered y′＝i , determine the sum of the number of items with the first label i and the second label k ∑ k C y′＝i,y＝k , where k is the enumeration value of the second label;

[0127] Calculate C y′＝i,y＝j Divide by ∑ k C y′＝i,y＝k The quotient of C y′＝i The product C′ y′＝i,y＝j As the corrected C y′＝i,y＝j .

[0128] Specifically, the count matrix is ​​modified according to the following formula (1):

[0129]

[0130] The process of correcting Table 7 according to formula (1) is as follows:

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] Therefore, the corrected count matrix is ​​shown in Table 8 below:

[0141] Table 8:

[0142] First label\Second label y＝1 y＝2 y＝3 y'＝1 2 1 0 y'＝2 5 / 4 15 / 4 0 y'＝3 0 3 / 4 9 / 4

[0143] For step S104, Figure 3 As shown, the erroneous data in the table to be filtered can be determined according to the following process:

[0144] Step S301: normalizing the elements in the corrected counting matrix;

[0145] Step S302: Calculate the number N of erroneous data corresponding to each manual label according to the number of all items in the table to be filtered and the normalized count matrix; wherein N is an integer greater than or equal to 0;

[0146] Step S303: for each artificial label, sort all items in descending order according to the first probability under the artificial label;

[0147] Step S304: The data of the N items with the lowest first probability are regarded as erroneous data.

[0148] For step S301, normalization means: 1) converting the data into a decimal between (0, 1), which is mainly proposed for the convenience of data processing. Mapping the data to the range of 0 to 1 for processing is more convenient and faster; 2) converting the dimensional expression into a dimensionless expression to facilitate the comparison of indicators of different units or magnitudes.

[0149] The elements in Table 8 are normalized according to the following formula (2) to obtain the following Table 9:

[0150]

[0151] Table 9:

[0152] First label\Second label y＝1 y＝2 y＝3 y'＝1 2 / 11 1 / 11 0 y'＝2 5 / 44 15 / 44 0 y'＝3 0 3 / 44 9 / 44

[0153] For step S302, the number N of erroneous data corresponding to each manual label is calculated according to the following formula (3):

[0154] N=S·∑ j∈1,2,3；i≠j Q y′＝i,y＝j (3)

[0155] That is, for small items, the number of erroneous data

[0156] For middleware, the number of erroneous data

[0157] For small items, the number of erroneous data N = S·∑ j＝3；i≠j Q y′＝i,y＝1 =11×(0+0)=0.

[0158] In this embodiment, the number of erroneous data is an integer, so it is necessary to round the calculated N. Then, for the above embodiment, for small items, the number of erroneous data is 1 record; for medium items, the number of erroneous data is 2 records, and for large items, there is no data that needs to be eliminated.

[0159] For step S303, according to Table 4 above, the sequence of items with the first label of small items after being sorted in descending order according to the first probability is: t=0, t=1, t=4. The sequence of items with the first label of medium items after being sorted in descending order according to the first probability is: t=2, t=3, t=9, t=10, t=5.

[0160] For step S304, according to the results of steps S302 and S303, the erroneous data are the records of items numbered 4, 10, and 5.

[0161] The data screening method of the embodiment of the present invention determines the attributes to be filtered in the table to be filtered by acquiring a table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label; for each item, the first probability of the item under each first label is determined using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model; a counting matrix is ​​calculated based on the first label and the first probability; and erroneous data in the table to be filtered is determined based on the first probability and the counting matrix, so that erroneous data in the table can be automatically found, thereby improving the efficiency and accuracy of data screening.

[0162] In an optional embodiment, after determining the erroneous data in the table to be filtered, the erroneous data may be modified. Specifically, the erroneous data may be modified according to the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered.

[0163] Continuing with the above embodiment as an example, the erroneous data is the record of items numbered 4, 5, and 10 in Table 1. According to the attribute values ​​of other attributes recorded in Table 1, the volume attribute of item numbered 4 is changed to large, the volume attribute of item numbered 5 is changed to medium, and the volume attribute of item numbered 10 is small.

[0164] Figure 4 4 is a schematic diagram of the main modules of the data screening device 400 according to an embodiment of the present invention. Figure 4 As shown, the device 400 includes:

[0165] The acquisition module 401 is used to acquire a table to be filtered, and determine an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label;

[0166] A probability calculation module 402 is used to determine, for each item, a first probability of the item under each first label by using attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model;

[0167] A matrix generating module 403, configured to calculate a count matrix according to the first label and the first probability;

[0168] The data screening module 404 is used to determine the erroneous data in the table to be filtered according to the first probability and the counting matrix.

[0169] Optionally, the device further comprises a correction module for correcting erroneous data in the table to be filtered.

[0170] Optionally, the number of the preset models is multiple;

[0171] The probability calculation module 402 is also used to: input the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered into the multiple preset models to obtain multiple second probabilities of the item under each first label; calculate the average value of the multiple second probabilities, and use the average value as the first probability.

[0172] Optionally, the matrix generation module 403 is further used to: for each first label, use the first label as the target label, and use the first probability of the item under the target label as the target probability; calculate the average of the target probabilities of all items, and use the average of the target probabilities of all items as the target confidence threshold corresponding to the target label; compare the target probability with the target confidence threshold, and if the target probability is greater than or equal to the target confidence threshold, use the target label as the second label; generate a count matrix based on the first label and the second label; wherein the element C of the count matrix is y′＝i,y＝j Denotes the number of items whose first label y′ is i and whose second label y is j.

[0173] Optionally, the matrix generation module 403 is further used to: modify the elements in the counting matrix so that the sum of the elements in the counting matrix is ​​equal to the number of all items in the table to be filtered.

[0174] Optionally, the matrix generation module 403 is further used to: for the element C in the counting matrix y′＝i,y＝j , determine the number C of items with the first label i in the table to be filtered y′＝i , determine the sum of the number of items with the first label i and the second label k ∑ k C y′＝i,y＝k , where k is the enumeration value of the second label; calculate C y′＝i,y＝j Divide by ∑ k C y′＝i,y＝k The quotient of C y′＝i The product C′ y′＝i,y＝j As the corrected C y′＝i,y＝j .

[0175] Optionally, the data screening module 404 is also used to: normalize the elements in the corrected counting matrix; calculate the number N of erroneous data corresponding to each manual label according to the number of all items in the table to be filtered and the normalized counting matrix; wherein N is an integer greater than or equal to 0; for each manual label, sort all items in descending order according to the first probability under the manual label; and take the data of the N items with the lowest first probability as erroneous data.

[0176] Optionally, the device also includes a model training module, which is used to: obtain training sample data, the training sample data includes a historical table, the historical table is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, one of the multiple attributes is used as a feature to be filtered, and the attribute value of the feature to be filtered is used as a third label; determine the erroneous third labels in the multiple historical tables and the fourth label corresponding to the erroneous third label; construct a logistic regression model, and use the cross entropy loss function as the target loss function of the logistic regression model; use the training sample data, the fourth label and the gradient descent strategy to train the logistic regression model to obtain the preset model.

[0177] The data screening device of the embodiment of the present invention determines the attributes to be filtered in the table to be filtered by acquiring the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label; for each item, the first probability of the item under each first label is determined using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model; a counting matrix is ​​calculated based on the first label and the first probability; and erroneous data in the table to be filtered is determined based on the first probability and the counting matrix, so that erroneous data in the table can be automatically found, thereby improving the efficiency and accuracy of data screening.

[0178] The above device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0179] Figure 5 An exemplary system architecture 500 is shown to which the data screening method or data screening device according to the embodiment of the present invention can be applied.

[0180] like Figure 5As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0181] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0182] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0183] The server 505 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information and product information) to the terminal device.

[0184] It should be noted that the data screening method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data screening device is generally disposed in the server 505 .

[0185] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0186] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0187] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0188] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0189] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0190] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0191] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0192] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a sending module, an acquisition module, a determination module, and a first processing module. The names of these modules do not, in some cases, constitute a limitation on the unit itself. For example, the sending module may also be described as a "module for sending a picture acquisition request to the connected server."

[0193] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0194] Acquire a table to be filtered, and determine an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as a first label;

[0195] For each item, using attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model, determine a first probability of the item under each first label;

[0196] Calculate a count matrix according to the first label and the first probability;

[0197] According to the first probability and the counting matrix, erroneous data in the table to be filtered is determined.

[0198] The technical solution of the embodiment of the present invention can automatically find out erroneous data in a table and improve the efficiency and accuracy of data screening.

[0199] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data screening method, characterized in that: include: Acquire a table to be filtered, and determine an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as a first label; For each item, using attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model, determine a first probability of the item under each first label; Calculating a count matrix according to the first label and the first probability includes: for each first label, taking the first label as a target label, and taking the first probability of the item under the target label as a target probability; calculating an average value of the target probabilities of all items, and taking the average value of the target probabilities of all items as a target confidence threshold corresponding to the target label; comparing the target probability with the target confidence threshold, and if the target probability is greater than or equal to the target confidence threshold, taking the target label as a second label; generating a count matrix according to the first label and the second label; According to the first probability and the counting matrix, erroneous data in the table to be filtered is determined.

2. The method according to claim 1, characterized in that: After determining the erroneous data in the table to be filtered, the method further includes: correcting the erroneous data in the table to be filtered.

3. The method according to claim 1, characterized in that: The number of the preset models is multiple; Determining the first probability of the item under each first label by using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model includes: Inputting the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered into multiple preset models to obtain multiple second probabilities of the item under each of the first labels; An average value of the plurality of second probabilities is calculated, and the average value is used as the first probability.

4. The method according to claim 1, characterized in that: The elements of the counting matrix C y′yi,y＝j Denotes the number of items whose first label y′ is i and whose second label y is j.

5. The method according to claim 4, characterized in that After generating the count matrix, the method further comprises: The elements in the counting matrix are modified so that the sum of the elements in the counting matrix is ​​equal to the number of all items in the table to be filtered.

6. The method according to claim 5, characterized in that Modifying the elements in the count matrix includes: For the elements C in the counting matrix y′＝i,y＝j , determine the number C of items with the first label i in the table to be filtered y′＝i , determine the sum of the number of items with the first label i and the second label k ∑ k C y′＝i,y＝k , where k is the enumeration value of the second label; Calculate C y′＝i,y＝j Divide by ∑ k C y′＝i,y＝k The quotient of C y′＝i The product C′ y′＝i,y＝j As the corrected C y′＝i,y＝j .

7. The method according to claim 6, characterized in that Determining, according to the first probability and the counting matrix, erroneous data in the table to be filtered includes: Normalize the elements in the corrected count matrix; According to the number of all items in the table to be filtered and the normalized count matrix, the number N of erroneous data corresponding to each manual label is calculated; wherein N is an integer greater than or equal to 0; For each manual label, all items are sorted in descending order according to the first probability under the manual label; The data of the N items with the lowest first probability are regarded as erroneous data.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises obtaining the preset model according to the following process: Acquire training sample data, the training sample data including a history table, the history table being used to record multiple attributes of multiple items, each attribute including multiple attribute values, taking one of the multiple attributes as a feature to be filtered, and taking the attribute value of the feature to be filtered as a third label; Determine an erroneous third label and a fourth label corresponding to the erroneous third label in a plurality of history tables; Constructing a logistic regression model, and using a cross entropy loss function as a target loss function of the logistic regression model; The logistic regression model is trained using the training sample data, the fourth label and the gradient descent strategy to obtain the preset model.

9. A data screening device, characterized in that: include: an acquisition module, used to acquire a table to be filtered, and determine an attribute to be filtered in the table to be filtered, wherein the table to be filtered is used to record multiple attributes of multiple items, each attribute includes multiple attribute values, the attribute to be filtered is one of the multiple attributes, and the attribute value of the attribute to be filtered is used as the first label; a probability calculation module, configured to determine, for each item, a first probability of the item under each of the first tags by using the attribute values ​​of other attributes in the table to be filtered except the attribute to be filtered and a preset model; A matrix generation module, for calculating a count matrix according to the first label and the first probability, including: for each first label, taking the first label as a target label, taking the first probability of the item under the target label as a target probability; calculating an average value of the target probabilities of all items, taking the average value of the target probabilities of all items as a target confidence threshold corresponding to the target label; comparing the target probability with the target confidence threshold, and if the target probability is greater than or equal to the target confidence threshold, taking the target label as a second label; generating a count matrix according to the first label and the second label; A data screening module is used to determine erroneous data in the table to be filtered according to the first probability and the counting matrix.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images