Method for calculating new characteristic value based on data packet containing request sample

Abstract

The invention provides a method for calculating a new characteristic value based on a data packet containing a request sample, and the method comprises the following steps: S1, a data requester determines a matched crowd data packet parameter according to the limitation of data cost; s2, making a crowd data packet by the data provider according to the crowd data packet parameters provided by the data requester; s3, after the data requester obtains all the crowd data packets, grouping and classifying the crowd data packets according to samples; and S4, the data requester calculates characteristic values based on the grouped data content of the crowd data packet. According to the method, when accurate and complete data under the granularity of the unique identifier of the user cannot be obtained, the data form of the data packet is enabled to reach an applicable state only through the mining data packet.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method for calculating a new feature value based on a data packet containing a request sample. Background Art

[0002] According to Statista data, China is the second largest market in the world after the United States in terms of digital marketing. Applications in digital marketing can help manufacturers quickly understand and use the local market through data application methods, and complete marketing brand building and construction. In today's era of data explosion, most companies have their own data, but data itself cannot help companies make profits. Data monetization ultimately requires data application methods. Many companies have explored methods such as applying prediction models to precision marketing to complete monetization. Companies have a lot of data, but they do not have comprehensive data. For example, when studying and analyzing the store behavior, defeated activation behavior, and repurchase behavior of Chinese new energy vehicle consumers, two or more parties need to share data on some of the users. Sharing data often faces compliance risks, so companies will refuse to share data at the granularity of user unique identification. Both data requesters and data providers seek a safe and compliant method to avoid risks.

[0003] Existing methods such as CN118523952A encrypt identities through a blockchain network-based method and establish strict access rights to ensure that only authorized entities have the ability to identify, access and obtain private medical data; there are also technologies such as CN118282633A that are designed to make it difficult for eavesdroppers to detect the existence of data by hiding the communication system, hiding the communication sender and hiding the communication receiver, so that the hidden data can be transmitted secretly. There is also a method that does not exchange user data through federated modeling, but only transmits model parameters to each other to complete model training. The above methods are all aimed at protecting the confidentiality, security and integrity of data, or solving compliance and security issues that may arise in data transmission on the basis of not contacting each other. But they also bring about the fact that the transmission process is complex in structure and high in cost.

[0004] In addition, in some data usage scenarios, the requirements for data integrity are not strict. For example, in the application scenario of dealer lead rating, statistical machine learning modeling becomes the key, and in the data preprocessing stage, in order to reduce noise, simplify the model structure, handle nonlinear relationships, and improve computing efficiency, the feature values ​​are often binned, that is, the continuous numerical interval is divided into several discrete intervals or categories, or different discrete features are merged into fewer categories. In this case, the value of the feature label does not need to be too precise, and even the data of probability values ​​can be effectively used in statistical machine learning or deep learning modeling, and the machine can autonomously extract meaningful features or patterns from the input data. Given that the label values ​​do not need to be overly precise in these scenarios, how to obtain fuzzy label values ​​at a lower cost to promote the practical application of data has become a key issue.

[0005] In the face of such scenarios where the accuracy of label values ​​is not high, how to obtain fuzzy label values ​​at a lower cost and apply data more effectively has become an urgent problem to be solved. In particular, when processing crowd data packets (such as user portraits, consumer behavior data, etc.), how to process them into an applicable state at a lower cost is of great significance to improving business efficiency and effectiveness. Summary of the invention

[0006] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a method for calculating a new feature value based on a data packet containing a request sample.

[0007] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for calculating a new feature value based on a data packet containing a request sample, comprising the following steps:

[0008] S1, according to the relationship between data cost and the number of requested samples n (data packet granularity), determine the number of data packets g(w) required for each requested sample and the number of samples w in the data packet, where w is a hyperparameter that can be manually specified and set by the data requester. g(w) can be calculated by the data requester or the data provider; w = (w1, w2, ..., w i , ...w n ), where n is the number of requested samples, and the samples in the sample set have been deduplicated;

[0009] w i represents the number of samples required for the data packet to which the ith sample belongs, a w i Corresponding to a number of data packets, the number of samples in each data packet is the same or different;

[0010] S2, the data provider prepares the data packet data according to the parameters G and w; G = {g(w i)}=(g(w1),g(w2),…),g(w i ) represents the number of data packets required to calculate a single request sample i, and w represents the number of samples required to form a data packet;

[0011] Each time data is requested, data of several samples (the number is n) are usually requested. The data provider provides several data packets for each requested sample, where one data packet contains one piece of data, and the one piece of data contains multiple samples, including requested samples and non-requested samples.

[0012] S3, the data provider sends the prepared data packet data to the data requester. After the data requester obtains all the data packets, it groups and classifies the data packets according to the request samples, that is, a certain request sample corresponds to a number of data packets;

[0013] S4, the data requester calculates the characteristic value based on the data content after the data packet is grouped;

[0014] S5, using the feature values ​​as training data for the model to perform model training.

[0015] Further, S1 comprises the following steps:

[0016] S11, define an n-dimensional vector G, denoted by G = (g(w1), g(w2), ..., g(w i )), g(w i ) represents the number of data packets required to calculate a single request sample i;

[0017] S12, construct a data packet granularity calculation formula based on the relationship between data cost and data granularity:

[0018]

[0019] Where c represents the data cost;

[0020] μ represents the unit price of a sample.

[0021] Furthermore, if we calculate the number of samples g(w i ) is less than the 1-norm of w, it is necessary to sample the samples, and the sampling coefficient is expressed as α.

[0022] Further, S2 comprises the following steps:

[0023] S21, for each sample requested, the data provider will return the corresponding data packet group, the number of data packets in the data packet group is g(w i ); In addition to 1 requested sample, each data packet also includes wi -1 unsolicited sample; requested w i -1 non-requested sample to be removed from the current request sample The sampling is not replaced in the number of sample combinations, so that the sample combination represented by each data packet in a data packet group (that is, all data packets corresponding to a request sample) is unique and non-repetitive.

[0024] S22, after all the data packets of the requested samples are prepared, duplicates are removed according to the samples contained in the data packets, and duplicate data packets consisting of the same samples are not returned; this is because g(w i ) packets will not be repeated, but the 2 groups of packets of the two request samples may be repeated.

[0025] S23, there are several samples in the deduplicated data packet, and each sample of the several samples contains several features; extract the feature values ​​of the same feature field of the several samples, so as to obtain the prepared data packet data; the feature field of the feature value of the feature field in the prepared data packet data is of numerical type or categorical type; when it is of categorical type, its value is the mode of the same feature field of the several samples. In general applications, in order to simplify processing, the mode of the feature field is usually converted into a binary variable (0 or 1). For example, if the mode is 5, the value of the feature field can be expressed as whether it is greater than or equal to 5, and the value of the feature field will be encoded as 0 or 1. When it is of numerical type, its value is the probability of a certain value (for example, 1) of the same feature field of several samples appearing.

[0026] Further, S4 includes the following steps:

[0027] S41, taking all data packets of a request sample;

[0028] S42, calculating the information contained in all data packets of the request sample according to the feature label calculation formula, and determining the probability value of each feature label. The feature label calculation formula is expressed as:

[0029]

[0030] Among them, p(I i，f =1) indicates I i The probability that the eigenvalue of the f-th feature is 1;

[0031] g(w i ) represents the number of data packets required to calculate a single request sample i;

[0032] P(S ij ) indicates S ij Probability of occurrence;

[0033] P(Ii，f =1|S ij ) indicates S ij Under the conditions I i，f =1, that is, the characteristic value of the characteristic field obtained in step S23.

[0034] S ij Represents sample I i The jth data packet;

[0035] I i，f Represented as the i-th sample I i The fth feature of

[0036] Sample I i Total g(w i ) packets;

[0037] S43, after calculating the probability value of each request sample, a new feature value of each request sample can be obtained.

[0038] In summary, due to the adoption of the above technical solution, the present invention can make the data packet data form reach an applicable state only by mining the data packet when it is impossible to obtain accurate and complete data at the granularity of the user's unique identification. The specific beneficial effects are:

[0039] (1) Data is transmitted securely by processing data packets instead of using complex blockchain technology and communication systems, so that the data sender and data receiver do not need to perform other operations such as encryption and decryption, which can greatly save costs.

[0040] (2) At a known and controllable cost, the data provider achieves confidentiality by mixing the requested sample data with the unsolicited data. The data requester, through analysis and calculation, ultimately obtains inaccurate but still usable information that can be used for decision making, for example, by converting a data packet into a single piece of data that can be used by a machine learning model.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 It is a schematic diagram of the process of the present invention.

[0044] Figure 2 This is a schematic diagram of the composition of crowd data packets. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0046] By processing the data in the form of crowd data packets, the data provider packages the data required by the requester and multiple non-requested data into one data packet, and returns the characteristic data of this data packet to avoid risks. In addition, for the data requester, the calculated data of the returned data packet can be directly used for data application methods such as modeling. The content method of the present invention is to use multiple crowd data packets of the same sample to infer the probability that the designated label of the sample is 1. The specific process is as follows Figure 1 As shown, the following steps are included:

[0047] S1, the data requester determines the matching population data packet parameters according to the data cost limit;

[0048] S2, the data provider creates a crowd data packet according to crowd data packet parameters provided by the data requester;

[0049] S3, after the data requester obtains all the crowd data packets, the crowd data packets are grouped and classified according to samples;

[0050] S4, the data requester calculates the characteristic value based on the data content after the crowd data packets are grouped.

[0051] In this embodiment, the raw data usually obtained is not cleaned and cannot be directly used to determine the matching population data packet parameters according to the data cost limit. By removing duplicate data, duplicate data can be avoided and the number of samples can be reduced, on which basis the optimal population data packet parameters can be calculated and determined.

[0052] According to the data cost limit, determining the matching crowd data packet parameters further includes the following steps:

[0053] S11, define an n-dimensional vector G, denoted by G = (g(w1), g(w2), ..., g(w i )), g(w i ) represents the number of population data packets required to calculate a single request sample i;

[0054] S12, construct a crowd data packet granularity calculation formula based on the relationship between data cost and crowd data granularity:

[0055]

[0056] Among them; wi Indicates the number of samples required to form a crowd data packet;

[0057] c represents data cost;

[0058] n represents the number of requested samples, and the samples in the sample set have been deduplicated;

[0059] μ represents the unit price of a sample.

[0060] Further, the step of preparing a crowd data packet according to the crowd data packet parameters of the data requester includes:

[0061] S21, such as Figure 2 As shown, each sample generates a set of g(w i ) crowd data packet. A crowd data packet for a sample should contain the sample currently being requested and should also include w i -1 sample not requested this time. i -1 non-requested sample to be removed from the current request sample The number of sample combinations is sampled without replacement.

[0062] The specific implementation example is shown in Table 1. Assume that data requester A needs the values ​​of the sample with ID 3 in the database of data provider B on the feature fields "100km fuel consumption>5" and "gender_1". We set w idd＝3 The value of is 3. At this time, the number of population data packets constituting the request sample is calculated to be 3, and the number of samples contained in a data packet should be 3. There can be 6 different population data packets here. We choose (ID1, ID3, ID4) as the first population data packet, denoted as S 31 The value of this population data packet for "fuel consumption per 100 kilometers>5" is 0.67, and the value for "gender_1" is 0.33.

[0063] Table 1

[0064] ID Fuel consumption per 100 kilometers>5 Gender_1 1 0 1 2 1 1 3 1 0 4 1 0 5 0 1

[0065] S22 data cube will g(w i ) number of population data packets are produced, deduplication is performed based on the samples contained in the population data packets, and duplicate population data packets consisting of the same samples are not returned.

[0066] Assuming that party A requests a crowd data packet with ID 1 at the same time, it is possible that (ID1, ID3, ID4) will constitute another crowd data packet, denoted as S 11 At this point, you can perform deduplication, and then use S 31 replace.

[0067] Each field of the data packet provided by the S23 data party is returned in the form of probability values.

[0068] Furthermore, the steps of converting the crowd data packet data into the label probability value of a single sample are:

[0069] S41 only takes one sample g(w i )Personal group data packets;

[0070] S42 calculates the information contained in all population data packets of the sample according to the label inference formula to determine the probability value of each label. The label inference formula is expressed as:

[0071]

[0072] Among them I i，f Represented as the i-th sample I i The f characteristic, p(I i，f =1) is I i The probability that the fth feature of is 1; S ij Represents sample I i The jth population data packet, sample I i Total g(w i )Personal group data packets;

[0073] Assume that the requester A obtains 4 packets of data about the crowd with ID 1, which are (S i1 , S i2 , S i3 , S i4 ,), where the feature values ​​of “being female” are (0.32, 0.15, 0.67, 0.2) respectively. According to the formula, it can be inferred that the feature value of “being female” for the sample with ID 1 is 0.335.

[0074] S43 loops through the calculations of each feature that can be calculated in each sample.

[0075] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for calculating a new feature value based on a data packet containing a request sample, characterized in that: The following steps are involved: S1, according to the relationship between the data cost and the number of requested samples, determine the number of data packets g(w) required for each requested sample and the number of samples w in the data packet, w = (w1, w2, ..., w i , ...w n ), where n is the number of requested samples, w i Indicates the number of samples required for the data packet to which the i-th sample belongs; S2, the data provider prepares the data packet data according to the parameters G and w; G = {g(w i )}=(g(w1),g(w2),…),g(w i ) represents the number of data packets required to calculate a single request sample i; S3, the data provider sends the prepared data packet data to the data requester. After the data requester obtains all the data packets, it groups and classifies the data packets according to the request samples, that is, a certain request sample corresponds to a number of data packets; S4, the data requester calculates the characteristic value based on the data content after the data packet is grouped.

2. A method for calculating a new feature value based on a data packet containing a request sample according to claim 1, characterized in that: S1 includes the following steps: S11, define an n-dimensional vector G, denoted by G = (g(w1), g(w2), ..., g(w i )), g(w i ) represents the number of data packets required to calculate a single request sample i; S12, construct a data packet granularity calculation formula based on the relationship between data cost and data granularity: Where c represents the data cost; μ represents the unit price of a sample.

3. A method for calculating a new feature value based on a data packet containing a request sample according to claim 2, characterized in that: If g(w i ) is less than the 1-norm of w, it is necessary to sample the samples, and the sampling coefficient is expressed as α.

4. A method for calculating a new feature value based on a data packet containing a request sample according to claim 1, characterized in that: S2 includes the following steps: S21, for each sample requested, the data provider will return the corresponding data packet group, the number of data packets in the data packet group is g(w i ); In addition to 1 requested sample, each data packet also includes w i -1 unsolicited sample; requested w i -1 non-requested sample to be removed from the current request sample The number of sample combinations is not sampled back, so that the sample combination represented by each data packet in a data packet group is unique and unrepeated; S22, after all the data packets of the requested samples are prepared, duplicates are removed according to the samples included in the data packets, and duplicate data packets consisting of the same samples are not returned; S23, there are several samples in the data packet after deduplication, and each sample of the several samples contains several features; extract the feature values ​​of the same feature fields of the several samples, so as to obtain the prepared data packet data; the feature value of the feature field in the prepared data packet data is a numerical type or a categorical type; when it is a categorical type, its value is the mode of the same feature field of the several samples; when it is a numerical type, its value is the probability of a certain specific value of the same feature field of the several samples appearing.

5. A method for calculating a new feature value based on a data packet containing a request sample according to claim 1, characterized in that: S4 includes the following steps: S41, taking all data packets of a request sample; S42, calculating the information contained in all data packets of the request sample according to the feature label calculation formula, and determining the probability value of each feature label. The feature label calculation formula is expressed as: Among them, p(I i，f =1) indicates I i The probability that the eigenvalue of the f-th feature is 1; g(w i ) represents the number of data packets required to calculate a single request sample i; P(S ij ) indicates S ij Probability of occurrence; P(I i，f =1|S ij ) indicates S ij Under the conditions I i，f =1 probability of occurrence; S ij Represents sample I i The jth data packet; I i，f Represented as the i-th sample I i The fth feature of Sample I i Total g(w i ) packets; S43, after calculating the probability value of each request sample, a new feature value of each request sample can be obtained.

Images