Multi-stage data fusion semi-supervised learning correction de-duplication user number intelligent estimation method

Abstract

The invention provides a multi-stage data fusion semi-supervised learning correction de-duplication user number intelligent estimation method. The method comprises the steps that in the first stage, multiple linear regression coefficients are generated based on a first sample set and a second sample set, and the regression coefficients are initially adjusted; in the second stage, parameter correction is carried out based on real-time real detection data, and regression coefficients are updated according to real-time errors and noise. According to the method, indexes related to deduplication user number estimation are fused, real-time two-stage fitting parameter adjustment is carried out based on flow bypass collection in a statistical time period, final state updating of a fitting coefficient is carried out according to real-time errors and noise, and the updated coefficient is used for estimating the deduplication user number. According to the real-time semi-supervised correction deduplication user number estimation method, the deduplication user number can be accurately estimated under the condition of low resource cost, huge resource overhead of full-coverage statistics is avoided, the accuracy of single-feature estimation is improved, and the complexity of neural network algorithm detection is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer networks, and in particular to a multi-stage data fusion semi-supervised learning correction and de-duplication user number intelligent estimation method. Background Art

[0002] At present, in the dynamic evaluation and detection of access security policies based on user concurrency, the mainstream detection schemes include full coverage statistics, linear feature association, and neural network mapping. The full coverage statistics method has the characteristics of high measurement data accuracy. This method is based on hash pile recursive statistics and performs full hash mapping counts on access user identifiers during the detection time period; the linear feature association method is simple and easy to use and has low latency. This method selects a single variable related to the statistics of the time period as the fitting input, and performs residual fitting through several samples to obtain the corresponding parameters for predictive analysis; the neural network mapping method has the characteristics of high fitting accuracy. It uses a large number of samples for model supervision training to learn the generalized mapping between input and output and then handle the later generalization sample prediction needs.

[0003] However, the full coverage statistical method has high latency and huge resource overhead, making it difficult to be widely used; the linear feature association method has low fitting accuracy and is difficult to meet the requirements in engineering practice; the offline training method and the inability to learn online of the neural network mapping method limit its use. In summary, the current mainstream detection methods have the problems of huge resource consumption, low estimation accuracy, and overly complex algorithms. Summary of the invention

[0004] To this end, an object of the present invention is to propose a semi-supervised learning correction and deduplication user number intelligent estimation method to solve the problems mentioned in the background technology and overcome the shortcomings of the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for estimating the number of duplicate users based on multi-stage data fusion, comprising:

[0007] Phase 1: Generate multiple linear regression coefficients based on the first sample set and the second sample set respectively and make preliminary adjustments to the regression coefficients;

[0008] The second stage: Parameter correction is performed based on real-time detection data, and the regression coefficient is updated according to real-time errors and noise.

[0009] Preferably, before the first stage, the method further includes:

[0010] Acquire real-time samples, that is, combine and record the three indicators of the total number of records, the total record flow, and the number of real deduplicated records in the incremental sub-period in a concurrent full-volume sampling manner within the sampling period of the sliding time window to obtain the first sample set and the second sample set;

[0011] The total number of records, total record traffic, and number of real user deduplication records are represented by variables X, Y, and Z, respectively;

[0012] The first sample set is specifically: s(X, Z) = {[x 1 , z 1 ],[x 2 , z 2 ],[x 3 , z 3 ]......[x n , z n ]};

[0013] The second sample set is specifically: s(Y, Z) = {[y 1 , z 1 ],[y 2 , z 2 ],[y 3 , z 3 ]......[y n , z n ]};

[0014] Where n is the number of samples.

[0015] Preferably, generating multiple linear regression coefficients based on the first sample set and the second sample set respectively comprises:

[0016] According to the first sample set and the second sample set, linear fitting description curves based on the total number of records and the total record flow for the number of real user deduplication records are made in the plane rectangular coordinate system, and the expected linear function Z = f is solved 0 (X) and Z = f 1 (Y) Get the regression coefficient.

[0017] Preferably, the regression coefficients are initially adjusted by weighted filtering.

[0018] As a preferred embodiment, more specifically, the second stage includes: introducing a real-time self-correction mechanism for regression error to dynamically and adaptively adjust the correction parameters, assuming that g(X), h(X), p(Y), and q(Y) are Z=f 0 (X) and Z = f 1 (Y) Upper and lower bounds of fitting curve parameter adjustment, f 0 (X) and f 1The mathematical expression of (Y) is in the form of y=kx+b, where k and b are the regression parameters to be fitted, i.e. Z=f 0 (X) and Z = f 1 (Y) The parameters k and b of both are slidably corrected within the feasible domain of g(X), h(X) and p(Y), q(Y) as new sampling period samples are added.

[0019] Preferably, after the second stage, the method further includes: using the updated regression parameters to estimate the number of duplicates removed in the new sampling period, assuming that the updated linear expressions are respectively and in They are the optimal estimates of the total number of records and the total record flow of the number of users with duplicate removal in the self-correction period, and the final prediction result F = ω 1 *f 0 (X)+ω 2 *f 1 (Y), where ω 1 ,ω 2 Yes 0 (X) and f 1 The relative weight of (Y) and ω 1 +ω 2 =1.

[0020] A deduplicated user number estimation system based on multi-stage data fusion, comprising:

[0021] The data collection module is used to collect data samples, including the total number of records, total record flow, the number of real deduplication records, and the incremental supervision sample data of the sampling period within the incremental sub-period;

[0022] The multivariate linear regression module performs curve fitting and function solving based on the data collected by the data acquisition module;

[0023] The coefficient initial adjustment module uses weighted filtering to perform initial adjustment on the regression coefficient;

[0024] The self-correction module dynamically and adaptively adjusts the correction parameters according to the incremental supervision samples of the new sampling period;

[0025] The deduplication user number estimation module uses the updated regression parameters to estimate the deduplication number of the new sampling period;

[0026] The curve generation module draws the tracking curve of the number of deduplicated users and the corresponding error regression curve according to the deduplication number estimation result.

[0027] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned method for estimating the number of duplicated users based on multi-stage data fusion are implemented.

[0028] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for estimating the number of duplicated users based on multi-stage data fusion as described above.

[0029] Therefore, the present invention has the following beneficial effects:

[0030] The present invention integrates two indicators related to the estimation of the number of deduplicated users, and performs real-time two-stage fitting parameter adjustment based on traffic bypass collection within the statistical time period. In the first stage, multiple linear regression supervision coefficients are generated based on partial sample information of each indicator and the prediction coefficients are initially adjusted by weighted filtering. In the second stage, parameter correction is performed based on real-time detection data, and the final state of the fitting coefficients is updated according to real-time errors and noise. The updated coefficients are then used to estimate the number of deduplicated users. The real-time semi-supervised correction deduplication user number estimation based on multi-stage data fusion of the present invention can accurately estimate the number of deduplication users at a relatively low resource cost, avoid the huge resource overhead of full coverage statistics, improve the accuracy of single feature estimation, and reduce the complexity of neural network algorithm detection.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0033] Figure 1 is a partial flow chart of the method of the present invention;

[0034] Figure 2 is a fitting relationship diagram of the total number of records and the number of real users without duplicates in an embodiment of the present invention;

[0035] Figure 3 is a fitting relationship diagram of the total recorded traffic and the number of real users without duplicates in an embodiment of the present invention;

[0036] Figure 4 Schematic diagram of dynamic correction of total number of records and number of real users without duplicates in an embodiment of the present invention;

[0037] Figure 5 1 is a schematic diagram of dynamic correction of total recorded traffic and real user deduplication number in an embodiment of the present invention;

[0038] Figure 6 is a real deduplication user number tracking curve chart of an embodiment of the present invention;

[0039] Figure 7 is a curve diagram of the tracking error rate of the real deduplicated user number according to an embodiment of the present invention;

[0040] Figure 8 is a schematic diagram of the system structure of an embodiment of the present invention;

[0041] Fig. 9 It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0043] A semi-supervised learning method based on multi-stage data fusion to correct the number of duplicate users. Figure 1 As shown, including:

[0044] Phase 1: Generate multiple linear regression coefficients based on the first sample set and the second sample set respectively and make preliminary adjustments to the regression coefficients;

[0045] The second stage: Parameter correction is performed based on real-time detection data, and the regression coefficient is updated according to real-time errors and noise.

[0046] The present invention is a method for real-time semi-supervised correction and deduplication user number estimation based on multi-stage data fusion, which can effectively solve the problems of huge resource consumption, low estimation accuracy and overly complex algorithms of current mainstream detection methods. It is an effective improvement on traditional monitoring methods and can accurately estimate the number of deduplication users with less resource cost, avoiding the huge resource overhead of full coverage statistics, improving the accuracy of single feature estimation and reducing the complexity of neural network algorithm detection.

[0047] Preferably, before the first stage, the following is also included:

[0048] Acquire real-time samples, that is, combine and record the three indicators of the total number of records, the total record flow, and the number of real deduplicated records in the incremental sub-period in a concurrent full-volume sampling manner within the sampling period of the sliding time window to obtain the first sample set and the second sample set;

[0049] The total number of records, total record traffic, and number of real user deduplication records are represented by variables X, Y, and Z, respectively;

[0050] The first sample set is specifically: s(X,Z)={[x 1 ,z 1 ],[x 2 ,x 2 ],[z 3 ,x 3 ]……[x n ,z n ]};

[0051] The second sample set is specifically: s(Y,Z)={[y 1 ,z 1 ],[y 2 ,z 2 ],[y 3 ,z 3 ]……[y n ,z n ]};

[0052] Where n is the number of samples.

[0053] Generally, from the perspective of business forms, the number of users without duplicate user IDs and the absolute total number of users are positively correlated with the data throughput; the number of users without duplicate users also satisfies the constraints of the central limit theorem of the overall distribution, that is, it is feasible to estimate the quantitative data characteristics of the population from some representative spatiotemporal samples.

[0054] The method of the present invention combines two indicators related to the estimation of the number of deduplicated users, namely the absolute total number of users and the data throughput, and performs real-time two-stage fitting parameter adjustment based on traffic bypass collection within the statistical time period. The number of samples n can be adjusted by external parameters to ensure that Gaussian regression has a sufficient number of samples.

[0055] Preferably, generating multiple linear regression coefficients based on the first sample set and the second sample set respectively includes:

[0056] According to the first sample set and the second sample set, linear fitting description curves based on the total number of records and the total record flow for the number of real user deduplication records are made in the plane rectangular coordinate system, and the expected linear function Z = f is solved 0 (X) and Z = f 1 (Y) Get the regression coefficient.

[0057] The linear fitting results are closely related to the probability distribution of the sampling data, that is, different monitoring objects have different quality fitting results and description functions. 0 (X) and f 1 The mathematical expressions of (Y) are all in the form of y=kx+b, where k and b are regression parameters to be fitted. Linear fitting and corresponding function solution belong to basic mathematical knowledge, which do not belong to the innovation of the present invention and the scope of protection of the application, and will not be elaborated here.

[0058] Preferably, the regression coefficients are initially adjusted by weighted filtering.

[0059] In a short statistical cycle, the number of samples is limited, so direct regression calculation of the sampled preliminary data pushed this time is prone to underfitting, and the algorithm estimation accuracy is not easy to converge and control. Therefore, the present invention performs preliminary adjustment of the regression coefficient by weighted filtering.

[0060] As a preferred embodiment, more specifically, the second stage includes: introducing a real-time self-correction mechanism for regression error to dynamically and adaptively adjust the correction parameters, assuming that g(X), h(X), p(Y), and q(Y) are Z=f 0 (X) and Z = f 1 (Y) Upper and lower bounds of fitting curve parameter adjustment, f 0 (X) and f 1 The mathematical expression of (Y) is in the form of y=kx+b, where k and b are the regression parameters to be fitted, i.e. Z=f 0 (X) and Z = f 1 (Y) The parameters k and b of both are slidably corrected within the feasible domain of g(X), h(X) and p(Y), q(Y) as new sampling period samples are added.

[0061] As a preferred method, after the second stage, the method further includes: using the updated regression parameters to estimate the number of duplicates removed in the new sampling period, assuming that the updated linear expressions are: and in They are the optimal estimates of the total number of records and the total record flow of the number of users with duplicate removal in the self-correction period, and the final prediction result F = ω 1 *f 0 (X)+ω 2 *f 1 (Y), where ω 1 ,ω 2 Yes 0 (X) and f 1 The relative weight of (Y) and ω 1 +ω 2 =1.

[0062] Next, a method for estimating the number of duplicated users based on multi-stage data fusion of the present invention is further described in detail with reference to the accompanying drawings.

[0063] Generally, from the perspective of business forms, the number of users without duplicate user IDs and the absolute total number of users are positively correlated with the data throughput; the number of users without duplicate users also satisfies the constraints of the central limit theorem of the overall distribution, that is, it is feasible to estimate the quantitative data characteristics of the population from some representative spatiotemporal samples.

[0064] The present invention is used to estimate and monitor the number of user deduplication records accessed in a given sliding time window. It mainly uses multivariate linear regression, regression coefficient sampling initial adjustment, and regression error real-time self-correction methods to estimate the number of deduplication records for data with hash features. The algorithm first counts the three indicators of the total number of records, total record flow, and actual number of deduplication records in the increasing sub-period in a full sampling manner within the sliding time window sampling period. Then, two groups of samples are used to perform regression fitting and the mean of the obtained regression parameters is initially adjusted. Finally, real-time real data is used to perform short-term self-correction operations to further adapt and correct the regression parameters, reduce the estimation error of the number of user deduplication records in subsequent sampling periods, and automatically stop correction after reaching the specified correction step size. In subsequent sampling periods, the algorithm performs continuous real-time and high-precision estimation of the number of user deduplication records. The specific steps are as follows:

[0065] (1) Identify the monitored objects and their indicators.

[0066] Monitored object: This refers to the application or network service related entity.

[0067] Estimated metric: the number of users with duplicates within a certain time range.

[0068] (2) Data sample collection.

[0069] The algorithm obtains real-time samples by combining and recording the total number of records, total record traffic, and real number of deduplicated records in the incremental sub-period in a concurrent full-volume sampling manner within the sampling period of the sliding time window. The total number of records, total record traffic, and real number of deduplicated records of users are represented by variables X, Y, and Z respectively. The number of samples n can be adjusted by external parameters to ensure that Gaussian regression has a sufficient number of samples. The sample sets are defined as s(X, Z) = {[x 1 , z 1 ],[x 2 , z 2 ],[x 3 , z 3 ]......[x n , z n ]}, s(Y, Z) = {[y 1 , z 1 ],[y 2 , z 2 ],[y 3 , z 3 ]......[y n , z n ]}.

[0070] (3)Multivariate linear regression.

[0071] According to the sample sets s(X, Z) and s(Y, Z), linear fitting description curves based on the total number of records and total record traffic for the number of real user deduplication records are made in the plane rectangular coordinate system, and the expected linear function Z = f is solved 0 (X) and Z = f 1 (Y), reference Figure 2 and Figure 3 , where Actual data and Fitted line represent the actual data points and the fitted curve respectively (the same below).

[0072] The linear fitting results are closely related to the probability distribution of the sampling data, that is, different monitoring objects have different quality fitting results and description functions. 0 (X) and f 1 The mathematical expressions of (Y) are all in the form of y=kx+b, where k and b are regression parameters to be fitted. Linear fitting and corresponding function solution belong to basic mathematical knowledge, which are not the innovation of the present invention and the scope of protection of the application, and are not described in detail herein.

[0073] (4) Preliminary adjustment of regression coefficients.

[0074] In a short statistical cycle, the number of samples is limited, so direct regression calculation of the sampled preliminary data pushed this time is prone to underfitting, and the algorithm estimation accuracy is not easy to converge and control. 0 (X) and f 1 The estimated result of (Y), that is, the final prediction result F = ω 1 *f 0 (X)+ω 2 *f 1 (Y), where ω 1 ,ω 2 Yes 0 (X) and f 1 The relative weight of (Y) and ω 1 +ω 2 =1.

[0075] (5) Real-time self-correction of regression error.

[0076] In order to better track the number of deduplication of user real-time streaming data, according to the incremental supervision samples of the new sampling period, a real-time self-correction mechanism of regression error is introduced to realize dynamic adaptive adjustment of correction parameters. The adaptive parameter curves of the two groups of samples are as follows: Figure 4 and Figure 5 shown.

[0077] Where g(X), h(X), p(Y), q(Y) are respectively Z=f 0 (X) and Z = f 1(Y) The upper and lower bounds of the fitting curve parameter adjustment, i.e., Z = f 0 (X) and Z = f 1 (Y) Both parameters k and b are slidably corrected within this feasible domain as new sampling period samples are added.

[0078] (6) Estimation of the number of duplicate users.

[0079] After step (5), that is, after the algorithm completes the parameter state space adjustment and reaches stability, the updated parameters are used to estimate the number of duplicates in the new sampling period. If the updated linear expressions are and in They are the optimal estimates of the number of user deduplication within the self-correction period with respect to the total number of records and the total record traffic, respectively.

[0080] After the algorithm estimation results are stable, the tracking curve of the number of duplicate users and the corresponding error regression curve are plotted as follows: Figure 6 and Figure 7 As shown, ActualData, PredictValue and TrackingError respectively represent the actual deduplication user number curve, the predicted deduplication user number curve and the relative error rate curve between the predicted deduplication user number and the actual deduplication user number.

[0081] The present invention integrates two indicators related to the estimation of the number of deduplicated users, namely the absolute total number of users and data throughput, and performs real-time two-stage fitting parameter adjustment based on traffic bypass collection within the statistical time period. In the first stage, multiple linear regression supervision coefficients are generated based on partial sample information of each indicator and the prediction coefficients are initially adjusted by weighted filtering. In the second stage, parameter correction is performed based on real real detection data, and the final state of the fitting coefficients is updated according to real-time errors and noise. The updated coefficients are then used to estimate the number of deduplicated users. The real-time semi-supervised correction deduplication user number estimation method based on multi-stage data fusion can accurately estimate the number of deduplicated users at a relatively low resource cost, avoid the huge resource overhead of full coverage statistics, improve the accuracy of single feature estimation, and reduce the complexity of neural network algorithm detection.

[0082] The present invention also provides a deduplication user number estimation system based on multi-stage data fusion, such as Figure 8 As shown, it includes: a data acquisition module, a multivariate linear regression module, a coefficient initial adjustment module, a self-correction module, a deduplication user number estimation module and a curve generation module which are connected in sequence.

[0083] The data collection module is used to collect data samples, including the total number of records, total record flow, the number of real deduplicated records, and the incremental supervision sample data of the sampling period within the incremental sub-period.

[0084] More specifically, the data collection module is used to combine and record the three indicators of the total number of records, total record flow, and real deduplication record number in the incremental sub-period in a concurrent full sampling manner within the sampling period of the sliding time window to obtain a first sample set and a second sample set; the total number of records, total record flow, and real user deduplication record number are represented by variables X, Y, and Z respectively; the first sample set is specifically:

[0085] s(X, Z) = {[x 1 , z 1 ],[x 2 , z 2 ],[x 3 , z 3 ]......[x n , z n ]};

[0086] The second sample set is specifically:

[0087] s(Y, Z)={[y 1 , z 1 ],[y 2 , z 2 ],[y 3 , z 3 ]......[y n , z n ]}; where n is the number of samples. n Indicates the total number of records for the nth row, y n represents the total recorded flow of the nth time, z n Indicates the number of deduplicated records for the nth real user.

[0088] The multivariate linear regression module performs curve fitting and function solving based on the data collected by the data acquisition module.

[0089] More specifically, the multiple linear regression module is used to generate multiple linear regression coefficients based on the first sample set and the second sample set respectively, including: making a linear fitting description curve based on the total number of records and the total record flow for the number of deduplicated records of real users in a plane rectangular coordinate system according to the first sample set and the second sample set, and solving the expected linear function Z=f 0 (X) and Z = f 1 (Y) Get the regression coefficient.

[0090] The coefficient initial adjustment module uses weighted filtering to perform initial adjustment on the regression coefficient. The coefficient initial adjustment module contains a weighted filtering algorithm, including but not limited to any one or more of Gaussian filtering, median filtering, mean filtering, etc.

[0091] It is understandable that weighted filtering includes but is not limited to Gaussian filtering, median filtering, mean filtering, etc., and weighted filtering is used to comprehensively utilize f 0 (X) and f 1 The estimation result of (Y) removes the noise and retains the signal data characteristics. The principle and application of weighted filtering belong to the existing mature technology, and the present invention will not be repeated here.

[0092] The self-correction module dynamically and adaptively adjusts the correction parameters according to the incremental supervision samples of the new sampling period.

[0093] More specifically, the self-correction module is used for dynamic adaptive adjustment of correction parameters. Let g(X), h(X), p(Y), q(Y) be Z = f 0 (X) and Z = f 1 (Y) Upper and lower bounds of fitting curve parameter adjustment, f 0 (X) and f 1 The mathematical expression of (Y) is in the form of y=kx+b, where k and b are the regression parameters to be fitted, i.e. Z=f 0 (X) and Z = f 1 (Y) The parameters k and b of both are slidably corrected within the feasible domain of g(X), h(X) and p(Y), q(Y) as new sampling period samples are added.

[0094] The deduplication user number estimation module uses the updated regression parameters to estimate the deduplication number of the new sampling period.

[0095] The curve generation module draws the tracking curve of the number of deduplicated users and the corresponding error regression curve according to the deduplication number estimation result.

[0096] More specifically, the curve generation module is used to draw a linear fitting description curve based on the total number of records and total record traffic for the number of deduplicated records of real users, an adaptation parameter curve, a tracking curve of the number of deduplicated users, and a corresponding error regression curve in a plane rectangular coordinate system.

[0097] In order to solve the above technical problems, the embodiment of the present invention further provides a computer device, for details, please refer to Fig. 9 , including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a method for estimating the number of duplicated users based on multi-stage data fusion as described above are implemented.

[0098] The computer device includes a memory, a processor, and a network interface that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device with component memory, a processor, a network interface, and an operating system, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0099] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0100] The memory may be one or more, and may include at least one type of readable storage medium, the readable storage medium including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device. Of course, the memory may also include both an internal storage unit of a computer device and an external storage device thereof. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device, such as a program code of a method for estimating the number of duplicate users based on multi-stage data fusion, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0101] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the method for estimating the number of duplicated users based on multi-stage data fusion.

[0102] The network interface may include a wireless network interface and / or a wired network interface, which is generally used to establish a communication connection between a computer device and other electronic devices.

[0103] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which is a non-volatile readable storage medium, and the computer-readable storage medium stores the deduplication user number estimation program based on multi-stage data fusion, and the deduplication user number estimation program based on multi-stage data fusion can be executed by at least one processor so that the at least one processor performs the steps of the deduplication user number estimation method based on multi-stage data fusion as described above.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0105] It is not difficult for those skilled in the art to understand that the present invention includes any combination of the invention content and specific implementation methods of the above specification and the various parts shown in the drawings. Due to the limited space and to make the specification concise, the various schemes composed of these combinations are not described one by one. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0106] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for estimating the number of duplicate users based on multi-stage data fusion, characterized in that: include: Phase 1: Generate multiple linear regression coefficients based on the first sample set and the second sample set respectively and make preliminary adjustments to the regression coefficients; The second stage: Parameter correction is performed based on real-time detection data, and the regression coefficient is updated according to real-time errors and noise.

2. The method for estimating the number of duplicate users based on multi-stage data fusion according to claim 1, characterized in that: Before the first stage, it also includes: Acquire real-time samples, that is, combine and record the three indicators of the total number of records, the total record flow, and the number of real deduplicated records in the incremental sub-period in a concurrent full-volume sampling manner within the sampling period of the sliding time window to obtain the first sample set and the second sample set; The total number of records, total record traffic, and number of real user deduplication records are represented by variables X, Y, and Z, respectively; The first sample set is specifically: s(X,Z)={[x1,z1],[x2,z2],[x3,z3]…[x n ,z n ]}; The second sample set is specifically: s(Y,Z)={[y1,z1],[y2,z2],[y3,z3]…[y n ,z n ]}; Where n is the number of samples.

3. The method for estimating the number of duplicate users based on multi-stage data fusion according to claim 1, characterized in that: The generating of multiple linear regression coefficients based on the first sample set and the second sample set respectively comprises: According to the first sample set and the second sample set, linear fitting description curves are made based on the total number of records and the total record flow for the number of deduplicated records of real users in the plane rectangular coordinate system, and the expected linear functions Z=f0(X) and Z=f1(Y) are solved to obtain the regression coefficients.

4. The method for estimating the number of duplicate users based on multi-stage data fusion according to claim 1, characterized in that: Specifically, the regression coefficient is initially adjusted by using a weighted filtering method.

5. The method for estimating the number of duplicate users based on multi-stage data fusion according to claim 1, characterized in that: More specifically, the second stage includes: introducing a real-time self-correction mechanism for regression error to dynamically and adaptively adjust the correction parameters, assuming that g(X), h(X), p(Y), and q(Y) are the upper and lower bounds of the adjustment of the fitting curve parameters of Z=f0(X) and Z=f1(Y), respectively, and the mathematical expressions of f0(X) and f1(Y) are both in the form of y=kx+b, and k and b are the regression parameters to be fitted, that is, the parameters k and b of Z=f0(X) and Z=f1(Y) are slidably corrected within the feasible domain of g(X), h(X) and p(Y), q(Y) as new sampling period samples are added.

6. The method for estimating the number of duplicated users based on multi-stage data fusion according to claim 1, characterized in that: After the second stage, the method further includes: using the updated regression parameters to estimate the number of duplicates removed in the new sampling period, assuming that the updated linear expressions are and in They are the optimal estimates of the number of user deduplication within the self-correction period with respect to the total number of records and the total record traffic, respectively. The final prediction result is F = ω1*f0(X)+ω2*f1(Y), where ω1 and ω2 are the relative weights of f0(X) and f1(Y) and ω1+ω2=1.

7. A system for estimating the number of duplicate users based on multi-stage data fusion, applicable to the method for estimating the number of duplicate users based on multi-stage data fusion according to claim 1, characterized in that: include: The data collection module is used to collect data samples, including the total number of records, total record flow, the number of real deduplication records, and the incremental supervision sample data of the sampling period within the incremental sub-period; The multivariate linear regression module performs curve fitting and function solving based on the data collected by the data acquisition module; The coefficient initial adjustment module uses weighted filtering to perform initial adjustment on the regression coefficient; The self-correction module dynamically and adaptively adjusts the correction parameters according to the incremental supervision samples of the new sampling period; The deduplication user number estimation module uses the updated regression parameters to estimate the deduplication number of the new sampling period; The curve generation module draws the tracking curve of the number of deduplicated users and the corresponding error regression curve according to the deduplication number estimation result.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a method for estimating the number of duplicated users based on multi-stage data fusion as described in claims 1-6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for estimating the number of duplicated users based on multi-stage data fusion as described in claims 1-6 are implemented.

Images