A data processing platform and method based on big data

Abstract

The present invention provides a data processing platform and method based on big data, belonging to the field of data processing; solving the problem of low data processing efficiency; specifically as follows: a data acquisition module acquires data to be stored and divides it into hot data, warm data and cold data; a data storage module encrypts and stores warm data and cold data; applies access rights to warm data and cold data, and opens access rights to hot data; a rights management module calculates the frequency of use of each software by each user, constructs a judgment equation, and judges whether to grant access rights to warm data and cold data to each type of user; a continuous monitoring module marks abnormal users and provides feedback; the present invention acquires, analyzes and processes relevant user data, saves the data, analyzes user usage habits, opens access rights to different data, and improves information security.

Description

Technical Field

[0001] The present invention provides a data processing platform and method based on big data, and relates to the field of data processing. Background Art

[0002] Existing data processing platforms and methods have the following deficiencies:

[0003] Data dependency: Existing data management platforms are highly dependent on user usage data. If the data collected by the system is insufficient or of low quality, the analysis results will be inaccurate or the system will be unable to provide effective services to users.

[0004] System stability: Existing data management platforms need to track and analyze user usage behavior data in real time, so the system stability and performance requirements are high. If the system fails or has performance bottlenecks, it may cause users to be unable to use the system normally or data may be lost.

[0005] Data quality: The data collected by existing data management platforms are easily affected by various factors (such as user errors, system failures, etc.), resulting in low data quality or errors. If the system does not take effective data cleaning and integration measures, it may lead to inaccurate analysis results or fail to provide effective services to users. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a data processing platform and method based on big data, aiming to solve the problem of low data processing efficiency.

[0007] In order to achieve the above-mentioned object, the present invention is implemented through the following technical solutions: A data processing platform based on big data includes:

[0008] Data acquisition module: used to obtain the user ID number of the data to be stored, obtain the data to be stored corresponding to each user ID number, and divide the data to be stored into hot data, warm data and cold data;

[0009] Data storage module: used to encrypt all users' warm data and store it in the local server; decentralize and encrypt all users' cold data and store it in the cloud; impose access permissions on warm and cold data, and open access permissions on hot data;

[0010] Permission management module: This module is used to obtain each user's working time on various software programs and calculate the usage frequency of each software program for each user. Based on the user's usage frequency, a clustering algorithm is used to calculate the expected probability of the user using each software program. Based on the expected probability, a judgment equation is constructed to determine whether to grant each user access rights to warm and cold data.

[0011] Continuous monitoring module: used to mark users who are not granted warm data access rights as abnormal users and provide feedback.

[0012] Furthermore, the specific process of the data acquisition module is as follows:

[0013] Process A1: Count the number of user ID numbers un; Count the number of data to be classified fl for the 1st to unth users (1) ~fl (un) ;

[0014] Process A2: fl (1) As fn, the data to be stored corresponding to the first user is used as the target data; the target data is divided into hot data, warm data, and cold data;

[0015] Process A3: Repeat the same steps of allocating the data to be classified for the first user (i.e., process A2), and divide the data to be classified corresponding to the second to unth user ID numbers into hot data, warm data, and cold data, and enter the data storage module.

[0016] Furthermore, the specific process of the data storage module is as follows:

[0017] Process B1: Encrypt all warm data and store them in the local server;

[0018] Process B2: Count the total number of cold data corresponding to the 1st to unth users cc (1) ~cc (un) ; Calculate cc (1) ~cc (un) The sum of is recorded as acc;

[0019] Set the first user from 1 to cc (1) The size of cold data is recorded as: fb(1,1)~fb(1,cc (1) );

[0020] Similarly, the unth user from 1st to ccth (un) The size of cold data is recorded as: fb(un,1)~fb(un,cc (un) );

[0021] Process B3: Calculate and extract fb(1,1)~fb(1,cc (1) ) the average value af (1) , maximum value mf (1) , minimum value lf (1) ;

[0022] By analogy, fb(un,1)~fb(un,cc (un) ) the average value af (un) , maximum value mf(un) , minimum value lf (un) ;

[0023] Process B4: The total amount of cold data for user i is recorded as cc (i) , the average value of cold data size is recorded as af (i) , the maximum value is recorded as mf (i) , the minimum value is recorded as lf (i) ; The value range of i is: 1~un;

[0024] The standard size of data segmentation is calculated and recorded as Δfi:

[0025]

[0026] Process B5: Decentralize and encrypt all cold data corresponding to users 1 to un based on Δfi, and construct a Merkle tree. Based on the node structure of the Merkle tree, store the cold data in the cloud blockchain, and provide the blockchain access interface to users 1 to un, allowing them to enter the permission management module.

[0027] Furthermore, the workflow of the rights management module is as follows:

[0028] Process C1: Get the user's working time wt, the current month m, and the number of working days wd in month (m-1);

[0029] Calculate the frequency of office software used by users 1 to un (1) ~ha (un) ;

[0030] Frequency of using social software hb (1) ~hb (un) ;

[0031] Frequency of using entertainment software hc (1) ~hc (un) ;

[0032] Frequency of using other software hd (1) ~hd (un) ;

[0033] Process C2: Using clustering algorithm, (1) ~ha (un) Perform clustering processing to obtain stable parameter clusters;

[0034] Process C3: Count the number of stable parameter clusters, denoted as cu;

[0035] Count the number of parameters in the 1st to cuth stable parameter clusters, denoted as an (1) ~an (cu);

[0036] The parameters in the 1st to cnth stable parameter clusters are recorded as parameters ct(1,1)~ct(cu,an (cu) );

[0037] Process C4: Extract the parameters ct(1,1)~ct(1,an in the first stable parameter cluster (1) ), calculate the probability Pa(1).

[0038] Furthermore, the subsequent process of process C4 is as follows:

[0039] Process C5: Repeat the same process of calculating Pa(1) to calculate the probabilities Pa(2)~Pa(cu) of the office software corresponding to the 2nd to cuth stable parameter clusters;

[0040] Calculate the average value Pa of Pa(1) to Pa(cu) as the expected probability of the user using office software;

[0041] Repeat the same process of calculating Pa to calculate the expected probability Pb ​​of the user using social software, the expected probability Pc of using entertainment software, and the expected probability Pd of using other software;

[0042] Process C6: Extract the frequency of the first user using office software ha (1) , frequency of using social software hb (1) , frequency of using entertainment software hc (1) , use other software frequency hd (1) ;

[0043] Build a behavioral model for the first user and determine whether to grant permissions for warm and cold data.

[0044] Process C7: Repeat the same process of determining the warm data and cold data permissions of the first user; determine whether to grant warm data and cold data permissions to the second to unth users.

[0045] Furthermore, the specific process of process C2 is as follows:

[0046] Process C21: ha (1) ~ha (un) Perform a clustering;

[0047] In ha (1) ~ha (un) Randomly select kn parameters as central parameters: haa (1) ~haa (kn) ; kn represents the number of central parameters;

[0048] Process C22: Define calculation formula C2-1:

[0049] Among them, ha (i) Indicates the frequency of the i-th user using office software, haa (k) represents the knth center parameter, d (i-k) Indicates ha (i) Relative haa (k) The matching coefficient of

[0050] Process C23: Using formula C2-1, calculate ha (1) ~ha (un) The matching coefficient d (1-1) ~d (un-kn) , for ha (1) Perform clustering;

[0051] Process C231: Extract matching coefficient d (1-1) ~d (1-kn) ;

[0052] Judge d (1-1) ~d (1-kn) Whether the minimum value in is unique, enter different processes;

[0053] Process C232: If d (1-1) ~d (1-kn) The minimum value in is unique;

[0054] Extract d (1-1) ~d (1-kn) The minimum value dl in the parameter ha (1) Divide the center parameter corresponding to dl;

[0055] Process C233: If d (1-1) ~d (1-kn) The minimum value in is not unique; (1-1) ~d (1-kn) The minimum value of the tie, denoted as dll (1) ~dll (s) ; Where s represents the number of tied minimum values;

[0056] Process C2331: Extract dll (1) ~dll (s) The corresponding central parameter is taken as the undetermined parameter; the undetermined parameter is recorded as cel (1) ~cel (s) ;

[0057] Define the relationship C2-2:

[0058] (|cel (i) |-|dll (i) |)=|cel (i)-dll (i) |;

[0059] Among them, cel (i) Indicates the i-th pending parameter, dll (i) Indicates dll (1) ~dll (s) The value range of the i-th parameter in is: 1~s;

[0060] Process C2332: cel (1) ~cel (s) Substitute into the relation C2-2, find the undetermined parameter that satisfies the relation C2-2, denoted as cey; replace the parameter ha (1) Divide the center parameter corresponding to cey.

[0061] Furthermore, the subsequent process of process C23 is as follows:

[0062] Process C24: Repeat the parameter ha (1) The same process of clustering is carried out, and the parameter ha (2) ~ha (un) Perform clustering to obtain the 1st to knth clusters, recorded as cluster c (1) ~cluster c (kn) ;

[0063] Among them, ha (2) Indicates the frequency of the second user using office software;

[0064] Process C25: Calculate the average parameters of the 1st to knth clusters and obtain ac (1) ~ac (kn) ;

[0065] Define the relationship C2-3:

[0066] (|haa (i) |-|ac (i) )×|haa (i) -ac (i) |>0;

[0067] Among them, haa (i) represents the i-th center parameter, ac (i) represents the average parameter of the i-th cluster, and the value range of i is: 1~kn;

[0068] Process C26: Haa (1) ~haa (kn) and ac (1) ~ac (kn) Substitute into relation A3 to determine the new center parameter of the secondary clustering;

[0069] If haa(1) and ac (1) If the relationship C2-3 is satisfied, the new center parameter is ac (1) ; If haa (1) and ac (1) If the relationship C2-3 is not satisfied, the new center parameter is haa (1) and ac (1) ;

[0070] By analogy, if haa (kn) and ac (kn) If the relationship C2-3 is satisfied, the new center parameter is ac (kn) ; If haa (kn) and ac (kn) If the relationship C2-3 is not satisfied, the new center parameter is haa (kn) and ac (kn) ;

[0071] Process C27: Using the new center parameter as the clustering benchmark, repeat the same clustering process. (1) ~ha (un) Perform secondary clustering until the central parameter of each cluster no longer changes, and obtain a stable parameter cluster.

[0072] Furthermore, the specific process of process C4 is as follows:

[0073] Process C41: Create an empty matrix ZO and fill it with ct(1,1)~ct(1,an (1) ), get the matrix ZE;

[0074] Process C42: Calculate ct(1,1)~ct(1,an (1) )’s average value lx;

[0075] Let the parameter of the i-th row in the matrix ZE be x(i), and define the calculation formula C2-4:

[0076] Y(i) = x(i) - lx; where Y(i) represents the central value of x(i);

[0077] According to formula C2-4, calculate the center Y of matrix ZE;

[0078] Process C43: denote the parameters in the first stable parameter cluster as ct(1, j);

[0079] The covariance matrix of the matrix ZE is denoted as E;

[0080] E=[(Y) T ×Y] / (j-1); construct the probability density function p(ct (1,j) ), we get formula C2-5:

[0081] in, represents the square root of the matrix E;

[0082] The exponential number of f(ct(1,j)):

[0083]

[0084] Among them, I (an(1)) indicates (an (1) ×1) represents a matrix of all 1s;

[0085] Process C44: Calculate the probability density of ct(1,1)~ct(1,an(1)): pp(ct(1,1))~pp(ct(1,an (1) ));

[0086] Define the relationship C2-6:

[0087] Count the number of parameters that satisfy equations 2-6, denoted as au;

[0088] Calculate the probability Pa(1) that the user corresponding to the first stable parameter cluster uses office software, Pa(1) = au / an (1) .

[0089] Furthermore, the specific process of process C6 is as follows:

[0090] Process C61: Calculate ha (1) 、hb (1) 、hc (1) and hd (1) The ratio of la, lb, lc and ld is obtained; the sum of la, lb, lc and ld is calculated and recorded as ll;

[0091] Process C62: constructing equation group C3;

[0092] Process C621: Let h' represent ha (1) 、hb (1) 、hc (1) and hd (1) ; Let a represent the software currently used by the user; P(a-h') represents the first user changing from software a to software h ， software probability;

[0093] If a represents office software, construct equation C3-1:

[0094]

[0095] If a represents social software, construct equation C3-2:

[0096]

[0097] If a represents entertainment software, construct equation C3-3:

[0098]

[0099] If a represents office software, construct equation C3-4:

[0100]

[0101] Equations C3-1 to C3-4, equation group C3;

[0102] Process C622: Obtain the software currently used by the first user and, using equation group C3, calculate the frequency of the first user using office software (denoted as ya), the frequency of using social software (denoted as yb), the frequency of using entertainment software (denoted as yc), and the frequency of using other software (denoted as yd);

[0103] Process C623: Arrange office software, social software, entertainment software, and other software in descending order of ya, yb, yc, and yd to obtain an optimal sequence;

[0104] Process C624: Check whether the subsequent software usage of the first user matches the preferred sequence;

[0105] If there is a match, partial warm data access rights are opened to the first user;

[0106] If they do not match, the warm data access permission will not be granted.

[0107] A data processing method based on big data includes:

[0108] Step S1: Obtain user ID numbers of data to be stored, obtain the data to be stored corresponding to each user ID number, and divide the data to be stored into hot data, warm data, and cold data;

[0109] Step S2: Encrypt all users' warm data and store it in a local server; decentralize and encrypt all users' cold data and store it in the cloud; enforce access rights on warm and cold data, and open access rights to hot data;

[0110] Step S3: Obtain each user's working time for each software and calculate the usage frequency of each software for each user; based on the user's usage frequency of each software, use a clustering algorithm to calculate the expected probability of the user using each software; construct a judgment equation based on the expected probability and determine whether to grant each user access rights to warm data and cold data;

[0111] Continuous monitoring module: used to mark users who are not granted warm data access rights as abnormal users and provide feedback;

[0112] Step S4: Mark users who are not granted warm data access rights as abnormal users and provide feedback.

[0113] Compared with the prior art, the present invention has the following beneficial effects:

[0114] Optimize cache: The present invention can dynamically adjust the cache mechanism to cache the data or query results that users frequently access locally or in memory, so that they can respond quickly when users access again; it reduces the system response time and reduces the load pressure on the server.

[0115] Access control: The present invention can set different access control policies according to the user's usage habits and permission level; for users who frequently use certain sensitive data, the system can strengthen access control and security audit of these data to ensure data security and privacy.

[0116] Anomaly detection: The present invention can promptly detect and identify abnormal access behaviors, such as illegal intrusions and data leaks; this anomaly detection function can not only help the system take timely measures to prevent data loss or damage, but also provide users with more secure and reliable data services. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0118] Figure 1 Schematic diagram of the platform of the present invention;

[0119] Figure 2 Schematic diagram of the method of the present invention;

[0120] Figure 3 This is a schematic diagram of the processing logic of the present invention. DETAILED DESCRIPTION

[0121] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0122] Example 1:

[0123] See also Figure 1 and Figure 3 , a data processing platform based on big data includes:

[0124] It should be noted that the present invention uses the “user-authorized scanning” method to obtain data;

[0125] The implementation steps of the "User Authorization Scan" method are as follows:

[0126] Step Ⅰ: The user creates a folder named "Files to be stored" on the PC and saves the absolute path of the "Files to be stored (folder)";

[0127] Step II: The user stores the user ID number and the data or files to be protected by the present invention in the "files (folders) to be stored" as data to be classified;

[0128] User ID number: used to distinguish the data source of the data to be classified;

[0129] Step III: The present invention automatically searches for the "file (folder) to be stored" through the absolute path, and then reads the data or files in the "file (folder) to be stored" to obtain the data to be classified;

[0130] Data acquisition module: used to obtain the user ID number of the data to be stored, obtain the data to be stored corresponding to each user ID number, and divide the data to be stored into hot data, warm data and cold data;

[0131] Process A: The specific process of the data acquisition module is as follows:

[0132] Process A1: Count the number of user ID numbers, denoted as un;

[0133] Count the number of unclassified data from the 1st to the unth user, denoted as: fl (1) ~fl (un) ;

[0134] Process A2: fl (1) As fn, the data to be stored corresponding to the first user is used as the target data; the target data is divided into hot data, warm data, and cold data;

[0135] It should be noted that, in the present invention, "the first user" means: "the first user ID number corresponds to the user"; "the second user" means: "the second user ID number corresponds to the user"; and so on, "the unth user" means: "the unth user ID number corresponds to the user";

[0136] Process A21: Get the size of the data from 1 to fn, recorded as b (1) ~b (fn) ;

[0137] Load the io library; use the IOException module in the io library to obtain the number of accesses from the 1st to the fnth data, recorded as: r (1) ~r (fn) ; The number of modifications is recorded as: w (1) ~w (fn) ;

[0138] Process A22: Calculate b (1) ~b (fn) The average value is recorded as ab; calculate r (1) ~r (fn) The average value is recorded as ar; calculate w (1) ~w (fn) The average value is denoted as aw;

[0139] Extract r (1) ~r (fn) The maximum value in is denoted as r (max) , the maximum value is recorded as r (min) ;

[0140] Extract w (1) ~w (fn) The maximum value in is denoted as w (max) , the maximum value is recorded as w (min) ;

[0141] Process A23: Let the size of the i-th target data be b (i) , the number of visits is r (i) , the number of modifications is w (i) ;The value range of i is: 1~fn;

[0142] Definition of relation 1: (r i +w i )×b i ≥(ar×aw) 1 / 2 ×ab;

[0143] r (1) ~r (fn) and w (1) ~w (fn) , substitute into equation 1;

[0144] Summarize the target data that does not satisfy equation 1 as cold data; count the number of cold data and record it as cn;

[0145] Summarize the target data that meets the relationship formula 1 as non-cold data; count the number of non-cold data and record it as mn;

[0146] Process A24: Let the size of the jth non-cold data be bu (j) , the number of visits is ru (j) , modification times is wu (j) ; The value range of j is: 1~mn;

[0147] Definition of relation 2: [(ru (j) +r (min) )>2×ar]∩[(w j +w(min) )>2×aw];

[0148] Definition relation 3: [(wu (j) +w (max) )>2×aw]∩[(ru (j) +r (max) )>2×ar];

[0149] Substitute the access count and modification count corresponding to non-cold data into equations 2 and 3;

[0150] The non-cold data that only satisfies equation 2 is aggregated as hot data; the non-cold data that only satisfies equation 3 or satisfies both equations 2 and 3 is aggregated as warm data.

[0151] Process A3: Repeat the same steps of allocating the data to be classified for the first user (i.e., process A2), and divide the data to be classified corresponding to the second to unth user ID numbers into hot data, warm data, and cold data, and enter the data storage module.

[0152] Data storage module: used to encrypt all users' warm data and store it in the local server; decentralize and encrypt all users' cold data and store it in the cloud; impose access permissions on warm and cold data, and open access permissions on hot data;

[0153] Process B: The specific process of the data storage module is as follows:

[0154] Process B1: Use the RAS algorithm to encrypt (all) the warm data corresponding to the 1st to unth users and store them in the local server;

[0155] Process B11: The warm data corresponding to the first user is used as quasi-data; the quasi-data is encrypted using the RAS algorithm;

[0156] Use the hash algorithm to process the first user ID number to obtain hn;

[0157] Use the Mersenne Twister algorithm to randomly generate two positive prime numbers, denoted as pr and pe; make pr and pe satisfy the following conditions:

[0158] {lim[(pr+pe) / hn]≈1}∩(pr+pe) min ;

[0159] Process B12: Calculate the Euler number of hn, denoted as φ(hn), φ(hn) = (pr - 1) × (pe - 1);

[0160] Randomly generate an integer e smaller than φ(hn) such that the greatest common divisor of e and φ(hn) is 1, and e is the public key;

[0161] Calculate the modular inverse d of e with respect to φ(hn), d = e (φ(hn)－2) , d is the private key;

[0162] Process B13: Use public key e to encrypt (all) quasi-data to obtain data set h (1) ; (When the user accesses the warm data corresponding to the first user, the private key d is used to access the dataset h (1) Decryption ;)

[0163] Process B14: Repeat the same steps of encrypting the first user's (all) warm data, and encrypt the (all) warm data corresponding to the second to unth user ID numbers to obtain the data set h (2) To dataset h (un) ;

[0164] The dataset h (1) To dataset h (un) Store in the local server;

[0165] Process B2: Count the total number of cold data corresponding to the 1st to unth users, recorded as cc (1) ~cc (un) ; Calculate cc (1) ~cc (un) The sum of is recorded as acc;

[0166] Set the first user from 1 to cc (1) The size of cold data is recorded as: fb(1,1)~fb(1,cc (1) );

[0167] Set the second user from 1st to cc (2) The size of cold data is recorded as: fb(2,1)~fb(2,cc (2) );

[0168] Similarly, the unth user from 1st to ccth (un) The size of cold data is recorded as: fb(un,1)~fb(un,cc (un) );

[0169] Process B3: Calculate and extract fb(1,1)~fb(1,cc (1) ) the average value af (1) , maximum value mf (1) , minimum value lf (1) ;

[0170] fb(2,1)~fb(2,cc (2) ) the average value af (2) , maximum value mf (2), minimum value lf (2) ;

[0171] By analogy, fb(un,1)~fb(un,cc (un) ) the average value af (un) , maximum value mf (un) , minimum value lf (un) ;

[0172] Process B4: The total amount of cold data for user i is recorded as cc (i) , the average value of (all) cold data size is recorded as af (i) , the maximum value is recorded as mf (i) , the minimum value is recorded as lf (i) ; The value range of i is: 1~un;

[0173] The standard size of data segmentation is calculated and recorded as Δfi:

[0174]

[0175] Process B5: Decentralize and encrypt all cold data corresponding to users 1 to un based on Δfi, and construct a Merkle tree. Based on the node structure of the Merkle tree, store the cold data in the blockchain on the cloud, and provide the blockchain access interface to users 1 to un.

[0176] Permission management module: This module is used to obtain each user's working time on various software programs and calculate the usage frequency of each software program for each user. Based on the user's usage frequency, a clustering algorithm is used to calculate the expected probability of the user using each software program. Based on the expected probability, a judgment equation is constructed to determine whether to grant each user access rights to warm and cold data.

[0177] Process C: The workflow of the rights management module is as follows:

[0178] Process C1: Get the working hours of (all) users, recorded as wt; get the current month, recorded as m; get the number of working days of (all) users in month (m-1), recorded as wd;

[0179] Obtain the usage time of office software, social software, entertainment software and other software by the 1st to unth users on working days in month (m-1), and calculate the frequency of office software use by the 1st to unth users. (1) ~ha (un) ;

[0180] Frequency of using social software hb (1) ~hb (un) ;

[0181] Frequency of using entertainment software hc (1) ~hc (un) ;

[0182] Frequency of using other software hd (1) ~hd (un) ;

[0183] It should be noted that if month m is January of a certain year, then month (m-1) is December of the previous year; for example, if month m is January 2023, then month (m-1) is December 2022;

[0184] "Other software" refers to computer software other than "office software, social software and entertainment software", such as programming software;

[0185] Process C11: record the time that users 1 to un use the office software on the 1st to wdth working days of month (m-1) as ta(1,1) to ta(un,wd);

[0186] The time of using social software is recorded as tb(1,1)~tb(un,wd);

[0187] The time of using entertainment software is recorded as tc(1,1)~tc(un,wd);

[0188] The time spent using other software is recorded as td(1,1) to td(un,wd);

[0189] Process C12: Assume that the time the i-th user uses office software on the j-th working day of month (m-1) is ta(i, j), the time used for social software is tb(i, j), the time used for entertainment software is tc(i, j), and the time used for other software is td(i, j); the value range of i is: 1 to un, and the value range of j is: 1 to wd;

[0190] Define calculation formula C1-1:

[0191] Define calculation formula C1-2:

[0192] Define calculation formula C1-3:

[0193] Define calculation formula C1-4:

[0194] Among them, ha (i) 、hb (i) 、hc (i) and hd (i), respectively represent the frequency of the i-th user (at work) using office software, social software, entertainment software, and other software;

[0195] Process C13: Calculate the frequency ha of office software use by the 1st to unth users according to formulas C1-1 to C1-4 (1) ~ha (un) ;

[0196] Frequency of using social software hb (1) ~hb (un) ;

[0197] Frequency of using entertainment software hc (1) ~hc (un) ;

[0198] Frequency of using other software hd (1) ~hd (un) ;

[0199] Process C2: Using clustering algorithm, (1) ~ha (un) Perform clustering processing to obtain stable parameter clusters;

[0200] Process C21: ha (1) ~ha (un) Perform a clustering;

[0201] In ha (1) ~ha (un) Randomly select kn parameters as central parameters; where kn represents the number of central parameters, and the calculation formula of kn is: kn=(un) 1 / 2 , kn rounded up;

[0202] The 1st to knth central parameters are denoted as haa (1) ~haa (kn) ;

[0203] Process C22: Define calculation formula C2-1:

[0204] Among them, ha (i) Indicates the frequency of the i-th user using office software (at work), the value range of i is: 1 ~ un;

[0205] haa (k) represents the knth center parameter, and the value range of k is: 1 to kn;

[0206] d (i-k) Indicates ha (i) Relative haa (k) The matching coefficient of

[0207] Process C23: Using formula C2-1, calculate ha (1) ~ha (un) The matching coefficient d (1-1) ~d (un-kn) , for ha (1) Perform clustering;

[0208] Process C231: Extraction of ha (1) Relative haa (1) ~haa (kn) The matching coefficient d (1-1) ~d (1-kn) ;

[0209] Judge d (1-1) ~d (1-kn) Whether the minimum value in is unique, enter different processes;

[0210] Process C232: If d (1-1) ~d (1-kn) The minimum value in is unique;

[0211] Extract d (1-1) ~d (1-kn) The minimum value in is denoted as dl; the parameter ha (1) Divide the center parameter corresponding to dl;

[0212] Process C233: If d (1-1) ~d (1-kn) The minimum value in is not unique; (1-1) ~d (1-kn) The minimum value of the tie, denoted as dll (1) ~dll (s) ; where s represents d (1-1) ~d (1-kn) The number of tied minimum values ​​in , the value range of s is: 2~kn;

[0213] Process C2331: Extract dll (1) ~dll (s) The corresponding central parameter is taken as the undetermined parameter; the undetermined parameter is recorded as cel (1) ~cel (s) ;

[0214] Define the relationship C2-2:

[0215] (|cel (i) |-|dll (i) |)=|cel (i) -dll (i) |;

[0216] Among them, cel(i) Indicates the i-th pending parameter, dll (i) Indicates dll (1) ~dll (s) The value range of the i-th parameter in is: 1~s;

[0217] Process C2332: cel (1) ~cel (s) Substitute into the relation C2-2, find the undetermined parameter that satisfies the relation C2-2, denoted as cey; replace the parameter ha (1) Divide the central parameters corresponding to cey;

[0218] Process C24: Repeat the parameter ha (1) The same process of clustering is carried out, and the parameter ha (2) ~ha (un) Perform clustering to obtain the 1st to knth clusters, recorded as cluster c (1) ~cluster c (kn) (Complete one clustering);

[0219] Among them, ha (2) Indicates the frequency of the second user using office software (at work);

[0220] Process C25: Calculate the average parameters of the 1st to knth clusters and obtain ac (1) ~ac (kn) ;

[0221] Define the relationship C2-3:

[0222] (|haa (i) |-|ac (i) |)×|haa (i) -ac (i) |>0;

[0223] Among them, haa (i) represents the i-th center parameter, ac (i) represents the average parameter of the i-th cluster, and the value range of i is: 1~kn;

[0224] Process C26: Haa (1) ~haa (kn) and ac (1) ~ac (kn) Substitute into relation A3 to determine the new center parameter of the secondary clustering;

[0225] If haa (1) and ac (1) If the relationship C2-3 is satisfied, the new center parameter is ac (1) ; If haa (1) and ac(1) If the relationship C2-3 is not satisfied, the new center parameter is haa (1) and ac (1) ;

[0226] By analogy, if haa (kn) and ac (kn) If the relationship C2-3 is satisfied, the new center parameter is ac (kn) ; If haa (kn) and ac (kn) If the relationship C2-3 is not satisfied, the new center parameter is haa (kn) and ac (kn) ;

[0227] Process C27: Using the new center parameter as the clustering benchmark, repeat the same clustering process. (1) ~ha (un) Perform secondary clustering until the central parameter of each cluster no longer changes, and obtain a stable parameter cluster;

[0228] Process C3: Count the number of stable parameter clusters, denoted as cu;

[0229] Count the number of parameters in the 1st to cuth stable parameter clusters, denoted as an (1) ~an (cu) ;

[0230] The parameters in the 1st to cnth stable parameter clusters are recorded as parameters ct(1,1)~ct(cu,an (cu) );

[0231] Process C4: Extract the parameters ct(1,1)~ct(1,an in the first stable parameter cluster (1) ), calculate the probability Pa(1) that the user corresponding to the first stable parameter cluster uses office software;

[0232] Process C41: Create an (1) ×1), denoted as matrix ZO;

[0233] Convert ct(1,1) to ct(1,an (1) ), fill in the matrix ZO to obtain the matrix ZE;

[0234] Process C42: Calculate ct(1,1)~ct(1,an (1) ) is the average value, denoted as lx;

[0235] Let the parameter of the i-th row in the matrix ZE be x(i), and define the calculation formula C2-4:

[0236] Y(i)=x(i)-lx;where Y(i) represents the central value of x(i); the value range of i is 1~an (1) ;

[0237] According to the calculation formula C2-4, the centralization value corresponding to each parameter in the matrix ZE is calculated to obtain the centralization matrix, which is recorded as matrix Y;

[0238] Process C43: The parameters in the first stable parameter cluster are recorded as ct(1, j), and the value range of j is 1 to an (1) ;

[0239] The covariance matrix of the matrix ZE is denoted as E;

[0240] E=[(Y) T ×Y] / (j-1); where T represents the transpose of the matrix and the minimum value of (j-1) is 1;

[0241] Construct the probability density function p(ct (1,j) ), we get formula C2-5:

[0242] in, represents the square root of the matrix E (consider the matrix E as the determinant and use a recursive algorithm (such as Laplace expansion) to calculate it);

[0243] e represents a natural constant (valued at 2.7); (j / 2) is a positive integer, rounded up;

[0244] The exponential order of f(ct(1,j)), the mathematical expression of f(ct(1,j)) is:

[0245]

[0246] Among them, I (an(1)) indicates (an (1) ×1) represents a matrix of all 1s, T represents the transpose of the matrix, and -1 represents the inverse of the matrix;

[0247] Process C44: Parameters ct(1,1)~ct(1,an (1) ) into the calculation formula C2-5, and the probability density of the parameters ct(1,1)~ct(1,an(1)) is obtained, which is recorded as pp(ct(1,1))~pp(ct(1,an (1) ));

[0248] Define the relationship C2-6:

[0249] Change pp(ct(1,1))~pp(ct(1,an (1)) into equation 2-6, and count the number of parameters that satisfy equation 2-6, denoted as au;

[0250] Calculate the probability Pa(1) that the user corresponding to the first stable parameter cluster uses office software, Pa(1) = au / an (1) ;

[0251] Process C5: Repeat the same process of calculating Pa(1) to calculate the probability Pa(2)~Pa(cu) of users using office software corresponding to the 2nd to cuth stable parameter clusters;

[0252] Calculate the average value of Pa(1) to Pa(cu), record it as Pa; take Pa as the expected probability of the user using office software;

[0253] Repeat the same process of calculating Pa to calculate the expected probability Pb ​​of the user using social software, the expected probability Pc of using entertainment software, and the expected probability Pd of using other software;

[0254] Process C6: Extract the frequency of the first user using office software ha (1) , frequency of using social software hb (1) , frequency of using entertainment software hc (1) , use other software frequency hd (1) ;

[0255] Build a behavioral model for the first user and determine whether to grant permissions for warm and cold data.

[0256] Process C61: Calculate ha (1) 、hb (1) 、hc (1) and hd (1) The ratio of la, lb, lc and ld is obtained; the sum of la, lb, lc and ld is calculated and recorded as ll;

[0257] Process C62: Construct the probability equations of various software for the first user, recorded as equation group C3;

[0258] Process C621: Let h' represent ha (1) 、hb (1) 、hc (1) and hd (1) ; Let a represent the software currently used by the user; P(a-h') represents the first user changing from software a to software h ， software probability;

[0259] If a represents office software, construct equation C3-1:

[0260]

[0261] If a represents social software, construct equation C3-2:

[0262]

[0263] If a represents entertainment software, construct equation C3-3:

[0264]

[0265] If a represents office software, construct equation C3-4:

[0266]

[0267] Equations C3-1 to C3-4, equation group C3;

[0268] Process C622: Obtain the software currently used by the first user and, combined with equation group C3, calculate the frequency of the first user's (subsequent) use of office software (denoted as ya), the frequency of use of social software (denoted as yb), the frequency of use of entertainment software (denoted as yc), and the frequency of use of other software (denoted as yd);

[0269] Process C623: Arrange office software, social software, entertainment software, and other software in descending order of ya, yb, yc, and yd to obtain an optimal sequence;

[0270] Process C624: Check whether the subsequent software usage of the first user matches the preferred sequence;

[0271] If a match is found, partial warm data access rights are granted to the first user (i.e., 50% of warm data access rights. Users or relevant technical personnel can adjust warm and cold data access rights based on actual needs).

[0272] If there is no match, the warm data access permission will not be granted and the first user will be monitored again;

[0273] If the first user's software usage matches the preferred sequence multiple times (i.e., more than 3 times), the first user will be given access to all warm and cold data.

[0274] Process C7: Repeat the same process of determining the warm data and cold data permissions of the first user; determine whether to grant warm data and cold data permissions to the second to unth users.

[0275] Continuous monitoring module: used to mark users who are not granted warm data access rights as abnormal users and provide feedback.

[0276] Example 2

[0277] See also Figure 2, a data processing method based on big data includes:

[0278] Step S1: Obtain user ID numbers of data to be stored, obtain the data to be stored corresponding to each user ID number, and divide the data to be stored into hot data, warm data, and cold data;

[0279] Step S2: Encrypt all users' warm data and store it in a local server; decentralize and encrypt all users' cold data and store it in the cloud; enforce access rights on warm and cold data, and open access rights to hot data;

[0280] Step S3: Obtain each user's working time for each software and calculate the usage frequency of each software for each user; based on the user's usage frequency of each software, use a clustering algorithm to calculate the expected probability of the user using each software; construct a judgment equation based on the expected probability and determine whether to grant each user access rights to warm data and cold data;

[0281] Continuous monitoring module: used to mark users who are not granted warm data access rights as abnormal users and provide feedback;

[0282] Step S4: Mark users who are not granted warm data access rights as abnormal users and provide feedback.

[0283] The above formulas are all dimensionless and calculated by taking their numerical values. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions. For example, if there are weight coefficients and proportional coefficients, the size of the settings is to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. Regarding the size of the weight coefficient and the proportional coefficient, as long as it does not affect the proportional relationship between the parameter and the quantized value, it is fine.

[0284] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data processing platform and method based on big data, characterized in that: The platform includes: Data acquisition module: used to obtain the user ID number of the data to be stored, obtain the data to be stored corresponding to each user ID number, and divide the data to be stored into hot data, warm data and cold data; Data storage module: used to encrypt all users' warm data and store it in the local server; decentralize and encrypt all users' cold data and store it in the cloud; impose access permissions on warm and cold data, and open access permissions on hot data; Permission management module: This module is used to obtain each user's working time on various software programs and calculate the usage frequency of each software program for each user. Based on the user's usage frequency, a clustering algorithm is used to calculate the expected probability of the user using each software program. Based on the expected probability, a judgment equation is constructed to determine whether to grant each user access rights to warm and cold data. The workflow of the rights management module is as follows: Process C1: Get the user's working time wt, the current month m, and the number of working days wd in month (m-1); Calculate the frequency of office software used by users 1 to un (1) ~ha (un) ; Frequency of using social software hb (1) ~hb (un) Frequency of using entertainment software hc (1) ~hc (un) ; Frequency of using other software hd (1) ~hd (un) ; Process C2: ha (1) ~ha (un) Perform clustering processing to obtain stable parameter clusters; Process C3: Count the number of stable parameter clusters, denoted as cu; count the parameters an in the 1st to cuth stable parameter clusters (1) ~an (cu) ; The parameters in the 1st to cnth stable parameter clusters are recorded as parameters ct(1,1)~ct(cu,an ( cu ) ); Process C4: Extract the parameters ct(1,1)~ct(1,an in the first stable parameter cluster (1) ), calculate the probability Pa(1); The subsequent process of process C4 is as follows: Process C5: Repeat the same process of calculating Pa(1) to calculate the probabilities Pa(2)~Pa(cu) of the office software corresponding to the 2nd to cuth stable parameter clusters; Calculate the average value Pa of Pa(1) to Pa(cu) as the expected probability of the user using office software; Repeat the same process of calculating Pa to calculate the expected probability Pb ​​of the user using social software, the expected probability Pc of using entertainment software, and the expected probability Pd of using other software; Process C6: Extract the frequency of the first user using office software ha (1) , frequency of using social software hb (1) , frequency of using entertainment software hc (1) , use other software frequency hd (1) ; Build a behavioral model for the first user to determine whether to grant permissions for warm and cold data; Process C7: Repeat the same process of determining the permissions for warm data and cold data for the first user; determine whether to grant permissions for warm data and cold data to the second to unth users; Continuous monitoring module: used to mark users who are not granted warm data access rights as abnormal users and provide feedback.

2. A data processing platform based on big data according to claim 1, characterized in that: The specific process of the data acquisition module is as follows: Process A1: Count the number of user ID numbers un; Count the number of data to be classified fl for the 1st to unth users (1) ~fl (un) ; Process A2: fl (1) As fn, the data to be stored corresponding to the first user is used as the target data; the target data is divided into hot data, warm data, and cold data; Process A3: Repeat the same steps of allocating the data to be classified for the first user, and divide the data to be classified corresponding to the second to unth user ID numbers into hot data, warm data and cold data, and enter the data storage module.

3. A data processing platform based on big data according to claim 2, characterized in that: The specific process of the data storage module is as follows: Process B1: Encrypt all warm data and store them in the local server; Process B2: Count the total number of cold data corresponding to the 1st to unth users cc (1) ~cc (un) ; Calculate cc (1) ~cc (un) The sum of is recorded as acc; Set the first user from 1 to cc (1) The size of cold data is recorded as: fb(1,1)~fb(1,cc (1) ); and so on, the unth user 1 to cc (un) The size of cold data is recorded as: fb(un,1)~fb(un,cc (un) ); Process B3: Calculate and extract fb(1,1)~fb(1,cc (1) ) the average value af (1) , maximum value mf (1) , minimum value lf (1) ; and so on, fb(un,1)~fb(un,cc (un) ) the average value af (un) , maximum value mf (un) , minimum value lf (un) ; Process B4: The total amount of cold data for user i is recorded as cc (i) , the average value of cold data size is recorded as af (i) , the maximum value is recorded as mf (i) , the minimum value is recorded as lf (i) ; The value range of i is: 1~un; The standard size of data segmentation is calculated and recorded as Δfi: Process B5: Decentralize and encrypt all cold data corresponding to users 1 to un based on Δfi, and construct a Merkle tree. Based on the node structure of the Merkle tree, store the cold data in the cloud blockchain, and provide the blockchain access interface to users 1 to un, allowing them to enter the permission management module.

4. The data processing platform based on big data according to claim 1, characterized in that: The specific process of process C2 is as follows: Process C21: ha (1) ~ha (un) Perform a clustering; randomly select kn parameters as the central parameters: haa (1) ~haa (kn) ; Process C22: Define calculation formula C2-1: Among them, ha (i) Indicates the frequency of the i-th user using office software, haa (k) represents the kth center parameter, d (i-k) Indicates ha (i) Relative haa (k) The matching coefficient of Process C23: Using formula C2-1, calculate ha (1) ~ha (un) The matching coefficient d (1-1) ~d (un-kn) , for ha (1) Perform clustering; Process C231: Extract matching coefficient d (1-1) ~d (1-kn) ; judge d (1-1) ~d (1-kn) Whether the minimum value in is unique, enter different processes; Process C232: If d (1-1) ~d (1-kn) The minimum value in is unique; extract d (1-1) ~d (1-kn) The minimum value dl in the parameter ha (1) Divide the center parameter corresponding to dl; Process C233: If d (1-1) ~d (1-kn) The minimum value in is not unique; (1-1) ~d (1-kn) The minimum value of the tie, denoted as dll (1) ~dll (s) ; Process C2331: Extract dll (1) ~dll (s) The corresponding central parameter is taken as the undetermined parameter; the undetermined parameter is recorded as cel (1) ~cel (s) ; Define the relationship C2-2: (|the (i) |-|dll (i) |)=|the (i) -dll (i) |; Among them, cel (i) Indicates the i-th pending parameter, dll (i) Indicates dll (1) ~dll (s) The i-th parameter in; Process C2332: cel (1) ~cel (s) Substitute into the relation C2-2, and the undetermined parameter that satisfies the relation C2-2 is denoted as cey; the parameter ha (1) Divide the center parameter corresponding to cey.

5. The data processing platform based on big data according to claim 4, characterized in that: The subsequent process of process C23 is as follows: Process C24: Repeat the parameter ha (1) The same process of clustering is carried out, and the parameter ha (2) ~ha (un) Perform clustering and obtain cluster c (1) ~cluster c (kn) ; Process C25: Calculate the average parameters of the 1st to knth clusters and obtain ac (1) ~ac (kn) ; Define the relationship C2-3: (|haa (i) |-|ac (i) |)×|haa (i) -ac (i) |>0; Among them, haa (i) represents the i-th center parameter, ac (i) represents the average parameter of the i-th cluster; Process C26: Haa (1) ~haa (kn) and ac (1) ~ac (kn) Substitute into relation A3 to determine the new center parameter of the secondary clustering; If haa (1) and ac (1) If the relationship C2-3 is satisfied, the new center parameter is ac (1) ; If the relationship C2-3 is not satisfied, the new center parameter is haa (1) and ac (1) ; By analogy, if haa (kn) and ac (kn) If the relationship C2-3 is satisfied, the new center parameter is ac (kn) ; If the relationship C2-3 is not satisfied, the new center parameter is haa (kn) and ac (kn) ; Process C27: Using the new center parameter as the clustering benchmark, repeat the same clustering process. (1) ~ha (un) Perform secondary clustering until the central parameter of each cluster no longer changes, and obtain a stable parameter cluster.

6. The data processing platform based on big data according to claim 1, characterized in that: The specific process of process C4 is as follows: Process C41: Create an empty matrix ZO and fill it with ct(1,1)~ct(1,an (1) ), get the matrix ZE; Process C42: Calculate ct(1,1)~ct(1,an (1) )’s average value lx; let the parameter of the i-th row in the matrix ZE be x(i), and define the calculation formula C2-4: Y(i) = x(i) - lx; where Y(i) represents the central value of x(i); According to formula C2-4, calculate the center Y of matrix ZE; Process C43: denote the parameters in the first stable parameter cluster as ct(1, j); denote the covariance matrix of the matrix ZE as E; E = [(Y) T ×Y] / (j-1); construct the probability density function p(ct (1,j) ), we get formula C2-5: in, represents the square root of the matrix E; The exponential number of f(ct(1,j)): Among them, I (an(1)) indicates (an (1) ×1) represents a matrix of all 1s; Process C44: Calculate ct(1,1)~ct(1,an (1) ) probability density: pp(ct(1,1))~pp(ct(1,an (1) )); Define the relationship C2-6: Count the number of parameters that satisfy equations 2-6, denoted as au; Calculate the probability Pa(1) that the user corresponding to the first stable parameter cluster uses office software, Pa(1) = au / an (1) .

7. The data processing platform based on big data according to claim 1, characterized in that: The specific process of process C6 is as follows: Process C61: Calculate ha (1) 、hb (1) 、hc (1) and hd (1) The ratio of , we get la, lb, lc and ld; sum them up to get ll; Process C62: Construct equation group C3; let h' represent ha (1) 、hb (1) 、hc (1) and hd (1) ; Let a represent the software currently used by the user; P(a-h') represents the probability that the first user changes from software a to software h'; if a represents office software, construct equation C3-1: If a represents social software, construct equation C3-2: If a represents entertainment software, construct equation C3-3: If a represents office software, construct equation C3-4: Equations C3-1 to C3-4 are referred to as equation group C3; Get the software currently used by the first user and, using equation group C3, calculate the frequency of the first user using office software (denoted as ya), the frequency of using social software (denoted as yb), the frequency of using entertainment software (denoted as yc), and the frequency of using other software (denoted as yd); Arrange office software, social software, entertainment software, and other software in descending order of ya, yb, yc, and yd to obtain a preferred sequence; and check whether the subsequent software usage of the first user matches the preferred sequence; If there is a match, partial warm data access rights are opened to the first user; If they do not match, the warm data access permission will not be granted.

8. A data processing method based on big data, applicable to a data processing platform based on big data according to any one of claims 1 to 7, characterized in that: The method comprises: Step S1: Obtain user ID numbers of data to be stored, obtain the data to be stored corresponding to each user ID number, and divide the data to be stored into hot data, warm data, and cold data; Step S2: Encrypt all users' warm data and store it in a local server; decentralize and encrypt all users' cold data and store it in the cloud; enforce access rights on warm and cold data, and open access rights to hot data; Step S3: Obtain each user's working time for each software and calculate the usage frequency of each software for each user; based on the user's usage frequency of each software, use a clustering algorithm to calculate the expected probability of the user using each software; construct a judgment equation based on the expected probability and determine whether to grant each user access rights to warm data and cold data; Continuous monitoring module: used to mark users who are not granted warm data access rights as abnormal users and provide feedback; Step S4: Mark users who are not granted warm data access rights as abnormal users and provide feedback.

Images