User data processing optimization system oriented to cloud computing environment

Abstract

The invention belongs to the technical field of data processing, and discloses a user data processing optimization system oriented to a cloud computing environment. Comprising the steps of collecting original user data; preprocessing the original user data to obtain preprocessed user data; analyzing the preprocessed user data to obtain hot data and cold data, and storing the hot data and the cold data; aggregating the stored hot data and cold data to obtain an aggregated data packet; constructing a lattice space, mapping the aggregated data packet into the lattice space, obtaining the priority of the aggregated data packet, rendering a corresponding lattice into a corresponding color according to the priority of the aggregated data packet, and processing the aggregated data packet according to a rendering result; and the mutually associated data are processed together, so that the data processing speed is further improved, and cloud computing resources are saved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a user data processing optimization system for a cloud computing environment. Background Art

[0002] The patent with application publication number CN110874268A discloses a data processing method, device and equipment, which includes: first obtaining the data to be processed and the corresponding operating status information of multiple data processing systems. According to the obtained operating status information corresponding to each of the multiple data processing systems, the processing capacity of each data processing system is determined respectively. According to the processing capacity of each data processing system, the target data processing system corresponding to the first data block in the data to be processed is determined, and the first data block is sent to the target data processing system so that the target data processing system processes the first data block. By setting up multiple data processing systems, each data block can be handed over to the data processing system with the most suitable processing capacity for processing, so that the data block is processed in time, thereby ensuring that the data to be processed can be fully processed within a preset time. The user can obtain the data processing results corresponding to all the data to be processed, thereby improving the availability of the data processing results.

[0003] However, in the process of data processing, the amount of data is huge and chaotic. How to organize the data, obtain cold data and hot data, aggregate the data that affect each other, increase the speed of data processing, save cloud computing resources, and determine the priority of the data, render the corresponding grid into the corresponding color according to the priority of the data, and process the data according to the rendering result so that the data processing can be carried out in an orderly manner is the problem we need to solve. For this reason, a user data processing optimization system for cloud computing environment is provided. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a user data processing optimization system for a cloud computing environment, comprising:

[0005] Data collection module, used to collect original user data;

[0006] A data preprocessing module, used to preprocess the original user data to obtain preprocessed user data;

[0007] A data storage module is used to analyze the pre-processed user data, obtain hot data and cold data, and store the hot data and cold data;

[0008] A data aggregation module is used to aggregate the stored hot data and cold data to obtain an aggregated data packet;

[0009] The optimization processing module is used to construct a grid space, map the aggregated data packet into the grid space, obtain the priority of the aggregated data packet, render the corresponding grid into a corresponding color according to the priority of the aggregated data packet, and process the aggregated data packet according to the rendering result.

[0010] Further, the original user data includes structured data, unstructured data and semi-structured data;

[0011] The data categories included in the structured data are user account data, transaction order data and business log monitoring data, the data categories included in the unstructured data are document file data, media data and sensor Internet of Things data, and the data categories included in the semi-structured data are social data and web crawler data.

[0012] Furthermore, the method of preprocessing the original user data to obtain preprocessed user data includes:

[0013] Determine whether there are outliers in the original user data. If there are outliers, remove the outliers in the original user data and use interpolation to fill in the outliers in the removed original user data to obtain preliminary user data. If there are no outliers, record the original user data as preliminary user data.

[0014] Time-aligning data of different data categories in the preliminary user data to obtain pre-processed user data;

[0015] The process of time alignment includes:

[0016] For each data category, extract a timestamp sequence from the preliminary user data, obtain the time interval between adjacent timestamps in each timestamp sequence, compare the time intervals between adjacent timestamps in each timestamp sequence, and obtain the longest time interval;

[0017] If the time interval of the timestamp sequence is not the same as the longest time interval, it is recorded as the sequence to be converted, and the time intervals between adjacent timestamps in the sequence to be converted are converted to the longest time interval. All data between the current timestamp and the previous timestamp are collected, and the data on the current timestamp and the collected data are used as the data on the current timestamp.

[0018] Furthermore, the method for determining whether there are abnormal values ​​in the original user data includes:

[0019] Step A1: Randomly select a data category to build an isolation tree, randomly select a data in this data category as a cutting point, divide all the data in q data categories into two parts according to the cutting point, build a left subtree and a right subtree respectively, and recursively repeat for each subtree node until the number of data in the node is less than or equal to 1, where q is the number of data categories;

[0020] Step A2: Repeat step A1 to build q-1 isolation trees and calculate the deviation score of each data. i is the index of all data in q data categories, PL(i) is the deviation score of the i-th data, aj(i) is the average path length of the i-th data in all isolation trees, n is the number of all data in q data categories, and C(n) is the theoretical expected value of the path length;

[0021] in, r is the index of the isolation tree, a r (i) is the path length of the i-th data in the r-th isolation tree. The path length is the number of splits required from the cutting point to the node with only one data; H(n-1) is the harmonic number,

[0022] Step A3: Preset a deviation score threshold PY. When the deviation score of the data is higher than the deviation score threshold PY, it is determined to be an abnormal value, otherwise, it is determined to be a normal value.

[0023] Furthermore, the method for presetting the deviation score threshold includes:

[0024] Get m historical user data, get the historical deviation scores corresponding to the historical abnormal values ​​in the historical user data, and extract the minimum value PY in the historical deviation scores min and the maximum value PY max , construct a new threshold range:

[0025] Among them, k is the index of historical user data, PY kmin is the minimum value of the historical deviation score corresponding to the historical abnormal value in the k-th historical user data, PY kmax is the maximum value of the historical deviation scores corresponding to the historical abnormal values ​​in the k-th historical user data;

[0026] Obtain the number of occurrences of each historical deviation score, sort the historical deviation scores in descending order within the new threshold range according to the number of occurrences, and select the smallest historical deviation score among the first p_x historical deviation scores as the deviation score threshold.

[0027] Furthermore, the method of analyzing the pre-processed user data, obtaining hot data and cold data, and storing the hot data and cold data includes:

[0028] Get the access time, access frequency and data size closest to the current time of the preprocessed user data, and obtain the cold and hot data discrimination formula based on the linear weighted summation method:

[0029] Where t is the time difference, t = t 当 -t 近 ;t 当 is the current time, t 近 is the most recent access time, w1, w2 and w3 are weight coefficients, f is the access frequency, and s is the data size;

[0030] Set the discrimination threshold: LR0;

[0031] When LR>LR0, the pre-processed user data is recorded as hot data, and when LR≤LR0, the pre-processed user data is recorded as cold data;

[0032] The data storage module is provided with a hot data storage sub-interval and a cold data storage sub-interval, and stores hot data in the hot data storage sub-interval, and stores cold data in the cold data storage sub-interval.

[0033] Furthermore, the method of aggregating the stored hot data and cold data to obtain an aggregated data packet includes:

[0034] Step B1: Obtain the results of the hot and cold data discrimination formula corresponding to the hot data and the cold data, obtain the maximum value of the results, and mark the hot data or cold data corresponding to the maximum value of the results as aggregated sub-data;

[0035] Step B2: Acquire hot data and cold data that change simultaneously with the change of aggregated sub-data, mark the hot data and cold data as related change data, package the aggregated sub-data and the related change data, and obtain an aggregated data packet;

[0036] Step B3: Obtain hot data and cold data that are not marked as aggregate sub-data and contact change data, and record them as unmarked data, extract the results of the hot and cold data discrimination formula corresponding to the unmarked data, and obtain the maximum value of the results, mark the unmarked data corresponding to the maximum value of the results as aggregate sub-data, and repeat step B2;

[0037] Step B4: Repeat step B3 until all hot data and cold data are marked as aggregated sub-data or contact change data.

[0038] Further, the method of constructing a grid space, mapping the aggregated data packets into the grid space, obtaining the priorities of the aggregated data packets, and rendering the corresponding grids into corresponding colors according to the priorities of the aggregated data packets includes:

[0039] Construct a grid space with L rows and K columns;

[0040] Obtain the latest processing time of each hot data and cold data, obtain the processing urgency CJ of the hot data and cold data according to the latest processing time and the current time, and set the processing urgency threshold range (CJ0, CJ1);

[0041] where, CJ = t 晚 - t 当 ; t 晚 is the latest processing time;

[0042] When CJ ≤ CJ0, mark this hot data and cold data as the highest priority data, obtain the aggregated data packets with the highest priority data, set the priority of this aggregated data packet to the highest priority, and render the grid where this aggregated data packet is located red;

[0043] When CJ0 < CJ < CJ1, mark this hot data as hot first-level data and this cold data as cold first-level data;

[0044] When CJ ≥ CJ1, mark this hot data as hot second-level data and this cold data as cold second-level data;

[0045] where, CJ0 is the minimum value within the processing urgency threshold range, and CJ1 is the maximum value within the processing urgency threshold range;

[0046] Obtain the priority discrimination formula of the remaining aggregated data packets according to the quantities of hot first-level data, cold first-level data, hot second-level data, and cold second-level data: YX = g1×h(x1)+g2×d(x2)+g3×p(x3)+g4×y(x4);

[0047] where, g1, g2, g3, and g4 are respectively the quantities of hot first-level data, cold first-level data, hot second-level data, and cold second-level data in the aggregated data packet, h(x1) is the hot first-level data quantity influence function, d(x2) is the cold first-level data quantity influence function, p(x3) is the hot second-level data quantity influence function, y(x4) is the cold second-level data quantity influence function, and x1, x2, x3, and x4 are respectively the independent variables of the hot first-level data quantity influence function, the independent variable of the cold first-level data quantity influence function, the independent variable of the hot second-level data quantity influence function, and the independent variable of the cold second-level data quantity influence function;

[0048] where,

[0049] Preset priority discrimination formula result threshold range (YX0, YX1);

[0050] When YX ≤ YX0, set the priority of the aggregated data packet to the third-level priority, and render the grid where this aggregated data packet is located green;

[0051] When YX0 < YX < YX1, set the priority of the aggregated data packet to the second-level priority, and render the grid where this aggregated data packet is located yellow;

[0052] When YX ≥ YX1, set the priority of the aggregated data packet to the first-level priority, and render the grid where this aggregated data packet is located orange;

[0053] Among them, YX0 is the minimum value within the priority discrimination formula result threshold range, and YX1 is the maximum value within the priority discrimination formula result threshold range.

[0054] Further, the method for presetting the priority discrimination formula result threshold range includes:

[0055] Analyze historical user data to obtain historical hot data and historical cold data, aggregate the historical hot data and historical cold data to obtain historical aggregated data packets, and obtain the maximum value of the output result of the priority discrimination formula of the historical aggregated data packets with the first-level priority and the minimum value of the output result of the priority discrimination formula of the historical aggregated data packets with the third-level priority;

[0056] Obtain the priority discrimination formula result threshold range (YX0, YX1) according to the minimum value and the maximum value of the output result;

[0057] Among them, PJ_min is the minimum value of the output result, and PJ_max is the maximum value of the output result.

[0058] Further, the method for processing the aggregated data packet according to the rendering result includes:

[0059] Step C1: Obtain the number of red grids, green grids, yellow grids, and orange grids, and set the quantity threshold LS;

[0060] Step C2: When the number of red grids is greater than the quantity threshold, obtain the size of the aggregated data packets in the red grids, sort them in descending order, and process the first LS aggregated data packets in turn until all the aggregated data packets in the red grids are processed. When the number of red grids is less than or equal to the quantity threshold, directly process the aggregated data packets in the red grids;

[0061] Step C3: Repeat step C2 to process the aggregated data packets in the orange grid;

[0062] Step C4: repeat step C2 to process the aggregated data packets in the yellow grid;

[0063] Step C5: Repeat step C2 to process the aggregated data packets in the green grid.

[0064] The technical effects and advantages of the user data processing optimization system for cloud computing environment of the present invention are as follows:

[0065] The original user data is preprocessed to obtain preprocessed user data, and outliers in the original user data are removed, so that subsequent data processing is more accurate. Data of different data categories are time-aligned to make the data more orderly, which is convenient for subsequent data processing. The preprocessed user data is analyzed to obtain hot data and cold data, so as to understand whether the data is commonly used data, which provides a reference for subsequent determination of data processing priority. The stored hot data and cold data are aggregated to obtain aggregated data packets, and the data that affect each other are aggregated together. The aggregated data packets are processed to further improve the speed of data processing and save cloud computing resources. The corresponding grids are rendered in corresponding colors according to the priority of the aggregated data packets, and the aggregated data packets are processed according to the rendering results, so that the data processing is carried out in an orderly manner, ensuring that higher priority data is processed first and meeting the needs of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a user data processing optimization system for a cloud computing environment according to the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Example 1

[0069] See also Figure 1 As shown, the user data processing optimization system for a cloud computing environment described in this embodiment includes:

[0070] Data collection module, used to collect original user data;

[0071] A data preprocessing module, used to preprocess the original user data to obtain preprocessed user data;

[0072] A data storage module is used to analyze the pre-processed user data, obtain hot data and cold data, and store the hot data and cold data;

[0073] A data aggregation module is used to aggregate the stored hot data and cold data to obtain an aggregated data packet;

[0074] The optimization processing module is used to construct a grid space, map the aggregated data packet into the grid space, obtain the priority of the aggregated data packet, render the corresponding grid into a corresponding color according to the priority of the aggregated data packet, and process the aggregated data packet according to the rendering result.

[0075] The process of collecting raw user data includes:

[0076] The original user data includes structured data, unstructured data and semi-structured data;

[0077] The data categories included in the structured data are user account data, transaction order data and business log monitoring data; the data categories included in the unstructured data are document file data, media data and sensor Internet of Things data; the data categories included in the semi-structured data are social data and web crawler data;

[0078] The user account data includes user name, email, registration time, last login time, number of website visits, user identity verification information, etc. The transaction order data includes order number, purchase time, product information, payment amount, shopping cart information, order status (such as paid, shipped, completed, etc.), refund and after-sales records, etc. The business log monitoring data includes application operation logs (such as user login, operation records, etc.), audit logs, etc.;

[0079] The document file data includes text files, configuration files, log files, compressed package files, etc. The media data includes images, videos, audio, etc. The sensor IoT data includes data generated by smart homes, smart manufacturing equipment, vehicles, drones, etc.;

[0080] The social data includes user likes, comments, shared content, user interaction behaviors (such as clicks, dwell time, follow and unfollow records), etc. The web crawler data includes public data crawled from the Internet (such as news, blogs, forum posts, etc.), web page snapshots and content summaries generated during the data crawling process, etc.;

[0081] The original user data is obtained by calling the API interface, database query, web page crawling and message queue.

[0082] The original user data is preprocessed to obtain the preprocessed user data, including:

[0083] Determine whether there are outliers in the original user data. If there are outliers, remove the outliers in the original user data and use interpolation to fill in the outliers in the removed original user data to obtain preliminary user data. If there are no outliers, record the original user data as preliminary user data.

[0084] The process of determining whether there are outliers in the original user data includes:

[0085] Step A1: Randomly select a data category to build an isolation tree, randomly select a data in this data category as a cutting point, and divide all the data in q data categories into two parts according to the cutting point, that is, the data less than or equal to the cutting point is taken as one part, and the data greater than the cutting point is taken as the other part, and the left subtree and the right subtree are constructed respectively. Repeat recursively for the nodes of each subtree until the number of data in the node is less than or equal to 1, and stop. q is the number of data categories;

[0086] Step A2: Repeat step A1 to build q-1 isolation trees and calculate the deviation score of each data. i is the index of all data in q data categories, PL(i) is the deviation score of the i-th data, aj(i) is the average path length of the i-th data in all isolation trees, n is the number of all data in q data categories, and C(n) is the theoretical expected value of the path length;

[0087] in, r is the index of the isolation tree, a r (i) is the path length of the i-th data in the r-th isolation tree. The path length is the number of splits required from the cutting point to the node with only one data; H(n-1) is the harmonic number,

[0088] It is important to explain that the data categories selected are different each time the isolation tree is constructed;

[0089] Step A3: Preset a deviation score threshold PY. When the deviation score of the data is higher than the deviation score threshold PY, it is determined to be an abnormal value. Otherwise, it is determined to be a normal value.

[0090] The process of presetting the deviation score threshold includes:

[0091] Get m historical user data, get the historical deviation scores corresponding to the historical abnormal values ​​in the historical user data, and extract the minimum value PY in the historical deviation scores min and the maximum value PY max, construct a new threshold range:

[0092] Among them, k is the index of historical user data, PY kmin is the minimum value of the historical deviation score corresponding to the historical abnormal value in the k-th historical user data, PY kmax is the maximum value of the historical deviation scores corresponding to the historical abnormal values ​​in the k-th historical user data;

[0093] Obtain the number of occurrences of each historical deviation score, sort the historical deviation scores in descending order within the new threshold range according to the number of occurrences, and select the smallest historical deviation score among the first p_x historical deviation scores as the deviation score threshold;

[0094] Time-aligning data of different data categories in the preliminary user data to obtain pre-processed user data;

[0095] The process of temporally aligning data of different data categories in the preliminary user data includes:

[0096] For each data category, extract a timestamp sequence from the preliminary user data, obtain the time interval between adjacent timestamps in each timestamp sequence, compare the time intervals between adjacent timestamps in each timestamp sequence, and obtain the longest time interval;

[0097] If the time interval of the timestamp sequence is not the same as the longest time interval, it is recorded as the sequence to be converted, and the time intervals between adjacent timestamps in the sequence to be converted are converted to the longest time interval, all data between the current timestamp and the previous timestamp are collected, and the data at the current timestamp and the collected data are used as the data at the current timestamp;

[0098] For example: The timestamp sequence of sample data 1 is 2024-11-18 10:00:01, 2024-11-18 10:00:02, 2024-11-18 10:00:03, ..., 2024-11-18 10:02:00, and the sample data on the timestamp sequence of sample data 1 are sj1, sj2, sj3, ..., sj 120 ;

[0099] The timestamp sequence of sample data 2 is 2024-11-18 10:01, 2024-11-18 10:02, and the sample data on the timestamp sequence of sample data 1 is sr1, sr2;

[0100] Convert the time intervals between adjacent timestamps in the timestamp sequence of sample data 1 to the longest time interval. The timestamp sequence after conversion is 2024-11-18 10:01, 2024-11-18 10:02. The sample data on the timestamp sequence after conversion is [sj1, sj2, sj3, ..., sj 60 ],[sj 61 , sj 62 , sj 63 ,……,sj 120 ].

[0101] The process of analyzing the pre-processed user data, obtaining hot data and cold data, and storing the hot data and cold data includes:

[0102] Get the access time, access frequency and data size closest to the current time of the preprocessed user data, and obtain the cold and hot data discrimination formula based on the linear weighted summation method:

[0103] Where t is the time difference, t = t 当 -t 近 ;t 当 is the current time, t 近 is the most recent access time, w1, w2 and w3 are weight coefficients, f is the access frequency, the access frequency is the access frequency within one week from the current time, and s is the data size;

[0104] Set the discrimination threshold: LR0;

[0105] When LR>LR0, the pre-processed user data is recorded as hot data, and when LR≤LR0, the pre-processed user data is recorded as cold data;

[0106] For example, the most recent access time of the preprocessed user data from the current time is 10:00 on November 5, 2024, the access frequency of the preprocessed user data within a week from the current time is 50 times / week, the size of the preprocessed user data is 25MB, and the current time is 11:00 on November 6, 2024, then the time difference is 25 hours;

[0107] Assume that the weight coefficient w1 is set to 0.5, the weight coefficient w2 is set to 0.3, the weight coefficient w3 is set to 0.2, and the discrimination threshold LR0 is set to 0.60;

[0108] Input the access frequency, data size, and time difference of the pre-processed user data within one week from the current time into the cold and hot data discrimination formula to obtain the output result of the cold and hot data discrimination formula:

[0109] 0.65>0.60, this preprocessed user data is recorded as hot data;

[0110] The data storage module is provided with a hot data storage sub-interval and a cold data storage sub-interval, and stores hot data in the hot data storage sub-interval, and stores cold data in the cold data storage sub-interval.

[0111] The process of aggregating the stored hot data and cold data to obtain the aggregated data package includes:

[0112] Step B1: Obtain the results of the hot and cold data discrimination formula corresponding to the hot data and the cold data, obtain the maximum value of the results, and mark the hot data or cold data corresponding to the maximum value of the results as aggregated sub-data;

[0113] It should be explained that the hot data or cold data corresponding to the maximum value of the result is the most commonly used data. The most commonly used data is used as the aggregated sub-data, so that the commonly used data is processed first;

[0114] Step B2: Acquire hot data and cold data that change simultaneously with the change of aggregated sub-data, mark the hot data and cold data as related change data, package the aggregated sub-data and the related change data, and obtain an aggregated data packet;

[0115] Step B3: Obtain hot data and cold data that are not marked as aggregate sub-data and contact change data, and record them as unmarked data, extract the results of the hot and cold data discrimination formula corresponding to the unmarked data, and obtain the maximum value of the results, mark the unmarked data corresponding to the maximum value of the results as aggregate sub-data, and repeat step B2;

[0116] Step B4: Repeat step B3 until all hot data and cold data are marked as aggregated sub-data or contact change data.

[0117] The process of constructing a grid space, mapping the aggregated data packet into the grid space, obtaining the priority of the aggregated data packet, and rendering the corresponding grid into a corresponding color according to the priority of the aggregated data packet includes:

[0118] Construct a grid space of L rows and K columns, and map the aggregated data packets into the grid space. One grid can only map one aggregated data packet.

[0119] Get the latest processing time of each hot data and cold data, subtract the current time from the latest processing time, and obtain the processing urgency of the hot data and cold data;

[0120] Set a processing urgency threshold range. When the processing urgency of hot data and cold data is less than or equal to the minimum value in the processing urgency threshold range, the hot data and cold data are recorded as the highest priority data.

[0121] Get the aggregated data packet with the highest priority data, set the priority of this aggregated data packet to the highest priority, and render the grid where this aggregated data packet is located in red;

[0122] It should be explained that the highest priority data is the data that currently needs to be processed first, so the priority of the aggregated data packet containing the highest priority data is set to the highest priority;

[0123] When the processing urgency of hot data is greater than the minimum value in the processing urgency threshold range and less than the maximum value in the processing urgency threshold range, the hot data is recorded as hot first-level data; when the processing urgency of cold data is greater than the minimum value in the processing urgency threshold range and less than the maximum value in the processing urgency threshold range, the cold data is recorded as cold first-level data;

[0124] When the processing urgency of hot data is greater than or equal to the maximum value in the processing urgency threshold range, the hot data is recorded as hot secondary data; when the processing urgency of cold data is greater than or equal to the maximum value in the processing urgency threshold range, the cold data is recorded as cold secondary data;

[0125] According to the number of hot first-level data, cold first-level data, hot second-level data and cold second-level data, the priority determination formula of the remaining aggregated data packets is obtained: YX = g1×h(x1)+g2×d(x2)+g3×p(x3)+g4×y(x4);

[0126] Wherein, g1 is the number of hot primary data in the aggregated data packet, g2 is the number of cold primary data in the aggregated data packet, g3 is the number of hot secondary data in the aggregated data packet, g4 is the number of cold secondary data in the aggregated data packet, h(x1) is the hot primary data quantity influence function, d(x2) is the cold primary data quantity influence function, p(x3) is the hot secondary data quantity influence function, y(x4) is the cold secondary data quantity influence function, x1 is the independent variable of the hot primary data quantity influence function, x2 is the independent variable of the cold primary data quantity influence function, x3 is the independent variable of the hot secondary data quantity influence function, and x4 is the independent variable of the cold secondary data quantity influence function;

[0127] It should be explained that the input of the function affecting the number of hot first-level data is the number of hot first-level data, the input of the function affecting the number of cold first-level data is the number of cold first-level data, the input of the function affecting the number of hot second-level data is the number of hot second-level data, and the input of the function affecting the number of cold second-level data is the number of cold second-level data;

[0128] in,

[0129] Analyze historical user data to obtain historical hot data and historical cold data, aggregate the historical hot data and historical cold data to obtain a historical aggregated data packet, and obtain the maximum value of the output result of the priority discrimination formula for the historical aggregated data packet with the first-level priority and the minimum value of the output result of the priority discrimination formula for the historical aggregated data packet with the third-level priority;

[0130] Set the result threshold range (YX0, YX1) of the priority discrimination formula according to the minimum and maximum values of the output result of the priority discrimination formula for the historical aggregated data packet;

[0131] Among them, PJ_min is the minimum value of the output result, and PJ_max is the maximum value of the output result;

[0132] When YX ≤ YX0, set the priority of the aggregated data packet to the third-level priority;

[0133] When YX0 < YX < YX1, set the priority of the aggregated data packet to the second-level priority;

[0134] When YX ≥ YX1, set the priority of the aggregated data packet to the first-level priority;

[0135] Among them, YX0 is the minimum value within the result threshold range of the priority discrimination formula, and YX1 is the maximum value within the result threshold range of the priority discrimination formula;

[0136] When the priority of the aggregated data packet is the third-level priority, render the grid where this aggregated data packet is located green. When the priority of the aggregated data packet is the second-level priority, render the grid where this aggregated data packet is located yellow. When the priority of the aggregated data packet is the first-level priority, render the grid where this aggregated data packet is located orange;

[0137] The process of processing the aggregated data packet according to the rendering result includes:

[0138] Step C1: Obtain the number of red grids, green grids, yellow grids, and orange grids, and set the quantity threshold LS;

[0139] Step C2: When the number of red grids is greater than the quantity threshold, obtain the size of the aggregated data packets in the red grids and sort them in descending order, and process the first LS aggregated data packets in turn until all the aggregated data packets in the red grids are processed. When the number of red grids is less than or equal to the quantity threshold, directly process the aggregated data packets in the red grids;

[0140] Step C3: Repeat Step C2 to process the aggregated data packets in the orange grids;

[0141] Step C4: repeat step C2 to process the aggregated data packets in the yellow grid;

[0142] Step C5: Repeat step C2 to process the aggregated data packets in the green grid.

[0143] In this embodiment, the original user data is preprocessed to obtain preprocessed user data, and outliers in the original user data are removed, so that subsequent data processing is more accurate, and data of different data categories are time-aligned to make the data more orderly, which facilitates subsequent data processing, and the preprocessed user data is analyzed to obtain hot data and cold data, so as to understand whether the data is commonly used data, which provides a reference for subsequent determination of the priority of data processing, and the stored hot data and cold data are aggregated to obtain an aggregated data packet, and the data that affects each other are aggregated together, and the aggregated data packet is processed, which further improves the speed of data processing and saves cloud computing resources, and the corresponding grids are rendered into corresponding colors according to the priority of the aggregated data packet, and the aggregated data packet is processed according to the rendering result, so that the data processing is carried out in an orderly manner, ensuring that higher priority data is processed first, and meeting the needs of the system.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0145] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0147] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A user data processing optimization system for cloud computing environment, characterized in that: The user data processing optimization system for a cloud computing environment comprises: Data collection module, used to collect original user data; A data preprocessing module, used to preprocess the original user data to obtain preprocessed user data; A data storage module is used to analyze the pre-processed user data, obtain hot data and cold data, and store the hot data and cold data; A data aggregation module is used to aggregate the stored hot data and cold data to obtain an aggregated data packet; The optimization processing module is used to construct a grid space, map the aggregated data packet into the grid space, obtain the priority of the aggregated data packet, render the corresponding grid into a corresponding color according to the priority of the aggregated data packet, and process the aggregated data packet according to the rendering result.

2. The user data processing optimization system for cloud computing environment according to claim 1, characterized in that: The original user data includes structured data, unstructured data and semi-structured data; The data categories included in the structured data are user account data, transaction order data and business log monitoring data, the data categories included in the unstructured data are document file data, media data and sensor Internet of Things data, and the data categories included in the semi-structured data are social data and web crawler data.

3. The user data processing optimization system for cloud computing environment according to claim 2, characterized in that: The method of preprocessing the original user data to obtain the preprocessed user data includes: Determine whether there are outliers in the original user data. If there are outliers, remove the outliers in the original user data and use interpolation to fill in the outliers in the removed original user data to obtain preliminary user data. If there are no outliers, record the original user data as preliminary user data. Time-aligning data of different data categories in the preliminary user data to obtain pre-processed user data; The process of time alignment includes: For each data category, extract a timestamp sequence from the preliminary user data, obtain the time interval between adjacent timestamps in each timestamp sequence, compare the time intervals between adjacent timestamps in each timestamp sequence, and obtain the longest time interval; If the time interval of the timestamp sequence is not the same as the longest time interval, it is recorded as the sequence to be converted, and the time intervals between adjacent timestamps in the sequence to be converted are converted to the longest time interval. All data between the current timestamp and the previous timestamp are collected, and the data on the current timestamp and the collected data are used as the data on the current timestamp.

4. The user data processing optimization system for cloud computing environment according to claim 3 is characterized in that: The method for determining whether there are abnormal values ​​in the original user data includes: Step A1: Randomly select a data category to build an isolation tree, randomly select a data in this data category as a cutting point, divide all the data in q data categories into two parts according to the cutting point, build a left subtree and a right subtree respectively, and recursively repeat for each subtree node until the number of data in the node is less than or equal to 1, where q is the number of data categories; Step A2: Repeat step A1 to build q-1 isolation trees and calculate the deviation score of each data. i is the index of all data in q data categories, PL(i) is the deviation score of the i-th data, aj(i) is the average path length of the i-th data in all isolation trees, n is the number of all data in q data categories, and C(n) is the theoretical expected value of the path length; in, r is the index of the isolation tree, a r (i) is the path length of the i-th data in the r-th isolation tree. The path length is the number of splits required from the cutting point to the node with only one data; H(n-1) is the harmonic number, Step A3: Preset a deviation score threshold PY. When the deviation score of the data is higher than the deviation score threshold PY, it is determined to be an abnormal value, otherwise, it is determined to be a normal value.

5. The user data processing optimization system for cloud computing environment according to claim 4, characterized in that: The method for presetting the deviation score threshold comprises: Get m historical user data, get the historical deviation scores corresponding to the historical abnormal values ​​in the historical user data, and extract the minimum value PY in the historical deviation scores min and the maximum value PY max , construct a new threshold range as: Among them, k is the index of historical user data, PY kmin is the minimum value of the historical deviation score corresponding to the historical abnormal value in the k-th historical user data, PY kmax is the maximum value of the historical deviation scores corresponding to the historical abnormal values ​​in the k-th historical user data; Obtain the occurrence times of each historical deviation score, arrange the historical deviation scores in descending order within a new threshold range according to the occurrence times, and select the smallest historical deviation score among the top p_x historical deviation scores as the deviation score threshold.

6. The user data processing optimization system for cloud computing environment according to claim 5, characterized in that: The method for analyzing the preprocessed user data to obtain hot data and cold data and storing the hot data and cold data includes: Get the access time, access frequency and data size closest to the current time of the pre-processed user data, and obtain the cold and hot data discrimination formula based on the linear weighted summation method: Where t is the time difference, t = t 当 -t 近 ;t 当 is the current time, t 近 is the most recent access time, w1, w2 and w3 are weight coefficients, f is the access frequency, and s is the data size; Set the discrimination threshold: LR0; When LR > LR0, mark the preprocessed user data as hot data, and when LR ≤ LR0, mark the preprocessed user data as cold data; The data storage module is provided with a hot data storage sub-interval and a cold data storage sub-interval, and stores the hot data in the hot data storage sub-interval and the cold data in the cold data storage sub-interval.

7. The user data processing optimization system for cloud computing environment according to claim 6, characterized in that: The method for aggregating the stored hot data and cold data to obtain an aggregated data packet includes: Step B1: Obtain the results of the hot and cold data discrimination formulas corresponding to the hot data and cold data, and obtain the maximum value of the results. Mark the hot data or cold data corresponding to the maximum value of the results as aggregated sub-data; Step B2: Obtain the hot data and cold data that change simultaneously when the aggregated sub-data changes. Mark this hot data and cold data as associated change data, and package the aggregated sub-data and the associated change data to obtain an aggregated data packet; Step B3: Obtain the hot data and cold data that are not marked as aggregated sub-data and associated change data, and record them as unmarked data. Extract the results of the hot and cold data discrimination formulas corresponding to the unmarked data, and obtain the maximum value of the results. Mark the unmarked data corresponding to the maximum value of the results as aggregated sub-data, and repeat Step B2; Step B4: Repeat Step B3 until all hot data and cold data are marked as aggregated sub-data or associated change data.

8. The user data processing optimization system for cloud computing environment according to claim 7, characterized in that: The method for constructing a grid space, mapping the aggregated data packet into the grid space, obtaining the priority of the aggregated data packet, and rendering the corresponding grid as a corresponding color according to the priority of the aggregated data packet includes: Construct a grid space with L rows and K columns; Obtain the latest processing time of each hot data and cold data, obtain the processing urgency CJ of the hot data and cold data according to the latest processing time and the current time, and set the processing urgency threshold range (CJ0, CJ1); Where CJ = t 晚 -t 当 ;t 晚 The latest processing time; When CJ ≤ CJ0, mark this hot data and cold data as the highest priority data, obtain the aggregated data packet with the highest priority data, set the priority of this aggregated data packet to the highest priority, and render the grid where this aggregated data packet is located as red; When CJ0 < CJ < CJ1, mark this hot data as hot first-level data and this cold data as cold first-level data; When CJ ≥ CJ1, mark this hot data as hot second-level data and this cold data as cold second-level data; Among them, CJ0 is the minimum value within the processing urgency threshold range, and CJ1 is the maximum value within the processing urgency threshold range; Obtain the priority discrimination formula for the remaining aggregated data packets according to the quantities of hot first-level data, cold first-level data, hot second-level data, and cold second-level data: YX = g1×h(x1) + g2×d(x2) + g3×p(x3) + g4×y(x4); Among them, g1, g2, g3, and g4 are the quantities of hot primary data, cold primary data, hot secondary data, and cold secondary data in the aggregated data packet respectively. h(x1) is the hot primary data quantity influence function, d(x2) is the cold primary data quantity influence function, p(x3) is the hot secondary data quantity influence function, y(x4) is the cold secondary data quantity influence function, and x1, x2, x3, and x4 are the independent variables of the hot primary data quantity influence function, the independent variable of the cold primary data quantity influence function, the independent variable of the hot secondary data quantity influence function, and the independent variable of the cold secondary data quantity influence function respectively; in, The preset priority discrimination formula result threshold range (YX0, YX1); When YX ≤ YX0, set the priority of the aggregated data packet to the third-level priority, and render the grid where this aggregated data packet is located green; When YX0 < YX < YX1, set the priority of the aggregated data packet to the second-level priority, and render the grid where this aggregated data packet is located yellow; When YX ≥ YX1, set the priority of the aggregated data packet to the first-level priority, and render the grid where this aggregated data packet is located orange; Among them, YX0 is the minimum value within the priority discrimination formula result threshold range, and YX1 is the maximum value within the priority discrimination formula result threshold range.

9. The user data processing optimization system for cloud computing environment according to claim 8, characterized in that: The method for the preset priority discrimination formula result threshold range includes: Analyze historical user data to obtain historical hot data and historical cold data, aggregate the historical hot data and historical cold data to obtain historical aggregated data packets, and obtain the maximum value of the output result of the priority discrimination formula of the historical aggregated data packets with the first-level priority and the minimum value of the output result of the priority discrimination formula of the historical aggregated data packets with the third-level priority; Obtain the priority discrimination formula result threshold range (YX0, YX1) according to the minimum value and the maximum value of the output result; in, PJ_min is the minimum value of the output result, and PJ_max is the maximum value of the output result.

10. The user data processing optimization system for cloud computing environment according to claim 9, characterized in that: The method for processing the aggregated data packet according to the rendering result includes: Step C1: Obtain the quantities of red grids, green grids, yellow grids, and orange grids, and set the quantity threshold LS; Step C2: When the quantity of red grids is greater than the quantity threshold, obtain the sizes of the aggregated data packets in the red grids, sort them in descending order, and process the first LS aggregated data packets in turn until all the aggregated data packets in the red grids are processed. When the quantity of red grids is less than or equal to the quantity threshold, directly process the aggregated data packets in the red grids; Step C3: Repeat Step C2 to process the aggregated data packets in the orange grids; Step C4: Repeat Step C2 to process the aggregated data packets in the yellow grids; Step C5: Repeat Step C2 to process the aggregated data packets in the green grids.

Images