A user data processing optimization system for cloud computing environment

Abstract

The present invention belongs to the field of data processing technology, and discloses a user data processing optimization system for a cloud computing environment; the system comprises the following steps: collecting original user data; preprocessing the original user data to obtain preprocessed user data; analyzing the preprocessed user data to obtain hot data and cold data, and storing the hot data and cold data; aggregating the stored hot data and cold data to obtain an aggregated data packet; constructing a grid space, mapping the aggregated data packet into the grid space, obtaining the priority of the aggregated data packet, rendering the corresponding grid into a corresponding color according to the priority of the aggregated data packet, and processing the aggregated data packet according to the rendering result; processing mutually related data together, thereby further improving the data processing speed and saving cloud computing resources.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a user data processing optimization system for a cloud computing environment. Background Art

[0002] The patent application publication number CN110874268A discloses a data processing method, device and equipment, which includes: first obtaining the data to be processed and the corresponding operating status information of multiple data processing systems. The processing capacity of each data processing system is determined based on the obtained operating status information corresponding to each of the multiple data processing systems. The target data processing system corresponding to the first data block in the data to be processed is determined based on the processing capacity of each data processing system, and the first data block is sent to the target data processing system so that the target data processing system processes the first data block. By setting up multiple data processing systems, each data block can be handed over to the data processing system with the most suitable processing capacity for processing, so that the data block is processed in time, thereby ensuring that the data to be processed can be fully processed within the preset time. The user can obtain the data processing results corresponding to all the data to be processed, thereby improving the availability of the data processing results.

[0003] However, in the process of data processing, the amount of data is huge and chaotic. How to organize the data, obtain cold data and hot data, aggregate the data that affects each other, increase the speed of data processing, save cloud computing resources, and determine the priority of the data, render the corresponding grid into the corresponding color according to the priority of the data, process the data according to the rendering results, and make the data processing proceed in an orderly manner are the problems we need to solve. To this end, a user data processing optimization system for cloud computing environment is provided. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned objectives, the present invention provides the following technical solution: a user data processing optimization system for a cloud computing environment, comprising:

[0005] Data collection module, used to collect original user data;

[0006] A data preprocessing module is used to preprocess the original user data to obtain preprocessed user data;

[0007] A data storage module is used to analyze pre-processed user data, obtain hot data and cold data, and store the hot data and cold data;

[0008] The data aggregation module is used to aggregate the stored hot data and cold data to obtain aggregated data packets;

[0009] The optimization processing module is used to construct a grid space, map the aggregated data packet into the grid space, obtain the priority of the aggregated data packet, render the corresponding grid into the corresponding color according to the priority of the aggregated data packet, and process the aggregated data packet according to the rendering result.

[0010] Furthermore, the original user data includes structured data, unstructured data and semi-structured data;

[0011] The data categories included in the structured data are user account data, transaction order data and business log monitoring data; the data categories included in the unstructured data are document file data, media data and sensor Internet of Things data; the data categories included in the semi-structured data are social data and web crawler data.

[0012] Furthermore, the method of preprocessing the original user data to obtain preprocessed user data includes:

[0013] Determine whether there are outliers in the original user data. If there are outliers, remove the outliers from the original user data and use interpolation to fill in the outliers in the removed original user data to obtain preliminary user data. If there are no outliers, record the original user data as preliminary user data.

[0014] Time-aligning data of different data categories in the preliminary user data to obtain preprocessed user data;

[0015] The process of time alignment includes:

[0016] For each data category, extract a timestamp sequence from the preliminary user data, obtain the time intervals between adjacent timestamps in each timestamp sequence, compare the time intervals between adjacent timestamps in each timestamp sequence, and obtain the longest time interval;

[0017] If the time interval of the timestamp sequence is not the same as the longest time interval, it is recorded as the sequence to be converted. The time intervals between adjacent timestamps in the sequence to be converted are converted to the longest time interval. All data between the current timestamp and the previous timestamp are collected, and the data at the current timestamp and the collected data are used as the data at the current timestamp.

[0018] Furthermore, the method for determining whether there are outliers in the original user data includes:

[0019] Step A1: Randomly select a data category to build an isolation tree. Randomly select a data in this data category as a cut point. Divide all the data in q data categories into two parts based on the cut point, and build a left subtree and a right subtree respectively. Repeat recursively for each subtree node until the number of data in the node is less than or equal to 1. Stop, where q is the number of data categories.

[0020] Step A2: Repeat step A1 to build q-1 isolation trees and calculate the deviation score of each data. i is the index of all data in q data categories, PL(i) is the deviation score of the i-th data, aj(i) is the average path length of the i-th data in all isolation trees, n is the number of all data in q data categories, and C(n) is the theoretical expected value of the path length;

[0021] in, r is the index of the isolation tree, a r (i) is the path length of the i-th data in the r-th isolation tree. The path length is the number of splits required from the cutting point to the node with only one data; H(n-1) is the harmonic number,

[0022] Step A3: Preset a deviation score threshold PY. When the deviation score of the data is higher than the deviation score threshold PY, it is determined to be an abnormal value; otherwise, it is determined to be a normal value.

[0023] Furthermore, the method for presetting the deviation score threshold includes:

[0024] Get m historical user data, get the historical deviation scores corresponding to the historical abnormal values ​​in the historical user data, and extract the minimum value PY in the historical deviation scores min and the maximum value PY max , construct a new threshold range:

[0025] Among them, k is the index of historical user data, PY kmin is the minimum value of the historical deviation score corresponding to the historical abnormal value in the k-th historical user data, PY kmax is the maximum value of the historical deviation scores corresponding to the historical outliers in the k-th historical user data;

[0026] Obtain the number of occurrences of each historical deviation score, sort the historical deviation scores in descending order within the new threshold range according to the number of occurrences, and select the smallest historical deviation score among the first p_x historical deviation scores as the deviation score threshold.

[0027] Furthermore, the method of analyzing the pre-processed user data, obtaining hot data and cold data, and storing the hot data and cold data includes:

[0028] Obtain the most recent access time, access frequency, and data size of the pre-processed user data, and obtain the cold and hot data discrimination formula based on the linear weighted summation method:

[0029] Where t is the time difference, t=t 当 -t 近 ;t 当 is the current time, t 近 is the most recent access time, w1, w2 and w3 are weight coefficients, f is the access frequency, and s is the data size;

[0030] Set the discrimination threshold: LR0;

[0031] When LR>LR0, the pre-processed user data is recorded as hot data, and when LR≤LR0, the pre-processed user data is recorded as cold data;

[0032] The data storage module is provided with a hot data storage sub-interval and a cold data storage sub-interval, and stores hot data in the hot data storage sub-interval and cold data in the cold data storage sub-interval.

[0033] Furthermore, the method of aggregating the stored hot data and cold data to obtain an aggregated data packet includes:

[0034] Step B1: Obtain the results of the hot and cold data discrimination formula corresponding to the hot data and cold data, obtain the maximum value of the results, and mark the hot data or cold data corresponding to the maximum value of the results as aggregated sub-data;

[0035] Step B2: Acquire hot data and cold data that change simultaneously with the aggregate sub-data change, mark the hot data and cold data as related change data, and package the aggregate sub-data and the related change data to obtain an aggregate data packet;

[0036] Step B3: Obtain hot data and cold data that are not marked as aggregate sub-data and contact change data, and record them as unmarked data. Extract the results of the hot and cold data discrimination formula corresponding to the unmarked data, and obtain the maximum value of the results. Mark the unmarked data corresponding to the maximum value as aggregate sub-data, and repeat step B2.

[0037] Step B4: Repeat step B3 until all hot data and cold data are marked as aggregated sub-data or contact change data.

[0038] Further, the method of constructing a grid space, mapping the aggregated data packets into the grid space, obtaining the priorities of the aggregated data packets, and rendering the corresponding grids into corresponding colors according to the priorities of the aggregated data packets includes:

[0039] Construct a grid space with L rows and K columns;

[0040] Obtain the latest processing time of each hot data and cold data, obtain the processing urgency CJ of the hot data and cold data according to the latest processing time and the current time, and set the processing urgency threshold range (CJ0, CJ1);

[0041] where, CJ = t 晚 - t 当 ; t 晚 is the latest processing time;

[0042] When CJ ≤ CJ0, mark this hot data and cold data as the highest priority data, obtain the aggregated data packet with the highest priority data, set the priority of this aggregated data packet to the highest priority, and render the grid where this aggregated data packet is located red;

[0043] When CJ0 < CJ < CJ1, mark this hot data as hot first-level data and this cold data as cold first-level data;

[0044] When CJ ≥ CJ1, mark this hot data as hot second-level data and this cold data as cold second-level data;

[0045] where, CJ0 is the minimum value within the processing urgency threshold range, and CJ1 is the maximum value within the processing urgency threshold range;

[0046] Obtain the priority discrimination formula for the remaining aggregated data packets according to the quantities of hot first-level data, cold first-level data, hot second-level data, and cold second-level data: YX = g1 × h(x1) + g2 × d(x2) + g3 × p(x3) + g4 × y(x4);

[0047] where, g1, g2, g3, and g4 are respectively the quantities of hot first-level data, cold first-level data, hot second-level data, and cold second-level data in the aggregated data packet, h(x1) is the hot first-level data quantity influence function, d(x2) is the cold first-level data quantity influence function, p(x3) is the hot second-level data quantity influence function, y(x4) is the cold second-level data quantity influence function, and x1, x2, x3, and x4 are respectively the independent variables of the hot first-level data quantity influence function, the independent variable of the cold first-level data quantity influence function, the independent variable of the hot second-level data quantity influence function, and the independent variable of the cold second-level data quantity influence function;

[0048] where,

[0049] Preset priority discrimination formula result threshold range (YX 0, YX1);

[0050] When YX ≤ YX0, set the priority of the aggregated data packet to the third - level priority, and render the grid where this aggregated data packet is located green;

[0051] When YX0 < YX < YX1, set the priority of the aggregated data packet to the second - level priority, and render the grid where this aggregated data packet is located yellow;

[0052] When YX ≥ YX1, set the priority of the aggregated data packet to the first - level priority, and render the grid where this aggregated data packet is located orange;

[0053] Among them, YX0 is the minimum value within the priority discrimination formula result threshold range, and YX1 is the maximum value within the priority discrimination formula result threshold range.

[0054] Furthermore, the method for presetting the priority discrimination formula result threshold range includes:

[0055] Analyze historical user data to obtain historical hot data and historical cold data, aggregate the historical hot data and historical cold data to obtain historical aggregated data packets, and obtain the maximum value of the output result of the priority discrimination formula of the historical aggregated data packets with the first - level priority and the minimum value of the output result of the priority discrimination formula of the historical aggregated data packets with the third - level priority;

[0056] Obtain the priority discrimination formula result threshold range (YX 0, YX1) according to the minimum value and the maximum value of the output result;

[0057] Among them, PJ_min is the minimum value of the output result, and PJ_max is the maximum value of the output result.

[0058] Furthermore, the method for processing the aggregated data packet according to the rendering result includes:

[0059] Step C1: Obtain the number of red grids, green grids, yellow grids, and orange grids, and set the quantity threshold LS;

[0060] Step C2: When the number of red grids is greater than the quantity threshold, obtain the size of the aggregated data packets in the red grids and sort them in descending order, and process the first LS aggregated data packets in turn until all the aggregated data packets in the red grids are processed. When the number of red grids is less than or equal to the quantity threshold, directly process the aggregated data packets in the red grids;

[0061] Step C3: Repeat step C2 to process the aggregated data packets in the orange grid;

[0062] Step C4: Repeat step C2 to process the aggregated data packets in the yellow grid;

[0063] Step C5: Repeat step C2 to process the aggregated data packets in the green grid.

[0064] The technical effects and advantages of the user data processing optimization system for cloud computing environment of the present invention are as follows:

[0065] The original user data is preprocessed to obtain preprocessed user data, and outliers in the original user data are removed, making subsequent data processing more accurate. Data of different data categories are time-aligned to make the data more orderly, which facilitates subsequent data processing. The preprocessed user data is analyzed to obtain hot data and cold data, so as to understand whether the data is commonly used data, which provides a reference for subsequent determination of data processing priority. The stored hot data and cold data are aggregated to obtain aggregated data packets, and the data that affect each other are aggregated together. The aggregated data packets are processed, which further improves the speed of data processing and saves cloud computing resources. The corresponding grids are rendered in corresponding colors according to the priority of the aggregated data packets, and the aggregated data packets are processed according to the rendering results, so that the data processing is carried out in an orderly manner, ensuring that higher priority data is processed first, and meeting the needs of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of a user data processing optimization system for cloud computing environment according to the present invention. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0068] Example 1

[0069] See also Figure 1 As shown, the user data processing optimization system for cloud computing environment described in this embodiment includes:

[0070] Data collection module, used to collect original user data;

[0071] A data preprocessing module is used to preprocess the original user data to obtain preprocessed user data;

[0072] A data storage module is used to analyze pre-processed user data, obtain hot data and cold data, and store the hot data and cold data;

[0073] The data aggregation module is used to aggregate the stored hot data and cold data to obtain aggregated data packets;

[0074] The optimization processing module is used to construct a grid space, map the aggregated data packet into the grid space, obtain the priority of the aggregated data packet, render the corresponding grid into the corresponding color according to the priority of the aggregated data packet, and process the aggregated data packet according to the rendering result.

[0075] The process of collecting raw user data includes:

[0076] The original user data includes structured data, unstructured data and semi-structured data;

[0077] The data categories included in the structured data are user account data, transaction order data and business log monitoring data; the data categories included in the unstructured data are document file data, media data and sensor Internet of Things data; the data categories included in the semi-structured data are social data and web crawler data;

[0078] The user account data includes username, email address, registration time, last login time, number of website visits, user authentication information, etc. The transaction order data includes order number, purchase time, product information, payment amount, shopping cart information, order status (such as paid, shipped, completed, etc.), refund and after-sales records, etc. The business log monitoring data includes application operation logs (such as user login, operation records, etc.) and audit logs, etc.

[0079] The document file data includes text files, configuration files, log files, compressed package files, etc. The media data includes images, videos, audio, etc. The sensor IoT data includes data generated by smart homes, smart manufacturing equipment, vehicles, drones, etc.;

[0080] The social data includes user likes, comments, shared content, user interaction behaviors (such as clicks, dwell time, follow and unfollow records), etc. The web crawler data includes public data crawled from the Internet (such as news, blogs, forum posts, etc.), web page snapshots and content summaries generated during the data crawling process, etc.

[0081] Obtain original user data by calling API interfaces, database queries, web page crawling, and message queues.

[0082] The process of preprocessing the original user data to obtain the preprocessed user data includes:

[0083] Determine whether there are outliers in the original user data. If there are outliers, remove the outliers from the original user data and use interpolation to fill in the outliers in the removed original user data to obtain preliminary user data. If there are no outliers, record the original user data as preliminary user data.

[0084] The process of determining whether there are outliers in the original user data includes:

[0085] Step A1: Randomly select a data category to build an isolation tree. Randomly select a data in this data category as a cut point. Based on the cut point, divide all the data in q data categories into two parts, that is, the data less than or equal to the cut point is divided into one part, and the data greater than the cut point is divided into the other part. Build the left subtree and right subtree respectively. Repeat recursively for each subtree node until the number of data in the node is less than or equal to 1. Stop, q is the number of data categories.

[0086] Step A2: Repeat step A1 to build q-1 isolation trees and calculate the deviation score of each data. i is the index of all data in q data categories, PL(i) is the deviation score of the i-th data, aj(i) is the average path length of the i-th data in all isolation trees, n is the number of all data in q data categories, and C(n) is the theoretical expected value of the path length;

[0087] in, r is the index of the isolation tree, a r (i) is the path length of the i-th data in the r-th isolation tree. The path length is the number of splits required from the cutting point to the node with only one data; H(n-1) is the harmonic number,

[0088] It should be explained that the data categories selected are different each time the isolation tree is constructed;

[0089] Step A3: Preset a deviation score threshold PY. When the deviation score of the data is higher than the deviation score threshold PY, it is determined to be an abnormal value; otherwise, it is determined to be a normal value.

[0090] The process of presetting the deviation score threshold includes:

[0091] Get m historical user data, get the historical deviation scores corresponding to the historical abnormal values ​​in the historical user data, and extract the minimum value PY in the historical deviation scores min and the maximum value PY max, construct a new threshold range:

[0092] Among them, k is the index of historical user data, PY kmin is the minimum value of the historical deviation score corresponding to the historical abnormal value in the k-th historical user data, PY kmax is the maximum value of the historical deviation scores corresponding to the historical outliers in the k-th historical user data;

[0093] Get the number of occurrences of each historical deviation score, sort the historical deviation scores in descending order within the new threshold range according to the number of occurrences, and select the smallest historical deviation score among the first p_x historical deviation scores as the deviation score threshold;

[0094] Time-aligning data of different data categories in the preliminary user data to obtain preprocessed user data;

[0095] The process of temporally aligning data of different data categories in the preliminary user data includes:

[0096] For each data category, extract a timestamp sequence from the preliminary user data, obtain the time intervals between adjacent timestamps in each timestamp sequence, compare the time intervals between adjacent timestamps in each timestamp sequence, and obtain the longest time interval;

[0097] If the time interval of the timestamp sequence is different from the longest time interval, it is recorded as the sequence to be converted. The time intervals between adjacent timestamps in the sequence to be converted are converted to the longest time interval. All data between the current timestamp and the previous timestamp are collected. The data at the current timestamp and the collected data are used as the data at the current timestamp.

[0098] For example: The timestamp sequence of sample data 1 is 2024-11-18 10:00:01, 2024-11-18 10:00:02, 2024-11-18 10:00:03, ..., 2024-11-18 10:02:00, and the sample data on the timestamp sequence of sample data 1 are sj1, sj2, sj3, ..., sj 120 ;

[0099] The timestamp sequence of sample data 2 is 2024-11-18 10:01, 2024-11-18 10:02, and the sample data on the timestamp sequence of sample data 1 is sr1, sr2;

[0100] Convert the time intervals between adjacent timestamps in the timestamp sequence of sample data 1 to the longest time interval. The timestamp sequence after conversion is 2024-11-18 10:01, 2024-11-18 10:02. The sample data on the timestamp sequence after conversion is [sj1, sj2, sj3, ..., sj 60 ],[sj 61 , sj 62 , sj 63 ,……,sj 120 ].

[0101] The process of analyzing pre-processed user data, obtaining hot data and cold data, and storing the hot data and cold data includes:

[0102] Obtain the most recent access time, access frequency, and data size of the pre-processed user data, and obtain the cold and hot data discrimination formula based on the linear weighted summation method:

[0103] Where t is the time difference, t=t 当 -t 近 ;t 当 is the current time, t 近 is the most recent access time, w1, w2, and w3 are weight coefficients, f is the access frequency, which is the access frequency within one week from the current time, and s is the data size;

[0104] Set the discrimination threshold: LR0;

[0105] When LR>LR0, the pre-processed user data is recorded as hot data, and when LR≤LR0, the pre-processed user data is recorded as cold data;

[0106] For example, the most recent access time of the pre-processed user data is 10:00 on November 5, 2024. The access frequency of the pre-processed user data within a week is 50 times per week. The size of the pre-processed user data is 25 MB. The current time is 11:00 on November 6, 2024. Therefore, the time difference is 25 hours.

[0107] Assume that the weight coefficient w1 is set to 0.5, the weight coefficient w2 is set to 0.3, the weight coefficient w3 is set to 0.2, and the discrimination threshold LR0 is set to 0.60;

[0108] Input the access frequency, data size, and time difference of the pre-processed user data within one week from the current time into the hot and cold data discrimination formula to obtain the output result of the hot and cold data discrimination formula:

[0109] 0.65>0.60, this preprocessed user data is recorded as hot data;

[0110] The data storage module is provided with a hot data storage sub-interval and a cold data storage sub-interval, and stores hot data in the hot data storage sub-interval and stores cold data in the cold data storage sub-interval.

[0111] The process of aggregating stored hot and cold data to obtain aggregated data packets includes:

[0112] Step B1: Obtain the results of the hot and cold data discrimination formula corresponding to the hot data and cold data, obtain the maximum value of the results, and mark the hot data or cold data corresponding to the maximum value of the results as aggregated sub-data;

[0113] It should be explained that the hot data or cold data corresponding to the maximum value of the result is the most frequently used data. The most frequently used data is used as the aggregated sub-data, so that the frequently used data is processed first.

[0114] Step B2: Acquire hot data and cold data that change simultaneously with the aggregate sub-data change, mark the hot data and cold data as related change data, and package the aggregate sub-data and the related change data to obtain an aggregate data packet;

[0115] Step B3: Obtain hot data and cold data that are not marked as aggregate sub-data and contact change data, and record them as unmarked data. Extract the results of the hot and cold data discrimination formula corresponding to the unmarked data, and obtain the maximum value of the results. Mark the unmarked data corresponding to the maximum value as aggregate sub-data, and repeat step B2.

[0116] Step B4: Repeat step B3 until all hot data and cold data are marked as aggregated sub-data or contact change data.

[0117] The process of constructing a grid space, mapping the aggregated data packets into the grid space, obtaining the priority of the aggregated data packets, and rendering the corresponding grid into the corresponding color according to the priority of the aggregated data packets includes:

[0118] Construct a grid space of L rows and K columns, and map the aggregated data packets into the grid space. One grid can only map one aggregated data packet.

[0119] Get the latest processing time of each hot data and cold data, subtract the current time from the latest processing time, and obtain the processing urgency of the hot data and cold data;

[0120] Set a processing urgency threshold range. When the processing urgency of hot data and cold data is less than or equal to the minimum value in the processing urgency threshold range, the hot data and cold data are recorded as the highest priority data.

[0121] Obtain the aggregated data packet containing the highest priority data, set the priority of this aggregated data packet to the highest priority, and render the grid where this aggregated data packet is located in red;

[0122] It should be explained that the highest priority data is the data that currently needs to be processed first, so the priority of the aggregated data packet containing the highest priority data is set to the highest priority;

[0123] When the processing urgency of hot data is greater than the minimum value in the processing urgency threshold range and less than the maximum value in the processing urgency threshold range, the hot data is recorded as hot level 1 data; when the processing urgency of cold data is greater than the minimum value in the processing urgency threshold range and less than the maximum value in the processing urgency threshold range, the cold data is recorded as cold level 1 data;

[0124] When the processing urgency of hot data is greater than or equal to the maximum value in the processing urgency threshold range, the hot data is recorded as hot secondary data; when the processing urgency of cold data is greater than or equal to the maximum value in the processing urgency threshold range, the cold data is recorded as cold secondary data;

[0125] The priority determination formula for the remaining aggregated data packets is obtained based on the number of hot first-level data, cold first-level data, hot second-level data, and cold second-level data: YX = g1×h(x1)+g2×d(x2)+g3×p(x3)+g4×y(x4);

[0126] Among them, g1 is the number of hot first-level data in the aggregated data packet, g2 is the number of cold first-level data in the aggregated data packet, g3 is the number of hot second-level data in the aggregated data packet, g4 is the number of cold second-level data in the aggregated data packet, h(x1) is the influence function of the number of hot first-level data, d(x2) is the influence function of the number of cold first-level data, p(x3) is the influence function of the number of hot second-level data, y(x4) is the influence function of the number of cold second-level data, x1 is the independent variable of the influence function of the number of hot first-level data, x2 is the independent variable of the influence function of the number of cold first-level data, x3 is the independent variable of the influence function of the number of hot second-level data, and x4 is the independent variable of the influence function of the number of cold second-level data;

[0127] It should be explained that the input of the hot first-level data quantity influencing function is the number of hot first-level data, the input of the cold first-level data quantity influencing function is the number of cold first-level data, the input of the hot second-level data quantity influencing function is the number of hot second-level data, and the input of the cold second-level data quantity influencing function is the number of cold second-level data;

[0128] in,

[0129] Analyze historical user data to obtain historical hot data and historical cold data, aggregate the historical hot data and historical cold data to obtain a historical aggregated data packet, and obtain the maximum value of the output result of the priority discrimination formula for the historical aggregated data packet with the first-level priority and the minimum value of the output result of the priority discrimination formula for the historical aggregated data packet with the third-level priority;

[0130] Set the result threshold range (YX0, YX1) of the priority discrimination formula according to the minimum and maximum values of the output result of the priority discrimination formula for the historical aggregated data packet;

[0131] Among them, PJ_min is the minimum value of the output result, and PJ_max is the maximum value of the output result;

[0132] When YX ≤ YX0, set the priority of the aggregated data packet to the third-level priority;

[0133] [[ID=​​​​​​​​​​​​​​​​​​​​​​​Step C4: Repeat step C2 to process the aggregated data packets in the yellow grid;

[0142] Step C5: Repeat step C2 to process the aggregated data packets in the green grid.

[0143] In this embodiment, the original user data is preprocessed to obtain preprocessed user data, and outliers in the original user data are removed, thereby making subsequent data processing more accurate. Data of different data categories are time-aligned to make the data more orderly, which facilitates subsequent data processing. The preprocessed user data is analyzed to obtain hot data and cold data, thereby understanding whether the data is commonly used data, providing a reference for subsequent determination of the priority of data processing. The stored hot data and cold data are aggregated to obtain an aggregated data packet, and data that affects each other are aggregated together and processed, further improving the speed of data processing and saving cloud computing resources. The corresponding grids are rendered in corresponding colors according to the priority of the aggregated data packet, and the aggregated data packet is processed according to the rendering result, thereby making data processing orderly and ensuring that higher priority data is processed first, meeting the needs of the system.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0145] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only one type. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

[0147] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A user data processing optimization system for cloud computing environment, characterized in that: The user data processing optimization system for a cloud computing environment includes: Data collection module, used to collect original user data; A data preprocessing module is used to preprocess the original user data to obtain preprocessed user data; A data storage module is used to analyze pre-processed user data, obtain hot data and cold data, and store the hot data and cold data; The data aggregation module is used to aggregate the stored hot data and cold data to obtain aggregated data packets; An optimization processing module is used to construct a grid space, map the aggregated data packets into the grid space, obtain the priority of the aggregated data packets, render the corresponding grid into a corresponding color according to the priority of the aggregated data packets, and process the aggregated data packets according to the rendering results; The method of constructing a grid space, mapping the aggregated data packet into the grid space, obtaining the priority of the aggregated data packet, and rendering the corresponding grid into a corresponding color according to the priority of the aggregated data packet includes: Construct a grid space with L rows and K columns; Get the latest processing time of each hot data and cold data, and obtain the processing urgency of hot data and cold data based on the latest processing time and current time , and set the processing urgency threshold range ; in, ; is the current time, The latest processing time; when , record the hot data and cold data as the highest priority data, obtain the aggregate data packet containing the highest priority data, set the priority of the aggregate data packet to the highest priority, and render the grid where the aggregate data packet is located in red; when When , the hot data is recorded as hot first-level data, and the cold data is recorded as cold first-level data; when When , the hot data is recorded as hot secondary data, and the cold data is recorded as cold secondary data; in, is the minimum value within the processing urgency threshold range. It is the maximum value within the processing urgency threshold range; The priority determination formula for the remaining aggregated data packets is obtained based on the number of hot first-level data, cold first-level data, hot second-level data, and cold second-level data: ; in, 、 、 and are the number of hot primary data, cold primary data, hot secondary data, and cold secondary data in the aggregated data packet, respectively. is the impact function of the number of hot-level data, is the cold level 1 data quantity impact function, is the impact function of the number of hot secondary data, is the influence function of the number of cold secondary data, 、 、 and They are the independent variables of the hot first-level data quantity influence function, the independent variables of the cold first-level data quantity influence function, the independent variables of the hot second-level data quantity influence function, and the independent variables of the cold second-level data quantity influence function; in, ; ; ; ; Preset priority judgment formula result threshold range ; when , the priority of the aggregated data packet is set to the third level priority, and the grid where the aggregated data packet is located is rendered green; when , the priority of the aggregated data packet is set to the secondary priority, and the grid where the aggregated data packet is located is rendered yellow; when , the priority of the aggregated data packet is set to the first-level priority, and the grid where the aggregated data packet is located is rendered orange; in, is the minimum value within the threshold range of the priority discrimination formula result, The maximum value within the threshold range of the priority determination formula result.

2. The user data processing optimization system for cloud computing environment according to claim 1, characterized in that: The original user data includes structured data, unstructured data and semi-structured data; The data categories included in the structured data are user account data, transaction order data and business log monitoring data; the data categories included in the unstructured data are document file data, media data and sensor Internet of Things data; the data categories included in the semi-structured data are social data and web crawler data.

3. The user data processing optimization system for cloud computing environment according to claim 2, characterized in that: The method of preprocessing the original user data to obtain preprocessed user data includes: Determine whether there are outliers in the original user data. If there are outliers, remove the outliers from the original user data and use interpolation to fill in the outliers in the removed original user data to obtain preliminary user data. If there are no outliers, record the original user data as preliminary user data. Time-aligning data of different data categories in the preliminary user data to obtain preprocessed user data; The process of time alignment includes: For each data category, extract a timestamp sequence from the preliminary user data, obtain the time intervals between adjacent timestamps in each timestamp sequence, compare the time intervals between adjacent timestamps in each timestamp sequence, and obtain the longest time interval; If the time interval of the timestamp sequence is not the same as the longest time interval, it is recorded as the sequence to be converted. The time intervals between adjacent timestamps in the sequence to be converted are converted to the longest time interval. All data between the current timestamp and the previous timestamp are collected, and the data at the current timestamp and the collected data are used as the data at the current timestamp.

4. The user data processing optimization system for cloud computing environment according to claim 3, characterized in that: The method for determining whether there are outliers in the original user data includes: Step A1: Randomly select a data category to build an isolation tree, randomly select a data in this data category as a cutting point, and All data in a data category are divided into two parts, and the left subtree and the right subtree are constructed respectively. The nodes of each subtree are recursively repeated until the number of data in the node is less than or equal to Stop when is the number of data categories; Step A2: Repeat step A1 to build Isolation tree, calculate the deviation score of each data, ; for The index of all data in the data category, For the The deviation score of the data, For the The average path length of data in all isolation trees, for The number of all data in the data category, is the theoretical expected value of the path length; in, ; is the index of the isolation tree, For the The data in The path length in an isolation tree is the number of splits required to get from the cut point to the node with only one data; ; is the harmonic number, ; Step A3: Preset deviation score threshold , when the deviation score of the data is higher than the deviation score threshold , it is judged as an abnormal value; otherwise, it is judged as a normal value.

5. The user data processing optimization system for cloud computing environment according to claim 4, characterized in that: The method for presetting the deviation score threshold includes: Get Historical user data, and obtain the historical deviation scores corresponding to the historical abnormal values ​​in the historical user data, and extract the minimum value of the historical deviation scores and maximum value , construct a new threshold range: ; in, is the index of historical user data, For the The minimum value of the historical deviation score corresponding to the historical outliers in the historical user data, For the The maximum value of the historical deviation scores corresponding to the historical outliers in the historical user data; Obtain the number of occurrences of each historical deviation score, sort the historical deviation scores in descending order within the new threshold range according to the number of occurrences, and select the smallest historical deviation score among the first p_x historical deviation scores as the deviation score threshold.

6. The user data processing optimization system for cloud computing environment according to claim 5, characterized in that: The method of analyzing the pre-processed user data to obtain hot data and cold data, and storing the hot data and cold data includes: Obtain the most recent access time, access frequency, and data size of the pre-processed user data, and obtain the cold and hot data discrimination formula based on the linear weighted summation method: ; in, is the time difference, ; is the current time, The most recent access time, 、 and is the weight coefficient, is the access frequency, is the data size; Set the discrimination threshold: ; when When , the pre-processed user data is recorded as hot data. When , the pre-processed user data is recorded as cold data; The data storage module is provided with a hot data storage sub-interval and a cold data storage sub-interval, and stores hot data in the hot data storage sub-interval and cold data in the cold data storage sub-interval.

7. The user data processing optimization system for cloud computing environment according to claim 6, characterized in that: The method of aggregating the stored hot data and cold data to obtain an aggregated data packet includes: Step B1: Obtain the results of the hot and cold data discrimination formula corresponding to the hot data and cold data, obtain the maximum value of the results, and mark the hot data or cold data corresponding to the maximum value of the results as aggregated sub-data; Step B2: Acquire hot data and cold data that change simultaneously with the aggregate sub-data change, mark the hot data and cold data as related change data, and package the aggregate sub-data and the related change data to obtain an aggregate data packet; Step B3: Obtain hot data and cold data that are not marked as aggregate sub-data and contact change data, and record them as unmarked data. Extract the results of the hot and cold data discrimination formula corresponding to the unmarked data, and obtain the maximum value of the results. Mark the unmarked data corresponding to the maximum value as aggregate sub-data, and repeat step B2. Step B4: Repeat step B3 until all hot data and cold data are marked as aggregated sub-data or contact change data.

8. The user data processing optimization system for cloud computing environment according to claim 7, characterized in that: The method for presetting the threshold range of the priority determination formula result includes: Analyze historical user data to obtain historical hot data and historical cold data, aggregate the historical hot data and historical cold data to obtain historical aggregated data packets, and obtain the maximum value of the output result of the priority discrimination formula for the historical aggregated data packets of the first-level priority and the minimum value of the output result of the priority discrimination formula for the historical aggregated data packets of the third-level priority; Obtain the threshold range of the priority judgment formula result based on the minimum and maximum values ​​of the output results ; in, ; ; is the minimum value of the output result, The maximum value of the output result.

9. The user data processing optimization system for cloud computing environment according to claim 8, characterized in that: The method for processing the aggregated data packet according to the rendering result includes: Step C1: Obtain the number of red grids, green grids, yellow grids, and orange grids, and set a number threshold LS; Step C2: When the number of red grids is greater than the threshold, obtain the size of the aggregated data packets in the red grids, sort them in descending order, and process the first LS aggregated data packets in sequence until all the aggregated data packets in the red grids are processed. When the number of red grids is less than or equal to the threshold, directly process the aggregated data packets in the red grids. Step C3: Repeat step C2 to process the aggregated data packets in the orange grid; Step C4: Repeat step C2 to process the aggregated data packets in the yellow grid; Step C5: Repeat step C2 to process the aggregated data packets in the green grid.

Images