A cloud computing platform and method based on data mining and big data analysis

Abstract

The application discloses a cloud computing platform and method based on data mining and big data analysis, and relates to the technical field of cloud storage data processing, and comprises the following steps: constructing a cold data set based on data object writing time and cabinet height; constructing a node temperature profile reflecting vertical thermal stratification by using storage node temperature monitoring data; calculating the risk of data object maintenance and attenuation by combining the residence time and error correction statistical data; calculating the strip strength based on the statistical distribution difference of the risk in the height direction; generating a trusted weight according to the risk and the strip strength, and then performing weighted data analysis and correcting the search order. The application can improve the consistency of the robustness and the reliability of the data mining analysis result and the search order result.

Description

Technical Field

[0001] This invention relates to the field of cloud storage data processing technology, and in particular to a cloud computing platform and method based on data mining and big data analysis. Background Technology

[0002] With the development of cloud computing and big data technologies, cloud platforms support the storage, offline analysis, statistical aggregation, and retrieval services of massive amounts of data. In actual data center deployments, to improve space utilization, physical servers are often densely stacked in racks, resulting in systematic differences in intake air temperature along the vertical direction of the racks. Nodes at different heights experience inconsistent long-term thermal environments. At the same time, there is a large amount of long-term resident, rarely modified, cold data. The storage media on which this data depends, such as solid-state drives or hard disks, will naturally decay over time during long periods of inactivity, and this decay process is extremely sensitive to temperature. Under the combined effect of high temperature and long-term residence, the error correction pressure on the storage media will increase significantly, leading to fluctuations in the physical reliability of the data.

[0003] However, existing cloud computing platforms, when performing data mining, statistical analysis, or building retrieval indexes, typically treat data reading errors, verification anomalies, or quality degradation as approximately random disturbances, or assume that the physical reliability of all sample data is spatially uniform, ignoring the reliability variations of long-term resident cold data at different rack heights due to varying thermal environments. This neglect leads to data in high-risk areas being given excessive trust during big data analysis, resulting in location-biased distortions in statistical results. In retrieval services, data objects whose underlying media is nearing degradation may be returned to users, causing the ranking results to lack consideration for physical security and reducing the system's robustness. Summary of the Invention

[0004] The purpose of this invention is to address the problems in existing technologies that ignore the striping effect of vertical thermal stratification of data center racks on maintaining the reliability of cold data, leading to positional distortion in data mining and statistical analysis, and the inability of retrieval and sorting results to effectively perceive and avoid the risk of degradation of the underlying storage media. Therefore, this invention proposes a cloud computing platform and method based on data mining and big data analysis.

[0005] To address the problems existing in the prior art, the present invention adopts the following technical solution:

[0006] A cloud computing platform and method based on data mining and big data analysis, comprising:

[0007] S1. Obtain the write time, rack height position, and error correction statistics of the data object, use the difference between the current time and the write time as the dwell time, and construct a cold data set based on the write time;

[0008] S2. Construct a node temperature profile based on storage node temperature monitoring data and rack height location;

[0009] S3. Based on node temperature profiles, dwell time, and error correction statistics, calculate the retention decay risk of data objects in cold datasets.

[0010] S4. Calculate the strip strength based on the statistical distribution difference of the attenuation risk in the cabinet height direction, and associate the strip strength with the corresponding data object;

[0011] S5. Calculate the confidence weight based on the preserving attenuation risk and the strip strength, perform weighted data analysis based on the confidence weight, and correct the search ranking based on the confidence weight.

[0012] Preferably, constructing a cold data set includes:

[0013] Record the storage node identifier, rack height, write time, and error correction statistics for each data object. The error correction statistics are the error correction event count or error correction byte count of the data object within a unit of time.

[0014] The median dwell time of all data objects is calculated, and data objects with a dwell time not less than the median are selected to form a cold data set.

[0015] Preferably, a node temperature profile is constructed based on storage node temperature monitoring data and rack height, including:

[0016] Collect the intake air temperature sequence of each storage node, and take the median of the intake air temperature sequence within a preset time window to obtain a robust temperature characterization.

[0017] Temperature gradients are obtained by linear fitting based on cabinet height and robust temperature characterization.

[0018] The temperature at different heights in the vertical direction of the cabinet is corrected based on the temperature gradient to obtain the node temperature profile of each storage node.

[0019] Preferably, based on node temperature profiles, residence time, and error correction statistics, the retention decay risk of data objects in the cold data set is calculated, including:

[0020] The median of all node temperature profiles within the rack is used as the reference temperature. The temperature acceleration factor is calculated based on the reference temperature and the node temperature profiles. The formula for calculating the temperature acceleration factor is as follows:

[0021]

[0022] In the formula, As a temperature acceleration factor, This is the nodal temperature profile. For reference temperature, This is a preset constant;

[0023] The correction coefficient is determined based on the error correction statistics. The retention decay risk of the data object is obtained by multiplying the residence time, temperature acceleration factor and correction coefficient.

[0024] Preferably, the strip strength is calculated based on the statistical distribution difference of the attenuation risk along the rack height direction, including:

[0025] Divide data objects into height buckets based on the height of the server rack;

[0026] Calculate the median risk of attenuation within each bucket height and use it as the median risk within the bucket.

[0027] Calculate the median risk of decay for all data objects in the cold data set, and use it as the global risk median;

[0028] The ratio of the median risk within a bucket to the median risk globally is used as the strip strength of the height bucket, and the strip strength is assigned to the data object within the height bucket.

[0029] Preferably, the formula for calculating the confidence weight is:

[0030]

[0031] In the formula, For reliable weights, To mitigate the risk of degradation, The value represents the strip strength.

[0032] Preferably, weighted data analysis based on the confidence weights includes:

[0033] In data mining tasks, loss targets are weighted according to credibility weights in order to minimize the weighted loss target;

[0034] In statistical aggregation tasks, a weighted average is calculated for the statistical values ​​of data objects based on their credibility weights.

[0035] Preferably, adjusting the search ranking based on the trusted weight includes:

[0036] The credibility weights are normalized according to the maximum credibility weight among all data objects to obtain the normalized credibility.

[0037] The normalized confidence score is written as an extended field into the inverted index or vector index of the data object. In response to the retrieval request, the basic relevance score calculated based on the query is obtained. The corrected ranking score is obtained by multiplying the basic relevance score and the normalized confidence score.

[0038] A cloud computing platform based on data mining and big data analytics includes:

[0039] The data management module is used to obtain the write time, rack height position, and error correction statistics of data objects, and uses the difference between the current time and the write time as the dwell time to construct a cold data set based on the write time.

[0040] The temperature profile construction module is used to construct node temperature profiles based on storage node temperature monitoring data and rack height location.

[0041] The risk calculation module is used to calculate the retention decay risk of data objects in the cold data set by combining the node temperature profile and the residence time.

[0042] The strip analysis module calculates the strip strength based on the statistical distribution difference of attenuation risk along the height of the cabinet, and associates the strip strength with the corresponding data object;

[0043] The analysis and retrieval service module is used to calculate a reliable weight based on the preserving attenuation risk and the band strength, perform weighted data analysis based on the reliable weight, and correct the retrieval ranking based on the reliable weight.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. This invention constructs a node temperature profile by collecting the intake air temperature sequence of storage nodes and combining it with the height position of the rack. It uses temperature gradients to correct the node temperature characterization and further calculates the retention decay risk by combining the residence time of data objects with error correction statistics. By introducing strip intensity, it quantifies the spatial aggregation characteristics of risk in the vertical direction of the rack. Based on this, a reliable weight is constructed that can dynamically reflect the physical reliability of data objects affected by vertical thermal stratification. In data mining and statistical aggregation tasks, this weight is used to reduce the weight of data objects in high-risk stripes, effectively suppressing the positional bias distortion caused by the long-term residence of cold data and the superposition of temperature stratification, and improving the accuracy of big data analysis conclusions.

[0046] 2. This invention normalizes the trust weight and writes it as an extended field into the inverted index or vector index of the data object, realizing the transparent transmission of physical side reliability information to the logical side retrieval service. When responding to retrieval requests, the basic relevance score is integrated with the normalized trust score to correct the ranking results. This causes data objects in high-temperature and high-risk areas and whose media conditions are on the verge of deterioration to be automatically downgraded in the ranking. Thus, without changing the basic retrieval algorithm, the physical security and stability of the retrieval results are ensured, and data with high relevance but low reliability is not returned first. This enhances the robustness of the cloud computing platform when serving massive amounts of data. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart illustrating a cloud computing method based on data mining and big data analysis according to the present invention.

[0049] Figure 2 This is a functional block diagram of a cloud computing platform based on data mining and big data analysis according to the present invention;

[0050] Figure 3 This is a schematic diagram of the vertical thermal stratification of the cabinet according to the present invention;

[0051] Figure 4 This is a schematic diagram of the node temperature profile of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0053] Example: This example provides a cloud computing method based on data mining and big data analysis. See [link to example]. Figure 1 Specifically, including:

[0054] S1. Obtain the write time, rack height position, and error correction statistics of the data object, and construct a cold data set based on the write time;

[0055] In embodiments of the present invention, the write time, rack height position, and error correction statistics of the data object are obtained, and a cold data set is constructed based on the write time, including:

[0056] Record the storage node identifier, rack height, write time, and error correction statistics for each data object;

[0057] Calculate the difference between the current time and the writing time to obtain the dwell time, count the median dwell time of all data objects, and select data objects with a dwell time not less than the median to form a cold data set;

[0058] It should be noted that a data object refers to the smallest independently manageable and accessible data unit treated by the cloud computing platform. Physically, it corresponds to a continuous or discrete storage area in a solid-state drive or disk drive, along with its associated index entries. The storage node identifier is a unique identifier for the physical server or storage device hosting the data object, enabling the platform to establish a definite mapping between the data object and its specific hardware location. The write time refers to the point in time recorded by the storage controller or system clock when the data object was most recently written to or modified to the physical medium, used to characterize the charge retention and media aging process experienced by the data object since the last write. Error correction system... Count data refers to the count or byte count generated by the error correction mechanism of the storage device when it detects and repairs correctable errors that occur in the storage unit corresponding to the data object during the reading or verification process. It is used to characterize the intensity of bit errors in the medium and the degree of charge drift accumulation in the storage unit. Residence time is a measure of the length of time that a data object is held on the medium without being rewritten. The larger the residence time, the more sufficient the retention decay accumulation is and the easier it is to be reflected in the increase of the error correction count. The cold data set refers to the set of data objects with a residence time not less than the median. It corresponds to the subset of data that has not been refreshed on the physical medium for a long time and is more likely to be exposed to the effects of retention decay and temperature stratification.

[0059] Specifically, the cloud computing platform generates or reads a unique object identifier for each data object upon completion of data object entry or writing, and obtains the storage node identifier carrying the data object from the distributed storage metadata service. Simultaneously, it reads the rack height location code of the storage node in the rack asset ledger and writes it into the object metadata entry. The writing time is recorded and solidified into the object metadata field by the storage system using a unified time base when the write submission is successful. The unified time base preferably uses a monotonically increasing time source provided by the cluster's internal time synchronization service to avoid negative dwell time or abnormal jumps caused by clock rollback. Error correction statistics are generated by the storage device's error correction mechanism or storage driver during read verification and background inspection and are aggregated at the data object granularity. The preferred aggregation method is to accumulate the error correction count or error correction byte count within a unit time window within the object's physical block address range and write it into the object metadata field. The time window is preferably one hour to one day to balance statistical stability and update overhead, and can be selected based on the error fluctuation cycle of the storage medium.

[0060] When the platform needs to construct a cold data set, it reads the write times of all data objects in batches from the object metadata and subtracts the write times from the current time to obtain the residence time. The current time is taken from the unified time base and kept consistent in the same batch calculation to eliminate intra-batch bias. If a missing write time or a time later than the current time is detected, the data object is marked as metadata abnormal and removed from the current round of filtering to ensure the validity of the statistics. Then, the residence times of all valid data objects are written into the residence time list and the median is calculated using a selection algorithm or a sorting algorithm. The calculation preferably uses a linear time selection algorithm to reduce the computational overhead in large-scale object scenarios or uses external sorting to adapt to the residence time list that exceeds the memory capacity. The obtained median is used as the boundary benchmark for distribution adaptation. The platform compares the residence time of each data object with the median one by one and filters data objects whose residence time is not less than the median. The identifier of the data object that meets the conditions, along with its storage node identifier and rack height position, is written into the cold data set list or set index to form the input dataset for subsequent attenuation strip analysis and weighted mining processing.

[0061] S2. Construct a node temperature profile based on storage node temperature monitoring data and rack height location;

[0062] In an embodiment of the present invention, a node temperature profile is constructed based on storage node temperature monitoring data and rack height location, including:

[0063] Collect the intake air temperature sequence of each storage node, and take the median of the intake air temperature sequence within a preset time window to obtain a robust temperature characterization.

[0064] Temperature gradients are obtained by linear fitting based on cabinet height and robust temperature characterization.

[0065] It should be noted that the temperature gradient is a value used to characterize the intensity of vertical thermal stratification in a server rack. It reflects the rate at which the intake air temperature changes with the height of the rack. Specifically, it represents the increment of robust temperature characterization for each unit increase in rack height. A positive temperature gradient indicates that the intake air temperature at the top of the rack is relatively higher than that at the bottom, and the thermal stratification is rising upwards. A negative temperature gradient indicates that the intake air temperature at the top of the rack is relatively lower than that at the bottom, and the thermal stratification is falling upwards. The larger the absolute value of the temperature gradient, the more significant the difference in intake air temperature between different height positions within the same rack. This means that when cold data resides at different height nodes, its risk of attenuation is more likely to exhibit striping differences along the height direction.

[0066] Specifically, the cloud computing platform deploys temperature acquisition points on the air intake side of each storage node or utilizes the node's built-in air intake temperature sensor interface to continuously sample the air intake temperature at a fixed sampling period, forming an air intake temperature sequence. The sampling period is preferably one to ten minutes to cover short-cycle changes caused by fluctuations in the data center's air supply and fan speed adjustments, without significantly increasing the acquisition and reporting overhead. The platform writes the air intake temperature sequences reported by each storage node into the time-series data storage according to the node identifier and includes a sampling timestamp. Subsequently, it extracts the corresponding interval of the air intake temperature sequence from the time-series data storage using a preset time window. For calculating robust temperature characterization, the preset time window is preferably 12 to 72 hours to reduce the impact of instantaneous thermal disturbances and anomalies caused by operation and maintenance, and to reflect the long-term stable structure of vertical thermal stratification. For the intake air temperature sequence of each storage node within the preset time window, outlier removal and missing value imputation are performed to ensure the reliability of the statistics. Outlier removal preferably uses upper and lower bounds based on interquartile range to suppress the interference of occasional peak temperatures on the statistical results. Missing value imputation preferably uses linear interpolation of adjacent sampling points or keeps the most recent valid value to maintain the continuity of the time series.

[0067] After preprocessing, the median of the inlet air temperature sequence of each node is taken as a robust temperature characterization. This robust temperature characterization is then associated with the rack height position corresponding to the storage node. After obtaining the rack height position and robust temperature characterization of each storage node in the rack, a linear fitting sample set is constructed with the rack height position as the independent variable and the robust temperature characterization as the dependent variable. The slope of the fitted line is then calculated using the least squares method as the temperature gradient. During the calculation process, each sample point is preferably assigned a weight proportional to the number of valid sampling points in its temperature sequence to weaken the adverse effect of missing sampling nodes on gradient estimation. At the same time, iterative weight reduction of robust regression is performed on nodes with excessively large fitting residuals to suppress the influence of outliers caused by local backflow or sensor drift. This results in a temperature gradient that can characterize the intensity of vertical thermal stratification in the rack and serves as the input for subsequent node temperature profile construction.

[0068] The temperature at different heights in the vertical direction of the cabinet is corrected based on the temperature gradient to obtain the node temperature profile of each storage node.

[0069] It should be noted that the node temperature profile is used to characterize the temperature level of a storage node under the vertical thermal stratification of the rack. It is not an instantaneous temperature reading, but a result obtained by using a robust temperature characterization as a benchmark and correcting for the height position of the node in the rack by combining the temperature gradient. The node temperature profile can map the temperature change pattern of the rack from bottom to top to each specific storage node. When calculating the risk of cold data retention decay, the temperature level corresponding to the node can be directly used to reflect the differences in its long-term thermal environment. This can be used to characterize the striped differences that may occur when cold data resides on nodes at different heights.

[0070] Specifically, after obtaining the temperature gradient of each storage node within the rack, the rack height position of each storage node participating in gradient estimation is first uniformly calibrated to ensure that the rack height positions use the same scale and correspond one-to-one with the height scale of the asset ledger, thus guaranteeing comparability between different vertical height positions. A robust reference temperature is calculated from the robust temperature characterization of all storage nodes within the rack, and the reference height corresponding to the robust reference temperature is used as the benchmark point for profile correction. The robust reference temperature is preferably the median of the robust temperature characterization within the rack to reduce the impact of local backflow and sensor drift on the benchmark. The reference height is preferably the median of the rack height positions within the rack so that the correction is spread around the center of the rack, thereby reducing extrapolation errors at the top and bottom. Based on this, for each storage node, the temperature correction amount in the vertical direction is obtained by multiplying the difference between its rack height position and the reference height by the temperature gradient. This correction amount is then superimposed on the robust reference temperature to obtain the node temperature profile of that storage node. The calculation formula for the node temperature profile is as follows:

[0071]

[0072] In the formula, This is the node temperature profile of the storage node. For a robust reference temperature, For temperature gradient, This refers to the rack height of the storage node. The reference height of the storage node is defined by the temperature gradient, which is obtained by linear fitting and characterizes the rate of temperature change per unit height. Therefore, multiplying the height offset by the temperature gradient yields the temperature offset relative to the reference. Adding this to the reference temperature gives the profile temperature that reflects vertical thermal stratification. This method ensures that when the storage node is at the reference height, its node temperature profile is equal to the robust reference temperature and avoids the overall temperature baseline drift from masking the profile differences.

[0073] S3. Based on node temperature profiles, dwell time, and error correction statistics, calculate the retention decay risk of data objects in cold datasets.

[0074] In embodiments of the present invention, the retention decay risk of data objects in a cold dataset is calculated based on node temperature profiles, dwell time, and error correction statistics, including:

[0075] The median of all node temperature profiles within the cabinet is used as the reference temperature, and the temperature acceleration factor is calculated based on the reference temperature and the node temperature profiles.

[0076] Specifically, after calculating the node temperature profiles of each storage node within the rack, all node temperature profiles within the same rack are aggregated into a temperature profile list, and a consistency check is performed to remove missing values ​​and obvious outliers. This avoids sensor malfunctions or data collection interruptions causing deviations in the statistical baseline. The median of the temperature profile list is taken as the reference temperature. The reference temperature can stably characterize the typical temperature level of the rack and serve as a unified benchmark for temperature acceleration. After obtaining the reference temperature, the temperature difference between the node temperature profile and the reference temperature is calculated for each storage node, and this temperature difference is mapped to a temperature acceleration factor to characterize the trend of accelerated decay as temperature rises. The formula for calculating the temperature acceleration factor is as follows:

[0077]

[0078] In the formula, As a temperature acceleration factor, This is the nodal temperature profile. For reference temperature, The value is a preset constant, preferably 10, which corresponds to the commonly used Q10 expression. This value makes the acceleration factor approximately double for every ten degrees Celsius increase in temperature, thus establishing a monotonic mapping relationship between temperature and the rate of decay. After the calculation is completed, the platform writes the temperature acceleration factor of each storage node into the node metadata table.

[0079] The correction coefficient is determined based on the error correction statistics. The retention decay risk of the data object is obtained by multiplying the residence time, temperature acceleration factor and correction coefficient.

[0080] It should be noted that retention decay risk is used to characterize the degree of information retention capability decline of data objects on the storage medium due to prolonged periods without being rewritten. It reflects the cumulative effect of the gradual drift of the charge or magnetization state representing the data state within the storage cell over time and the accelerated change under temperature, making it easier for correctable or even uncorrectable errors to occur during reading. Retention decay risk comprehensively considers the cumulative time represented by the residence time of the data object since the last write, the accelerated effect corresponding to the temperature environment of the storage node where the data object is located, and the actual error density and repair pressure revealed by error correction statistics. It can characterize the possibility of striped reliability degradation of cold data objects under different rack heights and residence conditions with a comparable value.

[0081] Specifically, for each data object within the cold data set, its residence time is read from the object metadata, and the temperature acceleration factor corresponding to the storage node is read based on the storage node identifier of the data object. Simultaneously, error correction statistics for the data object within a preset statistical window are read to reflect the intensity of correctable errors occurring in its storage unit. Then, a correction coefficient is determined based on the error correction statistics. The correction coefficient preferably adopts a form linearly consistent with the error correction statistics to maintain interpretability and avoid introducing complex inference mechanisms. That is, the error correction statistics are normalized to the error correction count or error correction byte ratio within a unit window according to object granularity, and then directly added by one to obtain the correction coefficient. This ensures that when the error correction statistics are zero, the correction coefficient is one and will not amplify the risk; while as the error correction statistics increase, the correction coefficient monotonically increases accordingly, thus reflecting the actual anchoring effect of media degradation. The formula for calculating the correction coefficient is as follows:

[0082]

[0083] In the formula, Let be the correction factor for data object i. Let be the error correction statistical intensity of data object i. The error correction statistical intensity is preferably expressed as the error correction count or error correction byte ratio after normalization per unit time and per unit data volume. When the error correction statistics are zero, the correction coefficient remains at one so as not to generate additional amplification of risk. When the error correction statistics increase, the correction coefficient increases in the same direction to reflect that the retention attenuation is more severe due to the increase in media error density.

[0084] After obtaining the correction coefficient, the dwell time, temperature acceleration factor and correction coefficient are multiplied together to obtain the retention decay risk of the data object. The dwell time describes the cumulative effect of time, the temperature acceleration factor describes the amplification effect of temperature on the retention decay rate, and the correction coefficient describes the reweighting effect of the observed error density on the risk. The three factors together determine the retention decay risk level of the data object under the current physical environment and dwell conditions.

[0085] S4. Calculate the strip strength based on the statistical distribution difference of the attenuation risk in the cabinet height direction, and associate the strip strength with the corresponding data object;

[0086] In an embodiment of the present invention, the strip strength is calculated based on the statistical distribution difference of the attenuation risk along the rack height direction, and the strip strength is associated with the corresponding data object, including:

[0087] Divide data objects into height buckets based on the height of the server rack;

[0088] Calculate the median risk of attenuation within each bucket height and use it as the median risk within the bucket.

[0089] Calculate the median risk of decay for all data objects in the cold data set, and use it as the global risk median;

[0090] Specifically, after obtaining the rack height position and attenuation risk of each data object in the cold data set, the data objects in the cold data set are divided into multiple height buckets according to the rack height position. The height buckets are used to group data objects in similar vertical height bands into the same statistical unit to characterize the spatial clustering pattern of the attenuation band. The height buckets are preferably divided using an equal-width bucket method with a bucket width of one to five rack height units, so that the number of objects in the bucket can support robust statistics and retain the band resolution. A bucket width of one height unit can obtain higher resolution but requires a larger number of objects. A bucket width of three to five height units can maintain statistical stability in racks with a smaller number of objects. After determining the bucket width, the height bucket number to which each data object belongs is calculated and written into the object entry. The bucket number is obtained by dividing the rack height position by the bucket width and rounding down to ensure that objects in the same height band have a consistent affiliation.

[0091] The data objects in the cold data set are grouped according to the height bucket number, and the retention decay risk of all data objects in each height bucket is collected. Median statistics are performed for each height bucket to obtain the median risk within the bucket. The use of the median can suppress the impact of a single extremely high-risk or extremely low-risk object on the representative value within the bucket and improve the robustness of stripe identification. At the same time, the platform constructs a global risk list for the retention decay risk of all data objects in the cold data set and calculates the median as the global risk median. This global risk median can represent the typical retention decay level of the cabinet in the current time window and serve as a normalization benchmark for subsequent calculation of stripe strength. After the calculation is completed, the platform writes the median risk within each height bucket and the global risk median into the stripe statistics table, respectively.

[0092] The ratio of the median risk within a bucket to the median risk globally is used as the strip intensity of the height bucket, and the strip intensity is assigned to the data object within the height bucket.

[0093] It should be noted that the strip intensity is a measure of the degree of cold data retention degradation within a certain height band in the vertical direction of the rack, relative to the typical level of the entire rack. It is obtained by comparing the representative level of the retention degradation risk of data objects within that height band with the representative level of the retention degradation risk of the entire cold data set. This expresses the risk clustering effect caused by vertical thermal stratification as a comparable result. A strip intensity greater than one indicates that the overall retention degradation risk of cold data objects within that height band is higher than the typical level of the rack and presents a risk strip. Within the height band, there is a greater likelihood of concentrated phenomena of increased error correction and repair pressure and data reliability degradation. A strip intensity less than one indicates that the overall risk within that height band is lower than the typical level of the rack and the strip effect is not significant. A strip intensity close to one indicates that the risk of that height band is similar to the global level and tends to be evenly distributed.

[0094] Specifically, after obtaining the median risk within each height bucket and the median risk of the global cold data set, the ratio of the two is defined as the stripe strength of that height bucket. The reason for using the ratio is that the median risk within the bucket characterizes the typical maintenance attenuation level of that height band, while the median risk of the global risk characterizes the typical maintenance attenuation level of the entire rack. Comparing the two can eliminate the scaling effect caused by the change in the overall risk level of different batches of data, so that the stripe strength directly reflects the degree to which the risk of that height band is higher or lower than the global risk. After completing the stripe strength calculation, the platform reads the height bucket number to which each data object belongs and queries the stripe strength corresponding to that height bucket, and writes the stripe strength into the stripe strength field of the data object. Furthermore, when the number of data objects in a certain height bucket is insufficient to support robust statistics, the platform prefers to merge with adjacent height buckets to supplement the number of objects and recalculate the median risk within the bucket, thereby avoiding unstable fluctuations in stripe strength due to insufficient samples.

[0095] S5. Calculate the confidence weight based on the preserving attenuation risk and the strip strength, perform weighted data analysis based on the confidence weight, and correct the search ranking based on the confidence weight;

[0096] In embodiments of the present invention, a confidence weight is calculated based on the preserving attenuation risk and the stripe strength; weighted data analysis is performed based on the confidence weight; and the retrieval ranking is corrected based on the confidence weight, including:

[0097] Confidential weights are calculated based on the risk of attenuation and the strip strength.

[0098] Specifically, after obtaining the maintenance decay risk and band strength for each data object, a monotonically decreasing and bounded mapping method is used to synthesize the two into a confidence weight. This ensures that the higher the maintenance decay risk and the stronger the band strength, the smaller the confidence weight. This automatically reduces the impact of high-risk data objects within the maintenance decay band on statistics and model parameters during data mining and big data analysis. The formula for calculating the confidence weight is as follows:

[0099]

[0100] In the formula, For reliable weights, To mitigate the risk of degradation, The term represents the strip intensity. The product in the denominator can simultaneously reflect the risk accumulation caused by time dwell, temperature acceleration, and error correction pressure, as well as the strip amplification effect caused by vertical thermal stratification. When either the risk or the strip intensity increases, the denominator increases, thus the weight continuously decreases and maintains monotonicity. The fractional structure ensures that the credible weight is always between zero and one, making it easy to use directly under different mining tasks and different data scales without introducing additional scale parameters. At the same time, when the attenuation risk is kept close to zero or the strip intensity is kept close to zero, the credible weight approaches one to avoid unnecessary weakening of low-risk data.

[0101] In data mining tasks, loss targets are weighted according to credibility weights in order to minimize the weighted loss target;

[0102] In statistical aggregation tasks, the statistical values ​​of data objects are calculated by weighted average based on the confidence weights.

[0103] Specifically, after calculating the trusted weights of each data object and forming a list of objects available for the task to read, the scope of data objects to be processed is first determined from the task configuration for the data mining task, and the feature data and label data of each data object are loaded. At the same time, the trusted weights corresponding to the data objects are read and written into the sample weight column, so that the subsequent calculation link can explicitly inject the risk of striping attenuation into the mining target without changing the feature construction method. On this basis, a loss function matching the mining type is selected and a weighted loss target is constructed. The loss value of each sample is multiplied by its trusted weight and then summed or averaged to obtain the weighted loss of all samples. The optimization target of minimizing the weighted loss is used as the driving parameter update. In distributed training or distributed mining scenarios, it is preferable to first calculate the local weighted loss and the local weights in each computing segment and then perform global merging to ensure that the weighting effect remains consistent when the sample size of different segments is unbalanced. Furthermore, it is preferable to perform stabilization processing on the trusted weights to avoid the effective sample number being too low due to extremely small weights, which would make the optimization unstable. The stabilization processing preferably adopts the method of dividing the trusted weight by the maximum value of the trusted weight in the batch.

[0104] For statistical aggregation tasks, the statistical value of each data object is extracted from the object list and its trusted weight is read. The statistical value and trusted weight are multiplied and summed to obtain a weighted statistic. The trusted weights are then summed to obtain a weighted sum. Subsequently, the weighted statistic is divided by the weighted sum to obtain a weighted average result. This automatically weakens the contribution of data objects in the decay band and with high risk in the aggregation result and reduces the impact of positional bias distortion on the overall statistical conclusion. Finally, the platform writes the model parameters obtained by weighted mining or the statistical results obtained by weighted aggregation, along with the trusted weight version and time window identifier used, into the result database and generates a traceable task report.

[0105] The credibility weights are normalized according to the maximum credibility weight among all data objects to obtain the normalized credibility.

[0106] The normalized confidence score is written as an extended field into the inverted index or vector index of the data object. In response to the retrieval request, the basic relevance score calculated based on the query is obtained. The corrected ranking score is obtained by multiplying the basic relevance score and the normalized confidence score.

[0107] Specifically, after calculating the trusted weights of all data objects, the trusted weights of the data objects participating in the index update in the same batch are first aggregated, and the maximum trusted weight among them is calculated as the normalization benchmark. The maximum trusted weight is preferably calculated by first calculating the maximum value of each shard within the index shard range and then performing global merging to obtain the global maximum value to adapt to distributed storage and parallel computing scenarios. The trusted weight of each data object is divided by the maximum trusted weight to obtain the normalized trusted weight, so that the normalized trusted weight falls between zero and one and maintains the relative trusted differences between objects, while avoiding uncontrollable drift in the search ranking caused by changes in the trusted weight scale of different batches.

[0108] After normalization, the normalized confidence score is written as an extended field into the index structure. For inverted index scenarios, this extended field is written into the index entries along with the document identifier and remains consistent during segment merging or index rebuilding. For vector index scenarios, this extended field is written into the vector entry metadata and returned along with the candidate set during the vector recall stage. This ensures that the normalized confidence score corresponding to each candidate data object can be read during the retrieval and ranking stage. When the platform receives a retrieval request, it parses the query and performs basic retrieval calculations to obtain the candidate data object set and its basic relevance score. The basic relevance score is preferably obtained by matching terms in the inverted index. The similarity score is calculated by combining the weights or by the vector index. Then, the normalized confidence score of each data object in the candidate set is read, and the basic relevance score is multiplied by the normalized confidence score to obtain the corrected ranking score. The reason for the product fusion is that the basic relevance score reflects the degree of matching between the query and the content, while the normalized confidence score reflects the reliability level of the object under the influence of the retention decay band. Multiplying the two can proportionally weaken low-confidence objects without changing the calculation method of the basic relevance score and maintain the continuity of the ranking score. Finally, the platform sorts the candidate data objects in descending order according to the corrected ranking score and outputs the ranking result.

[0109] like Figure 2 The diagram shown is a functional block diagram of a cloud computing platform based on data mining and big data analysis provided in an embodiment of the present invention.

[0110] In this embodiment, the functions of each module are as follows:

[0111] The data management module is used to obtain the write time, rack height position, and error correction statistics of data objects, and uses the difference between the current time and the write time as the dwell time to construct a cold data set based on the write time.

[0112] The temperature profile construction module is used to construct node temperature profiles based on storage node temperature monitoring data and rack height location.

[0113] The risk calculation module is used to calculate the retention decay risk of data objects in the cold data set by combining the node temperature profile and the residence time.

[0114] The strip analysis module calculates the strip strength based on the statistical distribution difference of attenuation risk along the height of the cabinet, and associates the strip strength with the corresponding data object;

[0115] The analysis and retrieval service module is used to calculate a reliable weight based on the preserving attenuation risk and the band strength, perform weighted data analysis based on the reliable weight, and correct the retrieval ranking based on the reliable weight.

[0116] like Figure 3As shown in the figure, this diagram illustrates the vertical thermal stratification of a server rack. Using the server rack as a deployment carrier, it shows the physical environment in which storage nodes are installed vertically inside the rack. Cold air is introduced into the lower part of the rack and hot air is exhausted from the upper part, creating a vertical thermal stratification inside the rack from bottom to top. The temperature range in the vertical direction gradually transitions from lower to higher temperatures, thus reflecting the differences in long-term heating conditions corresponding to different rack heights.

[0117] like Figure 4 As shown in the figure, this is a schematic diagram of the node temperature profile, which shows the relationship between the node temperature profile and the rack height as the vertical dimension and temperature as the horizontal dimension. The dashed line represents the reference temperature, and the solid line represents the trend of the actual node temperature increasing with the height. This shows that the node temperature near the top of the rack is higher than the node temperature near the bottom of the rack. This diagram can intuitively illustrate that under vertical thermal stratification, the platform can map each storage node to different temperature levels according to the rack height and form a node temperature profile, providing a basis for assessing the retention and degradation risk of cold data objects and characterizing striped differences.

[0118] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A cloud computing method based on data mining and big data analysis, characterized in that, Includes the following steps: S1. Obtain the write time, rack height position, and error correction statistics of the data object, use the difference between the current time and the write time as the dwell time, and construct a cold data set based on the write time; S2. Construct a node temperature profile based on storage node temperature monitoring data and rack height location; S3. Based on node temperature profiles, dwell time, and error correction statistics, calculate the retention decay risk of data objects in cold datasets. Based on nodal temperature profiles, residence time, and error correction statistics, the specific steps for calculating the retention decay risk of data objects in a cold dataset are as follows: The median of all node temperature profiles within the rack is used as the reference temperature. The temperature acceleration factor is calculated based on the reference temperature and the node temperature profiles. The formula for calculating the temperature acceleration factor is as follows: In the formula, As a temperature acceleration factor, This is the nodal temperature profile. For reference temperature, This is a preset constant; The correction coefficient is determined based on the error correction statistics. The retention decay risk of the data object is obtained by multiplying the residence time, temperature acceleration factor and correction coefficient. S4. Calculate the strip strength based on the statistical distribution difference of the attenuation risk in the cabinet height direction, and associate the strip strength with the corresponding data object; The specific steps for calculating the strip strength based on the statistical distribution difference of attenuation risk along the rack height direction and associating the strip strength with the corresponding data object are as follows: Divide data objects into height buckets based on the height of the server rack; Calculate the median risk of attenuation within each bucket height and use it as the median risk within the bucket. Calculate the median risk of decay for all data objects in the cold data set, and use it as the global risk median; The ratio of the median risk within a bucket to the median risk globally is used as the strip intensity of the height bucket, and the strip intensity is assigned to the data object within the height bucket. S5. Calculate the confidence weight based on the preserving attenuation risk and the strip strength, perform weighted data analysis based on the confidence weight, and correct the search ranking based on the confidence weight.

2. The cloud computing method based on data mining and big data analysis according to claim 1, characterized in that, Constructing a cold data set includes: Record the storage node identifier, rack height, write time, and error correction statistics for each data object. The error correction statistics are the error correction event count or error correction byte count of the data object within a unit of time. The median dwell time of all data objects is calculated, and data objects with a dwell time not less than the median are selected to form a cold data set.

3. The cloud computing method based on data mining and big data analysis according to claim 1, characterized in that, A node temperature profile is constructed based on storage node temperature monitoring data and rack height, including: Collect the intake air temperature sequence of each storage node, and take the median of the intake air temperature sequence within a preset time window to obtain a robust temperature characterization. Temperature gradients are obtained by linear fitting based on cabinet height and robust temperature characterization. The temperature at different heights in the vertical direction of the cabinet is corrected based on the temperature gradient to obtain the node temperature profile of each storage node.

4. The cloud computing method based on data mining and big data analysis according to claim 1, characterized in that, The formula for calculating the credibility weight is: In the formula, For reliable weights, To mitigate the risk of degradation, The value represents the strip strength.

5. The cloud computing method based on data mining and big data analysis according to claim 1, characterized in that, Weighted data analysis is performed based on the aforementioned confidence weights, including: In data mining tasks, loss targets are weighted according to credibility weights in order to minimize the weighted loss target; In statistical aggregation tasks, a weighted average is calculated for the statistical values ​​of data objects based on their credibility weights.

6. The cloud computing method based on data mining and big data analysis according to claim 1, characterized in that, The search ranking is adjusted based on the aforementioned trust weights, including: The credibility weights are normalized according to the maximum credibility weight among all data objects to obtain the normalized credibility. The normalized confidence score is written as an extended field into the inverted index or vector index of the data object. In response to the retrieval request, the basic relevance score calculated based on the query is obtained. The corrected ranking score is obtained by multiplying the basic relevance score and the normalized confidence score.

7. A cloud computing platform based on data mining and big data analysis, characterized in that, The cloud computing platform includes: The data management module is used to obtain the write time, rack height position, and error correction statistics of data objects, and uses the difference between the current time and the write time as the dwell time to construct a cold data set based on the write time. The temperature profile construction module is used to construct node temperature profiles based on storage node temperature monitoring data and rack height location. The risk calculation module is used to calculate the retention decay risk of data objects in the cold data set based on node temperature profiles, residence time, and error correction statistics. Based on nodal temperature profiles, residence time, and error correction statistics, the specific steps for calculating the retention decay risk of data objects in a cold dataset are as follows: The median of all node temperature profiles within the rack is used as the reference temperature. The temperature acceleration factor is calculated based on the reference temperature and the node temperature profiles. The formula for calculating the temperature acceleration factor is as follows: In the formula, As a temperature acceleration factor, This is the nodal temperature profile. For reference temperature, This is a preset constant; The correction coefficient is determined based on the error correction statistics. The retention decay risk of the data object is obtained by multiplying the residence time, temperature acceleration factor and correction coefficient. The strip analysis module is used to calculate the strip strength based on the statistical distribution difference of the attenuation risk in the direction of cabinet height, and associate the strip strength with the corresponding data object; The specific steps for calculating the strip strength based on the statistical distribution difference of attenuation risk along the rack height direction and associating the strip strength with the corresponding data object are as follows: Divide data objects into height buckets based on the height of the server rack; Calculate the median risk of attenuation within each bucket height and use it as the median risk within the bucket. Calculate the median risk of decay for all data objects in the cold data set, and use it as the global risk median; The ratio of the median risk within a bucket to the median risk globally is used as the strip intensity of the height bucket, and the strip intensity is assigned to the data object within the height bucket. The analysis and retrieval service module is used to calculate a reliable weight based on the preserving attenuation risk and the band strength, perform weighted data analysis based on the reliable weight, and correct the retrieval ranking based on the reliable weight.

Images