Preprocessing method, system, device and medium for index data of alloy
By calculating indicators such as standard deviation, skewness, and kurtosis of alloy steel data and dynamically matching preprocessing strategies, the problems of dimensional differences and uneven distribution of alloy steel data were solved, thereby improving the training stability and prediction accuracy of machine learning models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PANGANG GROUP JIANGYOU CHANGCHENG SPECIAL STEEL COMPANY LIMITED
- Filing Date
- 2026-02-26
- Publication Date
- 2026-06-09
AI Technical Summary
In existing technologies, alloy steel data exhibits significant differences in dimensions, diverse distribution patterns, and poor numerical stability, making it difficult to train machine learning models. In particular, division-to-zero errors and data biases are prone to occur during gradient descent.
By acquiring a set of index data for alloy samples, calculating statistical indicators such as standard deviation, skewness, and kurtosis, and dynamically matching preprocessing strategies, such as zero-variance fracturing, data standardization, logarithmic smoothing, and normalization, different types of data can be processed.
It improves the training stability and convergence speed of machine learning models, enhances prediction accuracy, reduces the risk of model collapse, and results in a more uniform data distribution.
Smart Images
Figure CN122173775A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing, and specifically to a method, system, device, and medium for preprocessing index data of an alloy. Background Technology
[0002] In materials property prediction and machine learning model training, the quality of the input data directly determines the upper limit of the model. However, real-world industrial materials data generally face the following challenging problems: 1. Significant dimensional differences: For example, in alloy steel data, carbon (C) content is typically between 0.1% and 1.0%, while chromium (Cr) or nickel (Ni) can be as high as 10% to 20%, and trace elements (such as B and Ti) are at the ppm level (parts per million). Directly inputting these into the model will cause the gradient descent direction to be dominated by large numerical features, affecting convergence.
[0003] 2. Diverse distribution patterns: Some impurity elements (such as S and P) exhibit an extremely right-skewed distribution (the vast majority of samples have very low concentrations, while a few samples have high concentrations); some process parameters may exhibit a bimodal or long-tailed distribution. Traditional Z-score standardization assumes that the data follows a normal distribution, which can introduce bias when processing such data.
[0004] 3. Poor numerical stability: Some features may remain completely unchanged (variance is zero) in a specific batch of data. When using the traditional formula (x-μ) / σ for calculation, it will lead to a division by zero error, causing the program to crash or generate NaN (non-numeric) values. Summary of the Invention
[0005] In view of this, in order to overcome at least one aspect of the above-mentioned problems, embodiments of the present invention propose a preprocessing method for alloy index data, comprising the following steps: Obtain multiple index data for each alloy sample and statistically analyze the data for each index to obtain a set. Determine the data type of each collection; In response to the data type of the set being a first preset type, the standard deviation, skewness, kurtosis, and extreme value range of each set are calculated; A processing strategy is determined based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and the corresponding set is processed according to the processing strategy.
[0006] In some embodiments, a processing strategy is determined based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and data processing is performed on the corresponding sets according to the processing strategy, further including: In response to a standard deviation of 0, all data in the corresponding set are set to 0.
[0007] In response to a standard deviation that is not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and the corresponding set is processed according to the processing strategy.
[0008] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold, data standardization is performed based on the median and interquartile range in the corresponding set.
[0009] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being less than a second threshold, data standardization is performed based on the mean and standard deviation of the corresponding set.
[0010] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being not less than a second threshold, logarithmic smoothing is performed on the data in the corresponding set, and data standardization is performed based on the mean and standard deviation of the data obtained after logarithmic smoothing.
[0011] In some embodiments, logarithmic smoothing is performed on the data in the corresponding set, further including logarithmic smoothing based on the following formula: X log = ln(x + c) Where x is the original data in the set, and c is a constant.
[0012] In some embodiments, the method further includes: In response to the data type of the set being a second preset type, normalization processing is performed based on the maximum and minimum values in the set.
[0013] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a preprocessing system for alloy index data, including... The acquisition module is configured to acquire multiple index data for each alloy sample and statistically obtain a set of each index data. The module is configured to determine the data type of each collection; The calculation module is configured to calculate the standard deviation, skewness, kurtosis, and extreme value range of each set in response to the data type of the set being a first preset type. The processing module is configured to determine a processing strategy based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and to perform data processing on the corresponding set according to the processing strategy.
[0014] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising: At least one processor; and The memory stores a computer program that can run on the processor, which, when executing the program, performs the steps of a preprocessing method for index data of any alloy as described above.
[0015] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the preprocessing method for index data of any alloy as described above.
[0016] The present invention has one of the following beneficial technical effects: the solution proposed in the present invention can automatically identify feature attributes based on the data distribution characteristics of material data through statistical indicators, thereby dynamically matching preprocessing strategies. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart illustrating a method for preprocessing alloy index data provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a traffic accident detection system provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.
[0020] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.
[0021] According to one aspect of the present invention, embodiments of the present invention provide a method for preprocessing index data of alloys, such as... Figure 1 As shown, it may include the following steps: S1, obtain multiple index data for each alloy sample and statistically obtain a set of index data for each index; S2, determine the data type of each set; S3, in response to the data type of the set being a first preset type, calculate the standard deviation, skewness, kurtosis, and extreme value range of each set; S4. Determine the processing strategy based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and perform data processing on the corresponding set according to the processing strategy.
[0022] The proposed solution can automatically identify characteristic attributes based on the data distribution characteristics of material data through statistical indicators, thereby dynamically matching preprocessing strategies.
[0023] In some embodiments, S1, multiple indicator data for each alloy sample are acquired and a set of data for each indicator is statistically obtained. Specifically, firstly, complete indicator data for all alloy samples are obtained, with each alloy sample corresponding to a set of multi-dimensional indicator data. Then, based on the indicator type, the indicator data for all alloy samples are extracted one by one, and the specific data of the same indicator type in different alloy samples are separately collected to complete the preliminary screening and summarization of the same indicator data. The extracted same indicator data are integrated to form a dedicated data set corresponding to each indicator. The set contains all the data for that indicator for all alloy samples, and one indicator type corresponds to one independent data set. In this way, for the indicator data of multiple alloy samples, the set statistics are completed according to the principle of merging the same indicators, that is, the same type of indicator data of all alloy samples is extracted and summarized to form an independent data set corresponding to each indicator.
[0024] In some embodiments, a processing strategy is determined based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and data processing is performed on the corresponding sets according to the processing strategy, further including: In response to a standard deviation of 0, all data in the corresponding set are set to 0.
[0025] In response to a standard deviation that is not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and the corresponding set is processed according to the processing strategy.
[0026] Specifically, before performing any standardization, the standard deviation σ is checked first. If σ = 0 (or less than the machine precision ε), it means that the feature has the same value in all samples and contains no information. The processing strategy is to avoid performing division operations, that is, to directly set all values in the feature column to 0. This eliminates division-by-zero errors at the source and enhances the stability of the algorithm when running unattended.
[0027] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold, data standardization is performed based on the median and interquartile range in the corresponding set.
[0028] Specifically, if the kurtosis is greater than the first threshold, it indicates that the data in the set conforms to a long-tailed / outlier distribution. In this case, the median (Q2) and interquartile range (Q1, Q3) can be used instead of the mean and variance for data standardization. x' = (x - Q2) / (Q3 - Q1) Where x represents the original data.
[0029] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being less than a second threshold, data standardization is performed based on the mean and standard deviation of the corresponding set.
[0030] Specifically, if the kurtosis is greater than the first threshold and the absolute value of the skewness is less than the second threshold, it indicates that the data in this set conforms to an approximately normal distribution, and the data can be standardized using the standard Z-score, i.e.: x' = (x - μ) / σ Where σ is the standard deviation and μ is the mean.
[0031] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being not less than a second threshold, logarithmic smoothing is performed on the data in the corresponding set, and data standardization is performed based on the mean and standard deviation of the data obtained after logarithmic smoothing.
[0032] In some embodiments, logarithmic smoothing is performed on the data in the corresponding set, further including logarithmic smoothing based on the following formula: X log = ln(x + c) Where x is the original data in the set, and c is a constant.
[0033] Specifically, if the kurtosis is greater than the first threshold and the absolute value of the skewness is not less than the second threshold, it indicates that the data in the set conforms to a skewed distribution. In this case, log smoothing can be performed on the data in the corresponding set, and then data standardization can be performed based on the mean and standard deviation of the log smoothed data. This effectively stretches dense intervals and compresses long-tail intervals, making them closer to a normal distribution. X log = ln(x + c) x' = (X log - μ log ) / σ log Where, μ log For multiple X log The mean, σ log For multiple X log The standard deviation, where c is a constant (e.g., 10). -6 ).
[0034] In some embodiments, the method further includes: In response to the data type of the set being a second preset type, normalization processing is performed based on the maximum and minimum values in the set.
[0035] Specifically, if the data in the set meets the characteristic of having a strictly limited range of known values, then Min-Max normalization can be used for standardization, that is: x' = (x - X_min) / (X_max - X_min) Example: First, production sample data for multiple 20CrMnTi gear steels were collected. Each sample data contained three typical characteristics: chromium content (Cr), sulfur content (S), and vacuum degassing flag (VD_Flag). The system processing flow is as follows: Step 1: Calculation of statistical indicators Characteristic A (Cr): Mean μ=1.12, Standard deviation σ=0.05, Skewness S=0.12, Kurtosis K=0.8.
[0036] Feature B(S): mean μ=0.015, standard deviation σ=0.008, skewness S=2.5 (severely right-skewed), kurtosis K=1.2.
[0037] Feature C (VD_Flag): Mean μ=1.0, Standard Deviation σ=0.0 (all values are 1, indicating that all have undergone vacuum degassing).
[0038] For characteristic A (Cr): Judgment: Standard deviation σ>0 indicates non-zero variance.
[0039] Judgment: Skewness |0.12| < 1.0 (preset threshold T_skew).
[0040] Decision: Determined as "Approximately normal distribution", Z-score standardization is chosen. .
[0041] Calculate: x' = (x - 1.12) / 0.05.
[0042] For feature B(S): Judgment: Standard deviation σ > 0.
[0043] Judgment: Skewness |2.5| ≥ 1.0.
[0044] Decision: The distribution is determined to be "skewed" and logarithmic transformation is chosen for standardization.
[0045] Calculation: First, calculate x_log = ln(x + 1e-6); then calculate the mean and standard deviation of x_log, and finally standardize. This operation stretches the values that were originally crowded around 0, making the data distribution more even.
[0046] For feature C(VD_Flag): Judgment: Standard deviation σ = 0.
[0047] Decision: Trigger the "zero variance circuit breaker mechanism".
[0048] Operation: Directly output a vector of all zeros without performing division. This avoids program errors (Division by Zero).
[0049] Comparative example: To verify the effectiveness of the present invention, a neural network model (MLP) was trained using the same gear steel dataset, and the performance of "traditional Z-score normalization" and "adaptive normalization of the present invention" were compared.
[0050] Control group (all using Z-score): Because the distribution of sulfur (S) is extremely uneven, most of the data is compressed into the negative range after standardization, and a few high-sulfur samples become extremely positive, causing gradient oscillations during model training.
[0051] Training results: The model requires 500 epochs to converge, and the final test set R² is 0.82.
[0052] Experimental group (method of this invention): After logarithmic transformation, the distribution of sulfur (S) is close to normal; the zero variance characteristic is safely handled.
[0053] Training results: The model converged in just 350 epochs (a 30% speed improvement), and the final test set R² reached 0.91 (an approximately 11% improvement in accuracy).
[0054] Experiments show that data processed by the scheme proposed in this invention improves the convergence speed by about 30% when training neural networks or SVM models, and the prediction accuracy (R²) of the final model is improved by an average of more than 10%.
[0055] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a preprocessing system 400 for alloy index data, such as... Figure 2 As shown, it includes: The acquisition module 401 is configured to acquire multiple index data for each alloy sample and statistically obtain a set of each index data. Module 402 is configured to determine the data type of each collection; The calculation module 403 is configured to calculate the standard deviation, skewness, kurtosis, and extreme value range of each set in response to the data type of the set being a first preset type. The processing module 404 is configured to determine a processing strategy based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and to perform data processing on the corresponding set according to the processing strategy.
[0056] In some embodiments, a processing strategy is determined based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and data processing is performed on the corresponding sets according to the processing strategy, further including: In response to a standard deviation of 0, all data in the corresponding set are set to 0.
[0057] In response to a standard deviation that is not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and the corresponding set is processed according to the processing strategy.
[0058] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold, data standardization is performed based on the median and interquartile range in the corresponding set.
[0059] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being less than a second threshold, data standardization is performed based on the mean and standard deviation of the corresponding set.
[0060] In some embodiments, in response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being not less than a second threshold, logarithmic smoothing is performed on the data in the corresponding set, and data standardization is performed based on the mean and standard deviation of the data obtained after logarithmic smoothing.
[0061] In some embodiments, logarithmic smoothing is performed on the data in the corresponding set, further including logarithmic smoothing based on the following formula: X log = ln(x + c) Where x is the original data in the set, and c is a constant.
[0062] In some embodiments, the method further includes: In response to the data type of the set being a second preset type, normalization processing is performed based on the maximum and minimum values in the set.
[0063] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 3 As shown, embodiments of the present invention also provide a computer device 501, comprising: At least one processor 520; and The memory 510 stores a computer program 511 that can run on a processor. When the processor 520 executes the program, it performs the steps of the preprocessing method for the index data of any of the alloys described above.
[0064] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 4As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the steps of the preprocessing method for the index data of any of the alloys described above.
[0065] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.
[0066] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.
[0067] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.
[0068] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.
[0069] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.
[0070] The embodiment numbers disclosed in the above embodiments of the present invention are merely for description and do not represent the superiority or inferiority of the embodiments.
[0071] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0072] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.
Claims
1. A method for preprocessing index data of an alloy, characterized in that, Includes the following steps: Obtain multiple index data for each alloy sample and statistically analyze the data for each index to obtain a set. Determine the data type of each collection; In response to the data type of the set being a first preset type, the standard deviation, skewness, kurtosis, and extreme value range of each set are calculated; A processing strategy is determined based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and the corresponding set is processed according to the processing strategy.
2. The method as described in claim 1, characterized in that, A processing strategy is determined based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and data processing is performed on the corresponding sets according to the processing strategy, further including: In response to a standard deviation of 0, all data in the corresponding set are set to 0; In response to a standard deviation that is not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and the corresponding set is processed according to the processing strategy.
3. The method as described in claim 2, characterized in that, In response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold, data standardization is performed based on the median and interquartile range in the corresponding set.
4. The method as described in claim 2, characterized in that, In response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being less than a second threshold, data standardization is performed based on the mean and standard deviation of the corresponding set.
5. The method as described in claim 2, characterized in that, In response to a standard deviation not equal to 0, a processing strategy is determined based on skewness, kurtosis, and extreme value range, and data processing is performed on the corresponding set according to the processing strategy, further including: In response to the kurtosis being greater than a first threshold and the absolute value of the skewness being not less than a second threshold, logarithmic smoothing is performed on the data in the corresponding set, and data standardization is performed based on the mean and standard deviation of the data obtained after logarithmic smoothing.
6. The method as described in claim 5, characterized in that, Log-smoothing is performed on the data in the corresponding set, further including log-smoothing based on the following formula: X log = ln(x + c) Where x is the original data in the set, and c is a constant.
7. The method as described in claim 1, characterized in that, Also includes: In response to the data type of the set being a second preset type, normalization processing is performed based on the maximum and minimum values in the set.
8. A preprocessing system for index data of an alloy, characterized in that, include The acquisition module is configured to acquire multiple index data for each alloy sample and statistically obtain a set of each index data. The module is configured to determine the data type of each collection; The calculation module is configured to calculate the standard deviation, skewness, kurtosis, and extreme value range of each set in response to the data type of the set being a first preset type. The processing module is configured to determine a processing strategy based on the standard deviation, skewness, kurtosis, and extreme value range of each set, and to perform data processing on the corresponding set according to the processing strategy.
9. A computer device, comprising: At least one processor; as well as A memory storing a computer program executable on the processor, characterized in that the processor executes the program by performing the steps of the method as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-7.