A data analysis management method and system based on digital intelligence engineering

Abstract

The present invention discloses a data analysis management method and system based on digital intelligence engineering, which relates to the field of data analysis management technology, including performing position analysis on a data set to be managed and determining the position of a first data to be filled. The present invention first determines the position of the first data to be filled, then determines the position of the abnormal data, and sets the position of the abnormal data to be the position of the second data to be filled. Finally, the data set to be filled is calculated by Euclidean calculation to obtain a first data similarity set and a second data similarity set, and the first data similarity set and the second data similarity set are selected and calculated to obtain first filling data and second filling data. In order to avoid the calculated first filling data and second filling data being abnormal data, the first filling data and the second filling data are verified to determine a normal data set. The above method not only realizes the filling and modification of missing data and abnormal data, but also shortens the data call time of digital intelligence engineering.

Description

Technical Field

[0001] The present invention relates to the field of data analysis and management technology, and particularly to a data analysis and management method and system based on digital intelligence engineering. Background Art

[0002] Digital intelligence engineering is a systematic project that deeply integrates digitalization and intelligence. It aims to optimize and innovate the entire business process and reshape the industrial ecosystem through next-generation information technologies such as big data, artificial intelligence, and the Internet of Things. Its core is to use data to drive decision-making and intelligently empower efficiency, thereby developing "digital intelligence" capabilities to serve socioeconomic development and industrial upgrading.

[0003] There are missing values ​​and outliers in the data of digital intelligence projects. If the missing values ​​are not supplemented and the outliers are not modified, the data of the digital intelligence project will be incomplete. When the data of the digital intelligence project needs to be called subsequently, the missing values ​​and outliers will need to be supplemented and modified again, which prolongs the data calling time of the digital intelligence project. Summary of the Invention

[0004] In order to solve the above technical problems, a data analysis management method and system based on digital intelligence engineering are provided. This technical solution solves the problem that there are missing values ​​and outliers in the data of digital intelligence engineering proposed in the above background technology. If the missing values ​​are not supplemented and the outliers are not modified, the data of the digital intelligence engineering will be incomplete. When the data of the digital intelligence engineering needs to be called subsequently, the missing values ​​and outliers need to be supplemented and modified again, which prolongs the data calling time of the digital intelligence engineering.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0006] A data analysis and management method based on digital intelligence engineering, comprising:

[0007] Acquire a data set to be managed, perform a position analysis on the data set to be managed based on the analysis management terminal, and determine the position of the first data to be filled;

[0008] The analysis management terminal performs an abnormality analysis on the data set to be managed according to the first data position to be filled, and determines the second data position to be filled and the data set to be filled;

[0009] Based on the analysis management terminal, data is filled in the data set to be filled according to the first data position to be filled and the second data position to be filled, so as to obtain a normal data set.

[0010] Preferably, the step of obtaining the data set to be managed and performing position analysis on the data set to be managed based on the analysis management terminal to determine the position of the first data to be filled specifically includes the following steps:

[0011] Based on the analysis and management terminal, the database system is read and processed to obtain the data set to be managed;

[0012] Based on the analysis management terminal, the management data set is traversed and processed to determine the location of missing data;

[0013] Based on the analysis management terminal, the information of the management data set to be managed is read and processed based on the location of missing data, and the row and column information of the missing data location is obtained;

[0014] The analysis management terminal generates a missing location label based on the row and column information of the data missing location;

[0015] Based on the analysis management terminal, a missing position tag is embedded in the data missing position, and the data missing position containing the missing position tag is set as the first data position to be filled.

[0016] Preferably, the analysis management terminal performs an abnormality analysis on the data set to be managed according to the first data position to be filled, and determines the second data position to be filled and the data set to be filled, specifically comprising the following steps:

[0017] Obtain the data set to be analyzed for anomalies;

[0018] Based on the analysis and management terminal, the data set to be analyzed for abnormality is calculated and processed to determine the location of abnormal data;

[0019] The analysis management terminal deletes data from the to-be-managed data set according to the abnormal data location, and determines a second to-be-filled data location and a to-be-filled data set.

[0020] Preferably, the step of obtaining the data set to be analyzed for abnormality specifically includes the following steps:

[0021] Based on the analysis management terminal, information reading and processing is performed on the missing position tag of each first data position to be filled, and row information of each missing data is obtained;

[0022] Based on the analysis management terminal, count the row information of each missing data to obtain the number of missing data in each row;

[0023] Based on the analysis management terminal, the number of missing data in each row is judged and processed to determine the first deleted data set;

[0024] Based on the analysis management terminal, the missing data in the first deleted data set is deleted, and the remaining data is set as the data set to be analyzed for abnormality.

[0025] Preferably, the step of determining the number of missing items in each row of data based on the analysis management terminal to determine the first deleted data set specifically includes the following steps:

[0026] Based on the analysis and management terminal, the number of missing data in each row and the set missing number threshold are judged and processed;

[0027] If the number of missing rows in each row is greater than or equal to the set missing number threshold, the row of data is missing too much and cannot be filled in. The row of data is deleted and the remaining data is set as the first deleted data set;

[0028] If the number of missing rows in each row of data is less than the set missing number threshold, the row of data has few missing rows and can be filled with subsequent data. All row data with less than the set missing number threshold are set as the first deleted data set.

[0029] Preferably, the step of performing computational processing on the data set to be analyzed for abnormality based on the analysis management terminal and determining the location of abnormal data specifically includes the following steps:

[0030] Based on the analysis management terminal, multiple calculations are performed on the data set to be analyzed for abnormality to obtain the standard deviation and mean of the data set to be analyzed for abnormality;

[0031] Based on the analysis management terminal, each data, standard deviation and mean in the data set to be analyzed for abnormality are calculated and processed to obtain the information to be verified for each data;

[0032] Based on the analysis and management terminal, the information to be verified for each data is judged and processed to determine the location of abnormal data.

[0033] Preferably, the analysis and management terminal is used to judge and process the information to be verified for each data, and determine the location of abnormal data, including the following steps:

[0034] Based on the analysis and management terminal, the information to be verified for each data and the set verification data threshold are judged and processed;

[0035] If the absolute value of the information to be verified is greater than the set verification data threshold, the data corresponding to the information to be verified is abnormal data, and the row information and column information of the data are recorded and set as the abnormal data position;

[0036] If the absolute value of the information to be verified is less than or equal to the set verification data threshold, the data corresponding to the information to be verified is normal data.

[0037] Preferably, the step of filling the data set to be filled based on the first data position to be filled and the second data position to be filled based on the analysis management terminal to obtain the normal data set specifically includes the following steps:

[0038] Based on the analysis management terminal, the data set to be filled is standardized;

[0039] Based on the first data position to be filled and the second data position to be filled, respectively, calculating and processing the remaining data to obtain a first data similarity set and a second data similarity set;

[0040] Based on the minimum function, sorting the data in the first data similarity set and the second data similarity set respectively;

[0041] Based on the random number function, length selection is performed on the first data similarity set and the second data similarity set respectively to obtain the first data length and the second data length;

[0042] Based on the analysis management terminal, data is selected from the first data similarity set and the second data similarity set according to the first data length and the second data length, to obtain a first data similarity subset and a second data similarity subset;

[0043] Based on the analysis management terminal, respectively calculating the mean of the data in the first data similarity subset and the second data similarity subset to obtain first filling data and second filling data;

[0044] Based on the analysis management terminal, the first filling data and the second filling data are filled into the first data position to be filled and the second data position to be filled, respectively, to obtain a data set to be verified;

[0045] Based on the analysis management terminal, verify the data set to be verified and obtain the first data adjustment length and the second data adjustment length;

[0046] Based on the analysis management terminal, data calculation and data filling processing are re-performed on the first data similarity set and the second data similarity set according to the first data adjustment length and the second data adjustment length to obtain a normal data set.

[0047] Preferably, the step of performing verification processing on the data set to be verified based on the analysis management terminal and obtaining the first data adjustment length and the second data adjustment length specifically includes the following steps:

[0048] Based on the analysis management terminal, the verification data set is respectively calculated and processed to obtain the mean of the row where the first filling data is located and the mean of the row where the second filling data is located;

[0049] Based on the analysis management terminal, respectively, the mean of the row where the first filling data is located and the mean of the row where the first filling data and the second filling data are located are judged and processed;

[0050] If the mean value of the row where the first padding data is located is much larger or much smaller than the first padding data, adjusting the length of the first data based on the analysis management terminal to obtain the adjusted length of the first data;

[0051] If the mean value of the row where the first padding data is located is approximately equal to the first padding data, there is no need to adjust the length of the first data;

[0052] If the mean value of the row where the second padding data is located is much larger or much smaller than the second padding data, the length of the second data is adjusted based on the analysis management terminal to obtain the adjusted length of the second data;

[0053] If the mean value of the row where the second padding data is located is approximately equal to the second padding data, there is no need to adjust the length of the second data.

[0054] Furthermore, a data analysis and management system based on digital intelligence engineering is proposed, which is used to implement the above-mentioned data analysis and management method based on digital intelligence engineering, including:

[0055] An analysis and management terminal is used to control each module to perform location analysis, anomaly analysis, and data filling on the data set to be managed, and obtain a normal data set. The analysis and management terminal is used to control information interaction and data transmission between each module;

[0056] A database system, wherein the database system is used to store the data set to be managed;

[0057] a data missing location determination module, the data missing location determination module being used to traverse the data set to be managed, read information, and determine the first location of data to be filled;

[0058] a data anomaly location determination module, the data anomaly location determination module being used to perform data deletion and data calculation on the data set to be managed, and determine the second data location to be filled and the data set to be filled;

[0059] A filling data calculation module, wherein the filling data calculation module performs similarity calculation, length selection, and data mean calculation on the data set to be filled according to the first data position to be filled and the second data position to be filled, to obtain the data set to be verified;

[0060] The filling data verification module is used to perform data verification on the data set to be verified, perform secondary calculation of the filling data, and obtain a normal data set.

[0061] Furthermore, a storage medium is proposed, on which a computer program is stored. When the computer program is called and run, it executes a data analysis and management method based on digital intelligence engineering as described above.

[0062] Compared with the existing technology, the present invention provides a data analysis and management method and system based on digital intelligence engineering, which has the following beneficial effects:

[0063] The present invention first performs a position analysis on the data set to be managed to determine the position of the first data to be filled, then performs an anomaly analysis on the data set to be managed to determine the position of the abnormal data, and sets the position of the abnormal data to be the position of the second data to be filled, and finally, calculates the data set to be filled by Euclidean calculation to obtain a first data similarity set and a second data similarity set, and performs selection calculation on the first data similarity set and the second data similarity set to obtain first filling data and second filling data. In order to avoid the calculated first filling data and second filling data being abnormal data, the first filling data and the second filling data are verified to determine a normal data set. The above method not only realizes the filling and modification of missing data and abnormal data, but also shortens the data call time of digital intelligence engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the process of steps S100-S300 in a data analysis and management method based on digital intelligence engineering and a system proposed by the present invention;

[0065] Figure 2 This is a structural block diagram of a data analysis and management method and system based on digital intelligence engineering proposed in the present invention. DETAILED DESCRIPTION

[0066] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0067] Reference Figure 1 As shown, a data analysis and management method based on digital intelligence engineering includes:

[0068] S100: Acquire a data set to be managed, perform a position analysis on the data set to be managed based on the analysis management terminal, and determine the position of the first data to be filled;

[0069] It is understandable that the data processed here are all numerical data, and the subsequent series of data processing methods are all designed for numerical data;

[0070] S200: The analysis management terminal performs an abnormality analysis on the data set to be managed based on the first data location to be filled, and determines the second data location to be filled and the data set to be filled;

[0071] S300: Based on the analysis management terminal, fill the data set to be filled according to the first data position to be filled and the second data position to be filled, to obtain a normal data set;

[0072] It is understandable to those skilled in the art that data may be missing or abnormal during collection. If the data is stored without being processed, the missing data and abnormal data will also be stored in the storage space. When the data needs to be called later, the missing data and abnormal data in the data will need to be reprocessed, which extends the data calling time. If the missing data and abnormal data are not processed when the data is called, the error rate of subsequent data analysis will increase. The same is true for digital engineering data. Therefore, the missing data of the digital engineering data is filled and the abnormal data is modified before storage to ensure the integrity and correctness of the digital engineering data, shorten the subsequent calling time of the digital engineering data, and at the same time, improve the accuracy of the subsequent data analysis.

[0073] Example 1

[0074] Obtaining a data set to be managed, performing a position analysis on the data set to be managed based on the analysis management terminal, and determining the position of the first data to be filled specifically includes the following steps:

[0075] S101. Based on the analysis management terminal, data is read and processed from the database system to obtain the data set to be managed;

[0076] S102: Based on the analysis management terminal, the management data set is traversed and processed to determine the location of missing data;

[0077] S103: Based on the analysis management terminal, information reading and processing are performed on the management data set based on the location of missing data, and row and column information of the location of missing data is obtained;

[0078] S104: The analysis management terminal generates a missing location label based on the row information and column information of the data missing location;

[0079] S105: Based on the analysis management terminal, embed the missing position tag into the data missing position, and set the data missing position containing the missing position tag as the first data position to be filled;

[0080] In this embodiment, when numerical data is collected, it is recorded according to a certain rule. Data with the same characteristics or the same information are placed in the same row. After the data is collected, it needs to be recorded. The data may be missed or recorded incorrectly during the recording process. Therefore, there will be missing data and abnormal data. Therefore, the missing data needs to be filled. The location of the missing data needs to be determined first. Therefore, the data set to be managed is traversed to determine the missing data location. Then, a missing location label is generated based on the row information and column information of the missing data location, and then the missing location label is placed at the missing data location. Because when the data is missing, the staff may think that there is an extra space at the missing data location, resulting in an extra blank location. Therefore, the missing location label is used to remind the staff that the data is missing. The above method is to avoid the situation where the staff performs other tasks after the missing data location is determined, and another staff member performs the subsequent task. The staff member may not know the processing method of the previous staff member and will delete the missing data location, which will lead to incomplete data in the digital intelligence project.

[0081] Example 2

[0082] The analysis management terminal performs abnormal analysis on the data set to be managed based on the first data location to be filled, and determines the second data location to be filled and the data set to be filled, specifically including the following steps:

[0083] S201, obtaining a data set to be analyzed for abnormality;

[0084] S202: Based on the analysis management terminal, the data set to be analyzed for abnormality is calculated and processed to determine the location of the abnormal data;

[0085] S203, the analysis management terminal deletes data from the data set to be managed according to the abnormal data location, and determines a second data location to be filled and a data set to be filled;

[0086] Among them, S203, the analysis management terminal deletes data from the data set to be managed according to the abnormal data location, and determines the second data location to be filled and the data set to be filled, specifically including the following steps:

[0087] S2031. The analysis management terminal deletes abnormal data from the managed data set according to the abnormal data location to obtain a second deleted data set;

[0088] S2032, repeat steps S2011-S2014 and steps S20131-S20133 to obtain a third pruned data set;

[0089] S2033, repeat steps S103-S105, obtain the second data location to be filled, and set the third deleted data set as the data set to be filled;

[0090] In this embodiment, after the abnormal data is determined, the abnormal data needs to be replaced. Since the abnormal data is the result of an error, the abnormal data has no correlation with other data in the row where the abnormal data is located. Therefore, the abnormal data is deleted. However, after the abnormal data is deleted, a blank data bit will be left. At this time, the blank data bit left by the abnormal data is equivalent to missing data. Therefore, the blank data bit is processed using the same processing method as the data missing position to determine the second data position to be filled and the data set to be filled.

[0091] Example 3

[0092] Obtaining the data set for anomaly analysis specifically includes the following steps:

[0093] S2011: Based on the analysis management terminal, read and process the missing position tag of each first data position to be filled to obtain row information of each missing data;

[0094] S2012. Based on the analysis management terminal, count the row information of each missing data to obtain the number of missing data in each row;

[0095] It can be understood that the data of each row is traversed first, and when it is determined that the data at a certain position is missing, the counter is used to count;

[0096] S2013. Based on the analysis management terminal, determine the number of missing data in each row and determine a first deleted data set;

[0097] S2014: Based on the analysis management terminal, delete the missing data in the first deleted data set, and set the remaining data as the data set to be analyzed for anomalies;

[0098] S2013, based on the analysis management terminal, determines the number of missing data in each row to determine the first deleted data set, specifically includes the following steps:

[0099] S20131. Based on the analysis management terminal, determine the number of missing items in each row of data and set a missing item threshold;

[0100] S20132. If the number of missing items in each row of data is greater than or equal to the set missing item threshold, the row of data is too missing and data filling is impossible. The row of data is deleted, and the remaining data is set as the first deleted data set.

[0101] S20133. If the number of missing rows in each row of data is less than the set missing number threshold, the row of data has few missing rows and can be filled with subsequent data. All rows of data with less than the set missing number threshold are set as the first deleted data set.

[0102] In this embodiment, when too much data is missing from a row, the missing data cannot be supplemented based on the remaining data. This is because too much data is missing and the correlation between the data cannot be obtained, so subsequent supplementation is impossible. The row of data with too much missing data does not have much reference significance in the subsequent analysis process. Therefore, in order to reduce the usage rate of storage space, the row of data with too much missing data is deleted.

[0103] Example 4

[0104] Based on the analysis management terminal, the data set to be analyzed for abnormality is processed and the location of abnormal data is determined, which specifically includes the following steps:

[0105] S2021. Based on the analysis management terminal, perform multiple calculations on the data set to be analyzed for abnormality to obtain the standard deviation and mean of the data set to be analyzed for abnormality;

[0106] S2022. Based on the analysis management terminal, calculate and process each data, standard deviation, and mean in the data set to be analyzed for abnormality, and obtain information to be verified for each data;

[0107] The calculation formula for obtaining the information to be verified for each data is:

[0108]

[0109] Where, The information to be verified for each data in each row; For each data in each row of the data set to be analyzed for anomalies; is the mean of each row in the data set to be analyzed for anomalies; is the standard deviation of each row in the data set to be analyzed for anomalies;

[0110] It is understandable that the above calculation formula is calculated for each data in each row, and the standard deviation and mean are the standard deviation and mean of the row. After the calculation of each data in the row is completed, the standard deviation and mean corresponding to the to-be-verified information of each data in the next row need to be calculated. The standard deviation and mean of the data in the next row may be the same as or different from the standard deviation and mean of the data in the previous row.

[0111] S2023. Based on the analysis management terminal, the information to be verified for each data is judged and processed to determine the location of abnormal data;

[0112] It is understandable that abnormal data can be determined by the Z-Score method, that is, the data set to be analyzed for abnormality is calculated by the Z-Score method to obtain the information to be verified for each data, and then the information to be verified for each data is judged to determine the abnormal data;

[0113] Among them, S2023, based on the analysis management terminal, judges and processes the information to be verified for each data, and determines the location of abnormal data, specifically including the following steps:

[0114] S20231. Based on the analysis management terminal, determine and process the information to be verified for each data and the set verification data threshold;

[0115] S20232: If the absolute value of the information to be verified is greater than the set verification data threshold, the data corresponding to the information to be verified is abnormal data, and the row information and column information of the data are recorded and set as the abnormal data position;

[0116] S20233. If the absolute value of the information to be verified is less than or equal to the set verification data threshold, the data corresponding to the information to be verified is normal data;

[0117] In this embodiment, the specific data value of the verification data threshold is set to 3. When the absolute value of the information to be verified is greater than 3, it means that the data corresponding to the information to be verified is abnormal data, and it is necessary to record the row information and column information of the data, and set the row information and column information of the data to the abnormal data position. When the absolute value of the information to be verified is less than or equal to 3, the data corresponding to the information to be verified is normal data. It can be understood that in the Z-Score method, data greater than 3 or less than -3 can be regarded as abnormal data. In order to avoid setting two comparison thresholds, the absolute value of the information to be verified is calculated, making the comparison and judgment process more convenient and quick.

[0118] Example 5

[0119] Based on the analysis management terminal, data is filled in the to-be-filled data set according to the first to-be-filled data position and the second to-be-filled data position, and obtaining a normal data set specifically includes the following steps:

[0120] S301. Based on the analysis management terminal, perform data standardization on the data set to be filled;

[0121] It is understood that data normalization is to map the data to In the interval, since the subsequent similarity calculation is performed by the K-nearest neighbor algorithm, and the K-nearest neighbor algorithm is calculated based on the distance metric, it is necessary to perform data standardization on the filled data set;

[0122] S302: Based on the first data position to be filled and the second data position to be filled, respectively, calculate and process the remaining data to obtain a first data similarity set and a second data similarity set;

[0123] It can be understood that the calculation method of this step is to calculate by Euclidean distance;

[0124] The specific calculation formula for obtaining the first data similarity set is:

[0125]

[0126] Where, is the similarity data in the first data similarity set; and is the value of the kth feature in the i-th row and all other rows j; n is the number of features;

[0127] S303: sorting the data in the first data similarity set and the second data similarity set based on the minimum function;

[0128] S304: Based on a random number function, select the lengths of the first data similarity set and the second data similarity set respectively to obtain a first data length and a second data length;

[0129] S305: Based on the analysis management terminal, select data from the first data similarity set and the second data similarity set according to the first data length and the second data length, respectively, to obtain a first data similarity subset and a second data similarity subset;

[0130] It can be understood that the smaller the data in the first data similarity set and the second data similarity set, the higher the correlation between the data in the missing row and the missing data. Therefore, they are sorted by the minimum value function, and then the length is selected by the random number function. The first data similarity subset and the second data similarity subset are selected from small to large. The first data length and the second data length specifically refer to the number of data selected;

[0131] S306: Based on the analysis management terminal, respectively calculate the mean of the data in the first data similarity subset and the second data similarity subset to obtain first filling data and second filling data;

[0132] S307 , based on the analysis management terminal, filling the first filling data and the second filling data into the first data position to be filled and the second data position to be filled, respectively, to obtain a data set to be verified;

[0133] S308: Based on the analysis management terminal, verify the data set to be verified and obtain the first data adjustment length and the second data adjustment length;

[0134] S309: Based on the analysis management terminal, recalculate and fill the first data similarity set and the second data similarity set according to the first data adjusted length and the second data adjusted length to obtain a normal data set;

[0135] In this embodiment, the specific steps for obtaining the first filling data and the second filling data by using the K-nearest neighbor algorithm are as follows:

[0136] The first step is to standardize the data to be filled so that the data to be filled can be mapped to In the interval, it can be understood that the K nearest neighbor algorithm is for numerical data, and the data inside the data to be filled are all numerical data;

[0137] Step 2: Calculate the distance between the missing row data and the remaining rows data using Euclidean distance to obtain a first data similarity set and a second data similarity set. It is understood that similarity data specifically refers to the distance between two data.

[0138] Step 3: Sort the first data similarity set and the second data similarity set by using the minimum function. The smaller the value in the first data similarity set and the second data similarity set, the smaller the distance between the two data, which indirectly indicates that the correlation between the two data is stronger.

[0139] Step 4: Use the random number function to select the appropriate data length. The data length is specifically the number of selected data, and the data selection rule is to select from small to large.

[0140] Step 5: Calculate the mean of the selected data to obtain the first filling data and the second filling data;

[0141] It is understandable that when the data length selected by the random number function is too long, some larger data will be filtered out. This is because larger data means that the correlation between the two data is smaller, which will cause the padding data to be too large. When subsequent data analysis is required, it will indirectly increase the error rate. Therefore, it is necessary to verify the padding data and adjust the first data length and the second data length.

[0142] S308, based on the analysis management terminal, verifies the data set to be verified and obtains the first data adjustment length and the second data adjustment length, specifically includes the following steps:

[0143] S3081. Based on the analysis management terminal, respectively perform calculations on the validation data set to obtain the mean of the row where the first filling data is located and the mean of the row where the second filling data is located;

[0144] S3082: Based on the analysis management terminal, respectively, determine and process the mean of the row where the first filling data is located and the mean of the row where the first filling data and the second filling data are located;

[0145] S3083: If the mean value of the row where the first padding data is located is much larger or much smaller than the first padding data, adjust the length of the first data based on the analysis management terminal to obtain the adjusted length of the first data;

[0146] S3084: If the mean value of the row where the first padding data is located is approximately equal to the first padding data, there is no need to adjust the length of the first data;

[0147] S3085: If the mean value of the row where the second padding data is located is much larger or much smaller than the second padding data, based on the analysis management terminal, adjust the length of the second data to obtain an adjusted length of the second data;

[0148] S3086: If the mean value of the row where the second padding data is located is approximately equal to the second padding data, there is no need to adjust the length of the second data;

[0149] In this embodiment, if the length of the data selected by the random number function is too long, some larger data will be screened out, which will cause the padding data to be too large. If the length of the data selected by the random number function is too short, too little data will be screened out, which will cause the padding data to be too small. Therefore, by verifying the first padding data and the second padding data, it is determined whether the first padding data and the second padding data meet the standards. If they do not meet the standards, the length of the data selected by the random number function is adjusted, and then the judgment is made again until suitable data is screened out. The above method reduces the error rate generated by subsequent data analysis, while also ensuring the integrity of the data, so that there are enough samples for subsequent data analysis.

[0150] Reference Figure 2 As shown, a data analysis and management system based on digital intelligence engineering is used to implement the above-mentioned data analysis and management method based on digital intelligence engineering, including:

[0151] An analysis and management terminal is used to control each module to perform location analysis, anomaly analysis, and data filling on the data set to be managed, and obtain a normal data set. The analysis and management terminal is used to control information interaction and data transmission between each module;

[0152] A database system, wherein the database system is used to store the data set to be managed;

[0153] a data missing location determination module, the data missing location determination module being used to traverse the data set to be managed, read information, and determine the first location of data to be filled;

[0154] a data anomaly location determination module, the data anomaly location determination module being used to perform data deletion and data calculation on the data set to be managed, and determine the second data location to be filled and the data set to be filled;

[0155] A filling data calculation module, wherein the filling data calculation module performs similarity calculation, length selection, and data mean calculation on the data set to be filled according to the first data position to be filled and the second data position to be filled, to obtain the data set to be verified;

[0156] The filling data verification module is used to perform data verification on the data set to be verified, perform secondary calculation of the filling data, and obtain a normal data set.

[0157] Furthermore, a storage medium is proposed, on which a computer program is stored. When the computer program is called and run, it executes a data analysis and management method based on digital intelligence engineering as described above, wherein the storage medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; an optical medium, such as a DVD; or a semiconductor medium, such as a solid-state drive (SSD).

[0158] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A data analysis and management method based on digital intelligence engineering, characterized in that: include: Acquire a data set to be managed, perform a position analysis on the data set to be managed based on the analysis management terminal, and determine the position of the first data to be filled; The analysis management terminal performs an abnormality analysis on the data set to be managed according to the first data position to be filled, and determines the second data position to be filled and the data set to be filled; Based on the analysis management terminal, data is filled in the to-be-filled data set according to the first to-be-filled data position and the second to-be-filled data position, and obtaining a normal data set specifically includes the following steps: Based on the analysis management terminal, the data set to be filled is standardized; Based on the first data position to be filled and the second data position to be filled, respectively, calculating and processing the remaining data to obtain a first data similarity set and a second data similarity set; Based on the minimum function, sorting the data in the first data similarity set and the second data similarity set respectively; Based on the random number function, length selection is performed on the first data similarity set and the second data similarity set respectively to obtain the first data length and the second data length; Based on the analysis management terminal, data is selected from the first data similarity set and the second data similarity set according to the first data length and the second data length, to obtain a first data similarity subset and a second data similarity subset; Based on the analysis management terminal, respectively calculating the mean of the data in the first data similarity subset and the second data similarity subset to obtain first filling data and second filling data; Based on the analysis management terminal, the first filling data and the second filling data are filled into the first data position to be filled and the second data position to be filled, respectively, to obtain a data set to be verified; Based on the analysis management terminal, verify the data set to be verified and obtain the first data adjustment length and the second data adjustment length; Based on the analysis management terminal, data calculation and data filling processing are re-performed on the first data similarity set and the second data similarity set according to the first data adjustment length and the second data adjustment length to obtain a normal data set.

2. The data analysis and management method based on digital intelligence engineering according to claim 1 is characterized in that: The step of obtaining the data set to be managed and performing position analysis on the data set to be managed based on the analysis management terminal to determine the position of the first data to be filled specifically includes the following steps: Based on the analysis and management terminal, the database system is read and processed to obtain the data set to be managed; Based on the analysis management terminal, the management data set is traversed and processed to determine the location of missing data; Based on the analysis management terminal, the information of the management data set to be managed is read and processed based on the location of missing data, and the row and column information of the missing data location is obtained; The analysis management terminal generates a missing location label based on the row and column information of the data missing location; Based on the analysis management terminal, a missing position tag is embedded in the data missing position, and the data missing position containing the missing position tag is set as the first data position to be filled.

3. The data analysis and management method based on digital intelligence engineering according to claim 1 is characterized in that: The analysis management terminal performs an abnormality analysis on the data set to be managed based on the first data position to be filled, and determines the second data position to be filled and the data set to be filled, specifically comprising the following steps: Obtain the data set to be analyzed for anomalies; Based on the analysis and management terminal, the data set to be analyzed for abnormality is calculated and processed to determine the location of abnormal data; The analysis management terminal deletes data from the to-be-managed data set according to the abnormal data location, and determines a second to-be-filled data location and a to-be-filled data set.

4. The data analysis and management method based on digital intelligence engineering according to claim 3 is characterized in that: The step of obtaining the data set to be analyzed for abnormality specifically includes the following steps: Based on the analysis management terminal, information reading and processing is performed on the missing position tag of each first data position to be filled, and row information of each missing data is obtained; Based on the analysis management terminal, count the row information of each missing data to obtain the number of missing data in each row; Based on the analysis management terminal, the number of missing data in each row is judged and processed to determine the first deleted data set; Based on the analysis management terminal, the missing data in the first deleted data set is deleted, and the remaining data is set as the data set to be analyzed for abnormality.

5. The data analysis and management method based on digital intelligence engineering according to claim 4 is characterized in that: The analysis management terminal determines the number of missing data in each row and determines the first deleted data set, which specifically includes the following steps: Based on the analysis and management terminal, the number of missing data in each row and the set missing number threshold are judged and processed; If the number of missing rows in each row is greater than or equal to the set missing number threshold, the row of data is missing too much and cannot be filled in. The row of data is deleted and the remaining data is set as the first deleted data set; If the number of missing rows in each row of data is less than the set missing number threshold, the row of data has few missing rows and can be filled with subsequent data. All row data with less than the set missing number threshold are set as the first deleted data set.

6. The data analysis and management method based on digital intelligence engineering according to claim 3 is characterized in that: The analysis management terminal performs computational processing on the data set to be analyzed for abnormality and determines the location of abnormal data, specifically including the following steps: Based on the analysis management terminal, multiple calculations are performed on the data set to be analyzed for abnormality to obtain the standard deviation and mean of the data set to be analyzed for abnormality; Based on the analysis management terminal, each data, standard deviation and mean in the data set to be analyzed for abnormality are calculated and processed to obtain the information to be verified for each data; Based on the analysis and management terminal, the information to be verified for each data is judged and processed to determine the location of abnormal data.

7. The data analysis and management method based on digital intelligence engineering according to claim 6 is characterized in that: The analysis and management terminal is used to judge and process the information to be verified for each data and determine the location of abnormal data, which specifically includes the following steps: Based on the analysis and management terminal, the information to be verified for each data and the set verification data threshold are judged and processed; If the absolute value of the information to be verified is greater than the set verification data threshold, the data corresponding to the information to be verified is abnormal data, and the row information and column information of the data are recorded and set as the abnormal data position; If the absolute value of the information to be verified is less than or equal to the set verification data threshold, the data corresponding to the information to be verified is normal data.

8. The data analysis and management method based on digital intelligence engineering according to claim 1 is characterized in that: The verification process of the verification data set based on the analysis management terminal to obtain the first data adjustment length and the second data adjustment length specifically includes the following steps: Based on the analysis management terminal, the verification data set is respectively calculated and processed to obtain the mean of the row where the first filling data is located and the mean of the row where the second filling data is located; Based on the analysis management terminal, respectively, the mean of the row where the first filling data is located and the mean of the row where the first filling data and the second filling data are located are judged and processed; If the mean value of the row where the first padding data is located is much larger or much smaller than the first padding data, adjusting the length of the first data based on the analysis management terminal to obtain the adjusted length of the first data; If the mean value of the row where the first padding data is located is approximately equal to the first padding data, there is no need to adjust the length of the first data; If the mean value of the row where the second padding data is located is much larger or much smaller than the second padding data, the length of the second data is adjusted based on the analysis management terminal to obtain the adjusted length of the second data; If the mean value of the row where the second padding data is located is approximately equal to the second padding data, there is no need to adjust the length of the second data.

9. A data analysis and management system based on digital intelligence engineering, used to implement a data analysis and management method based on digital intelligence engineering according to any one of claims 1 to 8, characterized in that: include: An analysis and management terminal is used to control each module to perform location analysis, anomaly analysis, and data filling on the data set to be managed, and obtain a normal data set. The analysis and management terminal is used to control information interaction and data transmission between each module; A database system, wherein the database system is used to store the data set to be managed; A data missing location determination module, configured to perform data traversal and information reading on a data set to be managed, and determine a first location of data to be filled; a data anomaly location determination module, the data anomaly location determination module being used to perform data deletion and data calculation on the data set to be managed, and determine a second data location to be filled and a data set to be filled; A filling data calculation module, wherein the filling data calculation module performs similarity calculation, length selection, and data mean calculation on the data set to be filled according to the first data position to be filled and the second data position to be filled, to obtain the data set to be verified; The filling data verification module is used to perform data verification on the data set to be verified, perform secondary calculation of the filling data, and obtain a normal data set.

Images