An artificial intelligence digital technology system based on big data analysis

Abstract

The present invention discloses an artificial intelligence digital technology system based on big data analysis, including: a data format processing module: sending data from different data sources to a data format processing receiving system at intervals, and performing unified format conversion processing to make the converted data format consistent; a missing data judgment module: further analyzing whether there is data loss in the data from different data sources according to the preliminary classification of different data sources. This artificial intelligence digital technology system based on big data analysis, by sending data from different data sources to a data format processing receiving system at intervals, and performing unified format conversion processing, is convenient for unified analysis and management of data; while completing the unified conversion of data formats, it can also classify and mark data from different data sources, and can simultaneously analyze and delete lost data, thereby improving data quality and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data analysis, and in particular relates to an artificial intelligence digital technology system based on big data analysis. Background Art

[0002] With the advent of the big data era, the amount of data collected by enterprises and institutions has grown exponentially. How to effectively utilize these massive amounts of data has become one of the hot topics in the current digital technology research field. The application of artificial intelligence technology has also provided new ways and methods for data analysis.

[0003] Patent application number CN111177220B discloses a data analysis method based on big data, including the following steps: extracting and merging the initial data in the data result table to obtain a merged table, and importing the merged table into the big data platform through the data extraction tool Sqoop; performing data cleaning on the preprocessed data in the preprocessed data result table through the big data platform to obtain an apportionment result data table; splitting the apportionment result data table to obtain a split data result table set and importing it into the distributed memory real-time analysis system Druid to obtain an analysis result set, and if the deviation value is greater than or equal to a preset threshold, mining the initial data through a decision tree to obtain a mining result. The present invention also discloses a data analysis device, equipment and computer-readable storage medium based on big data. The data analysis method based on big data provided by the present invention improves the effect of data analysis.

[0004] The massive amounts of data collected come from different data sources, and the data formats of different data sources may be different, which makes it inconvenient to subsequently conduct unified analysis and management of the data. In addition, a large portion of the massive amounts of data collected is lost, and there will be a lot of blank data, which affects the data quality and accuracy. Therefore, the present invention provides an artificial intelligence digital technology system based on big data analysis. Summary of the invention

[0005] The purpose of the present invention is to provide an artificial intelligence digital technology system based on big data analysis to solve the technical problems raised in the background technology.

[0006] To achieve the above objectives, the specific technical solution of the present invention is as follows: An artificial intelligence digital technology system based on big data analysis, comprising:

[0007] Data acquisition module: uses acquisition equipment to collect data and transmits the data to the data processing format module;

[0008] Data format processing module: sends data from different data sources to the data format processing receiving system at intervals, performs unified format conversion processing, makes the converted data format consistent, and marks the converted data according to the different data sources, completing the preliminary classification of different data sources;

[0009] Missing data judgment module: Based on the preliminary classification of different data sources, further analyze whether there is data missing from different data sources, mark the missing data sources obtained from the analysis as abnormal data, reclassify the preliminary analyzed data sources, and finally classify the data from different data sources;

[0010] Data defect value analysis module: further analyze the data source that exists in the corresponding time period where the interval time period is greater than the normal interval time, determine whether the amount of data lost by the data source meets the normal data loss threshold, and delete the data corresponding to the data source that exceeds the normal loss threshold.

[0011] Preferably, the specific method of converting data formats of different data sources is as follows:

[0012] S1. Send data from different data sources to the data format processing receiving system at intervals, and set the interval time to t. The specific interval time is set manually;

[0013] S2. Data format processing The receiving system calculates the interval time of data transmission and marks the interval time periods with an interval time greater than t as t q , t q ≥t, q=1, 2, 3...n, where q represents different intervals and n represents the number of intervals;

[0014] S3. Mark the front-end and back-end data of the marked interval time period into different data source tags, marked as DS p ,p=1,2,3......n+1, p represents different data sources, n+1 represents the number of data sources, such as the marked results are: DS1, t1, DS2, t2, DS3, t3......DS n ,t n DS n+1 , complete the preliminary classification of different data sources.

[0015] Preferably, the specific method of analyzing whether there is data loss in data from different data sources is as follows:

[0016] T1, t q Make judgment analysis with t. If t q = t, indicating that this time period is the normal interval input of different data sources. If tq >t, it means that the interval is longer than the normal data transmission interval, indicating that there is blank data during this period, and the data source where the blank data exists is marked as abnormal, that is, marked as DS YC p ;

[0017] T2, there will be blank data t q The corresponding backend data source is divided into t q The data source of the front end is re-labeled. For example, when t2>t, it means that there is blank data corresponding to the time period t2. The results of labeling different data sources are: DS1, t1, DS2, t2, DS2, t3, DS3, .....DS n-1 ,t n DS n If t n The number of blank times in the front end is k, and the data source of the back end at this time point is marked as: DS n+1-k , thus completing the final classification of data from different data sources.

[0018] Preferably, the specific analysis method of the data defect value analysis module is as follows:

[0019] D1. Find the one marked as DS YC p The actual transmission time of the data source t q ;

[0020] D2. Set the normal loss threshold DS 正常 , use the formula to calculate the actual data loss threshold DS 实际 , if DS 实际 >DS 正常 , then delete the data in the data source. If DS 实际 ≦DS 正常 , then the data of the data source is retained;

[0021] DS 实际 The calculation formula is: DS 实际 =(t q -t)*j;

[0022] Where j is the loss threshold coefficient, j is the considered set value;

[0023] D3. For DS 实际 >DS 正常 The abnormal data is given an abnormal characteristic attribute.

[0024] Preferably, the unified format conversion process can be performed using one of the Pandas, NumPy and csv modules.

[0025] The artificial intelligence digital technology system based on big data analysis of the present invention has the following advantages: the artificial intelligence digital technology system based on big data analysis, by periodically sending data from different data sources to a data format processing and receiving system, performs unified format conversion processing, thereby facilitating unified analysis and management of data; while completing the unified conversion of data formats, it can also classify and mark data from different data sources, and can simultaneously analyze and delete lost data, thereby improving data quality and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a schematic diagram of the overall structure of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or a communication; it can be a direct connection, or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0030] The disclosure below provides many different embodiments or examples to implement different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the embodiments of the present invention. In addition, the embodiments of the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0031] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of an artificial intelligence digital technology system based on big data analysis of the present invention in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, an artificial intelligence digital technology system based on big data analysis of the present invention includes:

[0033] Data acquisition module: uses acquisition equipment to collect data and transmits the data to the data processing format module;

[0034] Data format processing module: The data from different data sources are sent to the data format processing receiving system at intervals for unified format conversion. The data format can be converted using one of the Pandas, NumPy and csv modules, or a custom encoding and decoding algorithm can be used to convert the data format uniformly, so that the converted data format is consistent, and the format-converted data can be marked according to the different data sources, completing the preliminary classification of different data sources, so that subsequent analysis applications can clearly know the source of the data and ensure data accuracy and reliability;

[0035] The specific methods for converting data formats from different data sources are as follows:

[0036] S1. Send data from different data sources to the data format processing receiving system at intervals, and set the interval time to t. The specific interval time is set manually. For example, the interval time is set to 5s, that is, t=5;

[0037] S2. Data format processing The receiving system calculates the interval time of data transmission and marks the interval time periods with an interval time greater than t as t q , t q ≥t, q=1, 2, 3...n, where q represents different intervals and n represents the number of intervals;

[0038] S3. Mark the front-end and back-end data of the marked interval time period into different data source tags, marked as DS p ,p=1,2,3......n+1, p represents different data sources, n+1 represents the number of data sources, such as the marked results are: DS1, t1, DS2, t2, DS3, t3......DS n ,t n DS n+1 , complete the preliminary classification of different data sources;

[0039] Missing data judgment module: Based on the preliminary classification of different data sources, further analyze whether there is data missing from different data sources, mark the missing data sources obtained from the analysis as abnormal data, reclassify the preliminary analyzed data sources, and finally classify the data from different data sources;

[0040] The specific method for analyzing whether there is data loss in data from different data sources is as follows:

[0041] T1, t q Make judgment analysis with t. If t q = t, indicating that this time period is the normal interval input of different data sources. If t q >t, it means that the interval is longer than the normal data transmission interval, indicating that there is blank data during this period, and the data source where the blank data exists is marked as abnormal, that is, marked as DS YC p ;

[0042] T2, there will be blank data t q The corresponding backend data source is divided into t q The data source of the front end is re-labeled. For example, when t2>t, it means that there is blank data corresponding to the time period t2. The results of labeling different data sources are: DS1, t1, DS2, t2, DS2, t3, DS3, .....DS n-1 ,t n DS n If t n The number of blank times in the front end is k, and the data source of the back end at this time point is marked as: DS n+1-k , thereby completing the final classification of data from different data sources;

[0043] Data defect value analysis module: further analyzes the data source in the corresponding time period with an interval time period longer than the normal interval time, determines whether the amount of data lost by the data source meets the normal data loss threshold, and deletes the data source corresponding to the data source that exceeds the normal loss threshold, because it is normal for data from different data sources to have a small amount of data loss;

[0044] The specific analysis method of the data defect value analysis module is as follows:

[0045] D1. Find the one marked as DS YC p The actual transmission time of the data source t q ;

[0046] D2. Set the normal loss threshold DS 正常, use the formula to calculate the actual data loss threshold DS 实际 , if DS 实际 >DS 正常 , indicating that there are too many missing data, delete the data in the data source. If DS 实际 ≦DS 正常 , indicating that there is indeed a small amount of data loss, which is within the normal range of data loss and does not affect the judgment of the data, so the data of the data source is retained;

[0047] DS 实际 The calculation formula is: DS 实际 =(t q -t)*j;

[0048] Where j is the loss threshold coefficient, j is the considered set value;

[0049] D3. For DS 实际 >DS 正常 The abnormal data is given an abnormal characteristic attribute.

[0050] For data with attached abnormal characteristic attributes, the abnormal data monitoring model and automatic recognition technology can be used to automatically identify the marked abnormal characteristic attributes and automatically delete the corresponding abnormal data.

[0051] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0052] The working principle of the artificial intelligence digital technology system based on big data analysis is as follows: use the acquisition equipment to collect data and transmit the data to the data processing format module. The data format processing module processes the data and sends the data from different data sources to the data format processing receiving system at intervals. The interval time is set to t, and the data in different formats are uniformly converted to make the converted data formats consistent. The format-converted data are marked according to the different data sources, and the preliminary classification of different data sources is completed. According to the preliminary classification of different data sources, the data from different data sources are further analyzed to see whether there is data loss, and the lost data sources obtained by the analysis are marked as abnormal data, and the data sources obtained by the preliminary analysis are reclassified, and the data from different data sources are finally classified. The data sources that exist in the corresponding time period whose interval time period is greater than the normal interval time are further analyzed to determine whether the amount of data lost by the data source meets the normal data loss threshold, and the data of the data source corresponding to the data loss threshold that exceeds the normal loss threshold is deleted.

[0053] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An artificial intelligence digital technology system based on big data analysis, characterized in that: include: Data acquisition module: uses acquisition equipment to collect data and transmits the data to the data processing format module; Data format processing module: sends data from different data sources to the data format processing receiving system at intervals, performs unified format conversion processing, makes the converted data format consistent, and marks the converted data according to the different data sources, completing the preliminary classification of different data sources; Missing data judgment module: Based on the preliminary classification of different data sources, further analyze whether there is data missing from different data sources, mark the missing data sources obtained from the analysis as abnormal data, reclassify the preliminary analyzed data sources, and finally classify the data from different data sources; The specific methods for converting data formats from different data sources are as follows: S1. Send data from different data sources to the data format processing receiving system at intervals, and set the interval time to t. The specific interval time is set manually; S2. Data format processing The receiving system calculates the interval time of data transmission and marks the interval time periods with an interval time greater than t as t q , t q ≥t, q=1, 2, 3...n, where q represents different intervals and n represents the number of intervals; S3. Mark the front-end and back-end data of the marked interval time period into different data source tags, marked as DS p ,p=1,2,3......n+1, p represents different data sources, n+1 represents the number of data sources, such as the marked results are: DS1, t1, DS2, t2, DS3, t3......DS n ,t n DS n+1 , complete the preliminary classification of different data sources; The specific method for analyzing whether there is data loss in data from different data sources is as follows: T1, t q Make judgment analysis with t. If t q = t, indicating that this time period is the normal interval input of different data sources. If t q >t, it means that the interval is longer than the normal data transmission interval, indicating that there is blank data in this time period, and the data source where the blank data exists is marked as abnormal, that is, marked as DS YC p ; T2, there will be blank data t q The corresponding backend data source is divided into t q The front-end data source is re-labeled.

2. The artificial intelligence digital technology system based on big data analysis according to claim 1, characterized in that: The specific method of re-marking the data source is as follows: when t2>t, it means that there is blank data corresponding to the time period t2. The results of marking different data sources are: DS1, t1, DS2, t2, DS2, t3, DS3, .....DS n-1 ,t n DS n If t n The number of blank times in the front end is k, and the data source of the back end at this time point is marked as: DS n+1-k , thus completing the final classification of data from different data sources.

3. The artificial intelligence digital technology system based on big data analysis according to claim 2 further includes: Data defect value analysis module: further analyze the data source that exists in the corresponding time period where the interval time period is greater than the normal interval time, determine whether the amount of data lost by the data source meets the normal data loss threshold, and delete the data corresponding to the data source that exceeds the normal loss threshold.

4. The artificial intelligence digital technology system based on big data analysis according to claim 3 is characterized by: The specific analysis method of the data defect value analysis module is as follows: D1. Find the one marked as DS YC p The actual transmission time of the data source t q ; D2. Set the normal loss threshold DS 正常 , use the formula to calculate the actual data loss threshold DS 实际 , if DS 实际 >DS 正常 , then delete the data in the data source. If DS 实际 ≦DS 正常 , then the data of the data source is retained; DS 实际 The calculation formula is: DS 实际 =(t q -t)*j; Where j is the loss threshold coefficient, j is the considered set value; D3. For DS 实际 >DS 正常 The abnormal data is given an abnormal characteristic attribute.

5. The artificial intelligence digital technology system based on big data analysis according to claim 1 is characterized by: The unified format conversion process is performed using one of the Pandas, NumPy, and csv modules.

6. The artificial intelligence digital technology system based on big data analysis according to claim 1 is characterized by: It includes an abnormal data monitoring model, which automatically identifies the marked abnormal feature attributes through automatic recognition technology and automatically deletes the corresponding abnormal data.

Images