A data intelligent analysis method based on distributed processing
By filtering and filling data in the encrypted database of the distributed data processing center, the problems of low data processing efficiency and insufficient fault tolerance in the distributed database system are solved, and more efficient data processing and fault tolerance are achieved.
Patent Information
- Application Number
- CN202110775157.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-07-07
AI Technical Summary
The distributed database system has high overhead and complex access structures when processing data, resulting in low data processing efficiency and inability to effectively distinguish and process errors or lossy data, increasing the load on running components.
Establish an encrypted database in a distributed data processing center, filter data through key management and encryption authentication, delete error data, fill lossy data, and use hot and cold data storage nodes to manage lossy data, and use multiple missing value filling methods to fill , improve data processing efficiency and system fault tolerance.
By classifying and filling lossy data, data processing efficiency is improved, operating components load is reduced, and the system's fault tolerance and data processing accuracy are improved.
Smart Images

Figure CN113672379B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed processing, and in particular to a data intelligent analysis method based on distributed processing. Background Art
[0002] The distributed database system is developed on the basis of the centralized database system and is the product of the combination of computer technology and network technology. The distributed database system is suitable for decentralized departments of the unit, allowing each department to store its commonly used data locally and implement local storage and local use, thereby improving response speed and reducing communication costs. Compared with the centralized database system, the distributed database system is scalable and improves the reliability of the system by adding appropriate data redundancy. The distributed database system has a flexible architecture and can adapt to distributed management and control organizations. The system has high reliability and good availability, fast response speed of local applications, and superior economic performance, good scalability, and easy integration with existing systems. However, the system overhead of the distributed database is relatively large, and it has a complex access structure. If the data is processed indiscriminately after obtaining the data processing task, it will not only put pressure on the operating components, but also reduce the efficiency of data processing. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a data intelligent analysis method based on distributed processing, comprising the following steps:
[0004] S1: The distributed data processing center establishes an encrypted database, the key management center generates an encryption key, and encrypts the data in the encrypted database according to the encryption key;
[0005] S2: The client sends an access request to the distributed data processing center;
[0006] S3: After receiving the access request data, the distributed data processing center performs a demand analysis on the access request data according to the access information database pre-stored in the distributed data processing. If the analysis finds that the data in the access request data is erroneous data, the process proceeds to step S4. If the analysis finds that the data in the access request data is damaged data, the process proceeds to step S5. If the analysis finds that the data in the access request data is correct data, the process proceeds to step S6.
[0007] S4: Delete the wrong data;
[0008] S5: Fill missing values in the damaged data and send the filled access request data to the distributed data processing center for re-demand analysis;
[0009] S6: The distributed data processing center authenticates the access request data according to the access request data of the client and based on the public key set in its own encryption database. If the authentication is successful, the process proceeds to step S7. If the authentication is not successful, the access request data is determined to be erroneous data and the process proceeds to step S4.
[0010] S7: The distributed data processing center sends the access request data to the encryption server, and the encryption server verifies whether the client has the authority to access the requested database content. If the verification is successful, the process proceeds to step S8. If the verification is not successful, the access request data is determined to be erroneous data, and the process proceeds to step S4.
[0011] S8: The distributed data processing center extracts the encrypted file from the encrypted database, decrypts the encrypted file according to the encryption key, and transmits the decrypted data to the client through the secure layer protocol encrypted channel.
[0012] Preferably, after the access request data is judged as damaged data in step S3, the following steps are performed:
[0013] S31: The distributed data processing center stores the lossy data in the hot data storage node and sets the storage time s1;
[0014] S32: If the distributed processing center detects that the damaged data in the hot data storage node receives a filling request before the s1 time is fully counted, then the process goes to step S5 and is read. If the distributed processing center detects that the damaged data in the hot data storage node does not receive a filling request after the s1 time is fully counted, then the process goes to step S33;
[0015] S33: the distributed processing center stores the unread damaged data in the cold data storage node and sets the storage time s2, the value of which is selected in accordance with the following principle: s2>>s1;
[0016] S34: If the distributed processing center detects that the s2 duration has not been fully calculated, and the damaged data in the cold data storage node receives a filling request, then enter step S5. If the distributed processing center detects that the damaged data in the cold storage node has not been read after the s2 duration has been fully calculated, then enter step S4.
[0017] Preferably, the configuration of the cold data storage node is higher than the configuration of the hot data storage node.
[0018] Preferably, in step S5, when filling missing values for damaged data, one or more missing value filling methods including mean interpolation, interpolation using the same mean, maximum likelihood estimation, and multiple interpolation can be used.
[0019] Preferably, in step S6, when the distributed data processing center authenticates the access request data based on the key set in its own encryption database, it also includes the following steps:.
[0020] S61: According to the length of the padding data value of the correct data in step S3, the distributed data processing center sets a first rule to establish a first data column of the access request data, the first rule sorts the correct data from small to large according to the padding value length, and prioritizes the data in the first data column according to the first rule to obtain priority processing information in the first data column;
[0021] S62: According to the time point when the access request data is judged as correct data in step S3, the distributed data processing center sets a second rule, the second rule sorts the access request data from the earliest to the latest according to the time point when the access request data is judged as correct data, and sorts the first priority processing information according to the second rule to obtain the first processing data;
[0022] S63: The distributed data processing center processes the first data slices through the first distributed data processing node to generate multiple sub-data;
[0023] S64: extract identification information of the plurality of sub-data, allocate the second distributed data processing nodes for processing the plurality of sub-data, and obtain an authentication sequence for accessing the request data.
[0024] Preferably, the first distributed data processing node sends a first heartbeat packet to the distributed data processing center.
[0025] Preferably, the second distributed data processing node sends a second heartbeat packet to the distributed data processing center.
[0026] Preferably, the encryption granularity of the data in the encrypted database is the field data of each record.
[0027] The beneficial effects of the present invention are:
[0028] The system can classify the data according to the data processing requirements of the distributed data processing center, process the filtered useful data and lossy data, remove erroneous data, improve the efficiency of data processing of the distributed data processing center, reduce the load of the running components, and fill the lossy data before screening, thereby increasing the chance of filling the lossy data and improving the fault tolerance of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 Shown is a method flow chart of a data intelligent analysis method based on distributed processing. DETAILED DESCRIPTION
[0030] Below in conjunction with the appended Figure 1, clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by ordinary technicians in this field without creative work.
[0031] In the description of the present invention, it should be understood that the terms "counterclockwise", "clockwise", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.
[0032] like Figure 1 As shown, a data intelligent analysis method based on distributed processing includes the following steps:
[0033] S1: The distributed data processing center establishes an encrypted database, the key management center generates an encryption key, and encrypts the data in the encrypted database according to the encryption key;
[0034] S2: The client sends an access request to the distributed data processing center;
[0035] S3: After receiving the access request data, the distributed data processing center performs a demand analysis on the access request data according to the access information database pre-stored in the distributed data processing. If the analysis finds that the data in the access request data is erroneous data, the process proceeds to step S4. If the analysis finds that the data in the access request data is damaged data, the process proceeds to step S5. If the analysis finds that the data in the access request data is correct data, the process proceeds to step S6.
[0036] S4: Delete the wrong data;
[0037] S5: Fill missing values in the damaged data and send the filled access request data to the distributed data processing center for re-demand analysis;
[0038] S6: The distributed data processing center authenticates the access request data according to the access request data of the client and based on the public key set in its own encryption database. If the authentication is successful, the process proceeds to step S7. If the authentication is not successful, the access request data is determined to be erroneous data and the process proceeds to step S4.
[0039] S7: The distributed data processing center sends the access request data to the encryption server, and the encryption server verifies whether the client has the authority to access the requested database content. If the verification is successful, the process proceeds to step S8. If the verification is not successful, the access request data is determined to be erroneous data, and the process proceeds to step S4.
[0040] S8: The distributed data processing center extracts the encrypted file from the encrypted database, decrypts the encrypted file according to the encryption key, and transmits the decrypted data to the client through the secure layer protocol encrypted channel.
[0041] In step S3, the distributed data processing center compares the access request data with the internal pre-stored access information database. If there is no pre-stored access information with a similarity of more than 80%, it is judged as error information. If there is pre-stored access information with a similarity of more than 80% with the access request data in the access information database, the pre-stored access information with the highest similarity with the access request data is extracted, and then the access request data is byte-by-byte compared with the pre-stored access information. First, the byte missing ratio range is 0-10%. If there is no byte missing in the access request data, that is, the byte missing ratio is 0, the distributed data processing center determines the access request data as complete data and continues to compare. If the access request data byte missing value is between 0-10% of the total byte length of the access request data segment, the distributed data processing center determines the access request data as data that can be filled to reach the correct data, that is, lossy data. If the access request data byte missing value is greater than 10% of the total byte length of the access request data segment, the distributed data processing center determines the access request data as data that cannot be filled to reach the correct data, that is, error data.
[0042] The public key in step S6 is the only public key agreed upon in advance by the client and the distributed data processing center;
[0043] In step S7, the encryption server pre-stores a permission verification set, which includes information of several users whose permissions have been verified. The encryption server verifies whether the client has the permission to access the requested database content based on the permission verification set.
[0044] Furthermore, after the access request data is determined to be damaged data in step S3, the following steps are performed:
[0045] S31: The distributed data processing center stores the lossy data in the hot data storage node and sets the storage time s1;
[0046] S32: If the distributed processing center detects that the damaged data in the hot data storage node receives a filling request before the s1 time is fully counted, then the process goes to step S5 and is read. If the distributed processing center detects that the damaged data in the hot data storage node does not receive a filling request after the s1 time is fully counted, then the process goes to step S33;
[0047] S33: the distributed processing center stores the unread damaged data in the cold data storage node and sets the storage time s2, the value of which is selected in accordance with the following principle: s2>>s1;
[0048] S34: If the distributed processing center detects that the s2 duration has not been fully calculated, and the damaged data in the cold data storage node receives a filling request, then enter step S5. If the distributed processing center detects that the damaged data in the cold storage node has not been read after the s2 duration has been fully calculated, then enter step S4.
[0049] After receiving the filling request, the distributed data processing center first searches for the damaged data corresponding to the filling request in the hot storage center. If the damaged data corresponding to the filling request is not found in the hot data storage center, the distributed data processing center searches for damaged data corresponding to the filling request in the cold data storage center. If the damaged data is still not found, the filling request is deleted.
[0050] The unread lossy data is classified and stored according to the interval time of receiving the filling request to avoid a significant reduction in the computing speed when the distributed processing center receives the filling request and retrieves the corresponding lossy data due to too much storage data in a single storage node. Setting s2>>s1 can extend the retention time of the lossy data, increase the chance of filling the lossy data, and improve the fault tolerance of the system.
[0051] Furthermore, the configuration of the cold data storage node is higher than the configuration of the hot data storage node.
[0052] Since the storage duration s2>>s1 is set, the operating load of the cold data storage node is much higher than that of the hot data storage node. If the cold data storage node is poorly configured and has insufficient capacity, data transfer restrictions or data packet loss may occur when the lossy data is transferred from the hot data storage node to the cold data storage node.
[0053] Furthermore, in step S5, when filling missing values for damaged data, one or more missing value filling methods including mean interpolation, interpolation using the same type of mean, maximum likelihood estimation, and multiple interpolation can be used.
[0054] Different missing value filling methods can be selected according to the different missing situations of lossy data. Mean interpolation and interpolation using the same mean are suitable for situations where the missing values are randomly missing. Maximum likelihood estimation and multiple interpolation have little interference on the sample. When the interpolated values are regressed as explanatory variables, the deviation between the estimated value of the parameter and the true value is small.
[0055] Furthermore, in the step S6, when the distributed data processing center authenticates the access request data based on the key set in its own encryption database, the following steps are also included:.
[0056] S61: According to the length of the padding data value of the correct data in step S3, the distributed data processing center sets a first rule to establish a first data column of the access request data, the first rule sorts the correct data from small to large according to the padding value length, and prioritizes the data in the first data column according to the first rule to obtain priority processing information in the first data column;
[0057] S62: According to the time point when the access request data is judged as correct data in step S3, the distributed data processing center sets a second rule, the second rule sorts the access request data from the earliest to the latest according to the time point when the access request data is judged as correct data, and sorts the first priority processing information according to the second rule to obtain the first processing data;
[0058] S63: The distributed data processing center processes the first data slices through the first distributed data processing node to generate multiple sub-data;
[0059] S64: extract identification information of the plurality of sub-data, allocate the second distributed data processing nodes for processing the plurality of sub-data, and obtain an authentication sequence for accessing the request data.
[0060] After the first rule sorts the correct data according to the length of the padding value, the correct data with the same padding value length is the priority processing information of the same level, and is sorted in order from small to large according to the second rule, and the priority processing information with the least padding value is sorted first; after the second rule sorts the priority processing information according to the time point when it is judged to be correct data, since there is only one group of access request data received at the same time point, the access request data that is first judged to be correct data is the first processing data; the basis for division during sharding processing can be based on the byte length of the first processing data, the feature words of the data requested for access, etc., and sharding processing is beneficial to improving the efficiency of data processing and data transmission efficiency.
[0061] Furthermore, the first distributed data processing node sends a first heartbeat packet to the distributed data processing center.
[0062] The distributed data processing center can determine whether an abnormality occurs when the first distributed data processing node slices the first data based on the first heartbeat packet. Since the data processing interval is short and the monitoring requirements for the first distributed data processing node are high, the judgment time of the first heartbeat packet is set to 6-9 seconds to ensure the timeliness of the reporting of the operating status of the first distributed data processing node.
[0063] Further, the second distributed data processing node sends a second heartbeat packet to the distributed data processing center.
[0064] The distributed data processing center can determine whether the second distributed data processing node has an abnormality when processing sub-data based on the first heartbeat packet. Since the data processing interval is short, the monitoring requirements for the second distributed data processing node are high, so the judgment time of the second heartbeat packet is set to 6-9 seconds to ensure the timeliness of the second distributed data processing node's operating status reporting.
[0065] Furthermore, the encryption granularity of the data in the encrypted database is the field data of each record.
[0066] When the encryption granularity is each record field data, the system security and flexibility are the highest.
Claims
1. A data intelligent analysis method based on distributed processing, characterized in that: The following steps are involved: S1: The distributed data processing center establishes an encryption server, the key management center generates an encryption key, and encrypts the data in the encryption server according to the encryption key; S2: The client sends an access request to the distributed data processing center; S3: After receiving the access request data, the distributed data processing center performs a demand analysis on the access request data according to the access information database pre-stored in the distributed data processing. If the analysis finds that the data in the access request data is erroneous data, then proceed to step S4. If the analysis finds that the data in the access request data is damaged data, then proceed to step S5. If the data in the access request data is found to be correct data after analysis, then go to step S6; S4: delete the wrong data; S5: Fill missing values in the damaged data and send the filled access request data to the distributed data processing center for re-demand analysis; S6: The distributed data processing center authenticates the access request data according to the access request data of the client and based on the public key set in its own encryption server. If the authentication is successful, the process proceeds to step S7. If the authentication is not successful, the access request data is determined to be erroneous data and the process proceeds to step S4. S7: The distributed data processing center sends the access request data to the encryption server, and the encryption server verifies whether the client has the authority to access the requested database content. If the verification is successful, the process proceeds to step S8. If the verification is not successful, the access request data is determined to be erroneous data, and the process proceeds to step S4. S8: The distributed data processing center extracts the encrypted file from the encryption server, decrypts the encrypted file according to the encryption key, and transmits the decrypted data to the client through the secure layer protocol encrypted channel.
2. The data intelligent analysis method based on distributed processing according to claim 1 is characterized in that: After the access request data is determined to be damaged data in step S3, the following steps are performed: S31: The distributed data processing center stores the lossy data in the hot data storage node and sets the storage duration s 1; S32: If the distributed processing center detects s When the time duration is not fully counted, the damaged data in the hot data storage node receives a fill request, and then enters step S5. If the distributed processing center monitors s After the time period 1 is reached, if the damaged data in the hot data storage node does not receive a filling request, the process proceeds to step S33; S33: The distributed processing center stores the unread damaged data in the cold data storage node and sets the storage duration. s 2. The value selection follows the following principles: ; S34: If the distributed processing center detects s 2. If the time duration is not fully calculated, the damaged data in the cold data storage node receives a fill request, then the process goes to step S5. If the distributed processing center detects s 2 After the time is up, the number of lossy nodes in the cold storage node If the data has not been read, the process proceeds to step S4.
3. The data intelligent analysis method based on distributed processing according to claim 2 is characterized in that: The configuration of the cold data storage node is higher than the configuration of the hot data storage node.
4. The data intelligent analysis method based on distributed processing according to claim 2 is characterized in that: In step S5, when filling missing values for damaged data, one or more missing value filling methods including mean interpolation, maximum likelihood estimation, and multiple interpolation can be used.
5. The data intelligent analysis method based on distributed processing according to claim 1 is characterized in that: In step S6, when the distributed data processing center authenticates the access request data based on the public key set in its own encryption server, the following steps are also included: S61: According to the length of the padding data value of the correct data in step S3, the distributed data processing center sets a first rule to establish a first data column of the access request data, the first rule sorts the correct data from small to large according to the padding value length, and prioritizes the data in the first data column according to the first rule to obtain priority processing information in the first data column; S62: According to the time point when the access request data is judged as correct data in step S3, the distributed data processing center sets a second rule, the second rule sorts the access request data from the earliest to the latest according to the time point when the access request data is judged as correct data, and sorts the first priority processing information according to the second rule to obtain the first processing data; S63: The distributed data processing center processes the first data slices through the first distributed data processing node to generate multiple sub-data; S64: extract identification information of the plurality of sub-data, allocate the second distributed data processing nodes for processing the plurality of sub-data, and obtain an authentication sequence for accessing the request data.
6. The data intelligent analysis method based on distributed processing according to claim 5 is characterized in that: The first distributed data processing node sends a first heartbeat packet to the distributed data processing center.
7. The data intelligent analysis method based on distributed processing according to claim 5 is characterized in that: The second distributed data processing node sends a second heartbeat packet to the distributed data processing center.
8. The data intelligent analysis method based on distributed processing according to claim 6 is characterized in that: The encryption granularity of data in the encryption server is the field data of each record.
