Data security collection and dispatching system based on big data

Abstract

The present invention is applicable to the field of data collection and scheduling technology, and in particular to a data security collection and scheduling system based on big data, including a microservice device, a hash device, a hash chain construction device and a backtracking device, wherein the microservice device is used to create a microservice framework and integrate microservice items, wherein the microservice items include at least a first microservice item and a second microservice item, wherein the first microservice item is used to transmit the collected original data to a constructed data lake, insert an identifier into the original data, and mark the position of the identifier; the hash device is used to select a reversible hash function and hash the identifier after the second microservice item extracts data from the data lake. By monitoring the original data, the present invention can timely trace back to the original data when the original data drifts, thereby avoiding changes in the original data during the collection and scheduling process and improving the accuracy of big data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition and scheduling, and in particular to a data security acquisition and scheduling system based on big data. Background Art

[0002] Big data refers to the collection, storage, processing and analysis of large-scale data, and the extraction of information and value therein. Big data involves multiple links, including data collection, storage, cleaning, processing and analysis. The main purpose of big data processing is to extract valuable information and knowledge from massive data to support decision-making and predict future trends. In order to achieve this goal, big data processing technology needs to adopt a series of methods and tools, including data mining, machine learning and data warehouses. In the process of using these methods and tools, it is inevitable to collect and schedule data multiple times. It can be seen that data collection and scheduling is one of the important links in big data processing and analysis. By effectively scheduling and managing tasks such as big data collection, processing, storage and transmission, the security and availability of data can be ensured, providing reliable guarantees for subsequent big data processing and analysis.

[0003] However, in the process of data collection and scheduling, due to the need to frequently process data, it is easy to cause data changes, deviate from the original data, and cause data drift, which will also have a great impact on the big data processing results; therefore, how to avoid data drift by tracing back the original data is a technical problem that the present invention needs to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a data security collection and scheduling system based on big data to solve the problem of "how to avoid data drift by tracing back the original data" raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] Data security collection and scheduling system based on big data, including microservice equipment, hash equipment, hash chain construction equipment and backtracking equipment;

[0007] The microservice device is used to create a microservice framework and integrate microservice items, wherein the microservice items include at least a first microservice item and a second microservice item, and the first microservice item is used to transmit the collected raw data to the constructed data lake, insert an identifier into the raw data, and mark the position of the identifier;

[0008] The hash device is used to select a reversible hash function to hash the identifier after the second microservice item extracts data from the data lake, and to overwrite the identifier in the original data with the obtained first hash value;

[0009] The hash chain construction device is used to use the identifier as the first data node and the first hash value as the second data node to construct a hash chain, create a trigger for monitoring changes in the original data, and when the trigger monitors that the original data is scheduled, continue to hash, and iterate the hash chain according to the hash result;

[0010] The backtracking device is used to use the position, hash chain and reversible hash function to trace back to the original data in the data lake when the original data needs to be retrieved.

[0011] Furthermore, the microservice device includes:

[0012] The communication module is used to define the microservice interface, build a transmission link based on the asynchronous communication mechanism, and determine the corresponding relationship between the transmission link and the microservice item;

[0013] The transmission module is used to transmit the original data to the data lake by using the first microservice item and its corresponding transmission link.

[0014] Furthermore, the microservice device also includes:

[0015] A summary module, used for generating a summary of the original data and converting the summary into a vector, obtaining at least a vector A and a vector B;

[0016] A similarity calculation module is used to calculate the similarity between vector A and vector B according to the cosine similarity formula;

[0017] The cosine similarity formula is:

[0018]

[0019] Where A·B is the dot product of two vectors (the sum of corresponding elements after multiplication), ||A||·||B|| is the dot product of the moduli of two vectors;

[0020] The classification module is used to classify the original data corresponding to vector A and vector B into the same management directory when the similarity is greater than a preset threshold, wherein the management directory has been pre-established.

[0021] Furthermore, the microservice device also includes:

[0022] A generation module, used to obtain identification principles, integrate the management directory and identification principles, and generate an identifier;

[0023] The inserting module is used to insert the identifier into the original data based on the identification principle and mark the position of the identifier.

[0024] Furthermore, the hash device includes:

[0025] A salting module, used to locate the identifier in the extracted data and add "salt" to the end of the identifier;

[0026] The storage module is used to hash the identifier and the "salt" and save the generated "salt" value and the first hash value.

[0027] Furthermore, the hash chain construction device includes:

[0028] A construction module capable of constructing a hash chain based on the identifier and the first hash value;

[0029] The description module is used to insert descriptive information into all data nodes of the hash chain, and the descriptive information at least includes a data stamp and a source identifier.

[0030] Furthermore, the hash chain construction device also includes:

[0031] An introduction module is used to introduce a change detection mechanism based on differential comparison into triggers;

[0032] The judgment module is used to judge whether there is a difference between the original data before and after scheduling. If yes, the hash value and "salt" in the original data after scheduling are hashed again. If no, the trigger is not started.

[0033] Furthermore, the transmission module includes:

[0034] A retrieving unit, used to integrate the transmission link and the second microservice item, traverse the management directory, and retrieve original data from the data lake;

[0035] The sending unit is used to send the original data to a target terminal.

[0036] Furthermore, the description module includes:

[0037] A copy unit, used for generating a copy of the descriptive information;

[0038] The superposition unit is used to build an additional set consisting of copies and superimpose the additional set into the management directory.

[0039] Furthermore, the judging module includes:

[0040] A difference calculation unit, used to calculate the mean square of data at corresponding positions in the original data and the scheduled original data using a mean square formula;

[0041] The mean square formula is:

[0042]

[0043] Where M is the mean square, A iis the sample data at location i, and n is the number of samples;

[0044] A substitution unit, used to substitute the calculated mean square into the difference percentage formula, calculate the difference percentage before and after the original data is scheduled, and use the difference percentage as a benchmark to determine whether there is a difference between the original data before and after the scheduling;

[0045] The difference percentage formula is:

[0046]

[0047] Where P is the difference percentage, M1 is the mean square of the original data, and M2 is the mean square of the original data after scheduling.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. By setting up microservice devices, a data lake can be created, thereby realizing the storage of data in different formats and greatly improving the efficiency of data calls. By using hash chain construction devices to build hash chains, the backtracking path of the original data is determined. By monitoring the original data, it is possible to discover and trace back to the original data in time when the original data drifts, avoiding changes in the original data during the collection and scheduling process and improving the accuracy of big data processing.

[0050] 2. By creating a similarity calculation module, the original data can be classified into different management directories according to the similarity of the original data, thereby realizing the classification of the original data. By using the difference calculation unit, the changes in the original data can be quantified, data drift can be discovered in time, and the consistency of the original data in the scheduling process can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A block diagram of the composition of a data security acquisition and scheduling system based on big data provided by an embodiment of the present invention;

[0052] Figure 2 A flowchart of the scheduling process of raw data in a data security acquisition and scheduling system based on big data provided by an embodiment of the present invention;

[0053] Figure 3 A flowchart of the backtracking process of original data in a data security acquisition and scheduling system based on big data provided by an embodiment of the present invention;

[0054] Figure 4 A block diagram of the composition of microservice devices in a data security collection and scheduling system based on big data provided by an embodiment of the present invention;

[0055] Figure 5A block diagram of the composition of a hash device in a data security collection and scheduling system based on big data provided by an embodiment of the present invention;

[0056] Figure 6 A block diagram of the composition of a hash chain construction device in a data security collection and scheduling system based on big data provided by an embodiment of the present invention;

[0057] Figure 7 A block diagram of the composition of a transmission module in a data security acquisition and scheduling system based on big data provided by an embodiment of the present invention;

[0058] Figure 8 A block diagram of the composition of the description module in the data security acquisition and scheduling system based on big data provided by an embodiment of the present invention;

[0059] Fig. 9 A block diagram of the composition of the judgment module in the data security acquisition and scheduling system based on big data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first data security acquisition and scheduling system script based on big data may be referred to as a second data security acquisition and scheduling system script based on big data, and similarly, a second data security acquisition and scheduling system script based on big data may be referred to as a first data security acquisition and scheduling system script based on big data.

[0062] In Example 1, Figure 1 The data security collection and scheduling system 1 based on big data provided by an embodiment of the present invention is shown, including a microservice device 11, a hash device 12, a hash chain construction device 13 and a backtracking device 14, which are described in detail below:

[0063] The microservice device 11 is used to create a microservice framework and integrate microservice items, wherein the microservice items include at least a first microservice item and a second microservice item, and the first microservice item is used to transmit the collected raw data to the constructed data lake, insert an identifier into the raw data, and mark the position of the identifier;

[0064] Create a microservice framework and integrate microservice items into the microservice framework. The advantage of doing this is that by dividing the collection and scheduling of raw data into multiple separate modules, these modules are microservice items. Each microservice item has clear functions and responsibilities, which helps to improve the efficiency of data collection and scheduling. At the same time, each microservice item is relatively independent, more flexible, and fault isolation is also achieved.

[0065] In this embodiment, taking the first microservice item and the second microservice item as an example, the first microservice item is mainly used for data collection, and the second microservice item is mainly used for data scheduling and transmission. After the first microservice item collects the original data, the original data is transferred to the data lake, wherein the data lake here is a centralized repository that can accommodate data in any format from various data sources; after the original data is collected, an identifier is inserted into the original data, wherein the identifier can be a numeric character segment, and the position of the identifier is marked in the original data, and the marking method here can be recording the position of the identifier, or other methods.

[0066] The hash device 12 is used to select a reversible hash function to hash the identifier after the second microservice item extracts data from the data lake, and to overwrite the identifier in the original data with the obtained first hash value.

[0067] The second microserver extracts the original data from the data lake and selects a reversible hash function to hash the identifier, where the reversible hash function is a special hash function that can map the original data to an output of a fixed length and restore the original data through the output, and uses the first hash value obtained after hashing to overwrite the identifier in the original data.

[0068] The hash chain construction device 13 is used to use the identifier as the first data node and the first hash value as the second data node to construct a hash chain, create a trigger for monitoring changes in the original data, and when the trigger monitors that the original data is scheduled, continue to hash, and iterate the hash chain according to the hash result;

[0069] The identifier is used as the first data node and the first hash value is used as the second data node to build a hash chain. It should be noted here that each time the original data is scheduled, the identifier, the first hash value, the second hash value, etc. are hashed once, and the third data node, the fourth data node, etc. are obtained, all the nodes are integrated, a hash chain is created, and a trigger is created to monitor changes in the original data. When the original data is transmitted, format converted, or other scheduling behaviors occur, the trigger will be activated and the original data will be hashed.

[0070] For example, when a certain original data is undergoing scheduling behaviors such as transmission, data cleaning, and format conversion, when the trigger monitors that the original data transmission is completed, the data after the transmission is completed is recorded as A, and the identifier in A is hashed to obtain the first hash value, and the identifier is overwritten with the first hash value. The data obtained after the A data is cleaned is recorded as B. After monitoring that the A cleaning is completed, the first hash value in B is hashed to obtain the second hash value, and the first hash value is overwritten with the second hash value, and so on, continue to perform format conversion or other scheduling. That is to say, each time the original data is scheduled, it is hashed once, and a hash chain is constructed using the obtained identifier, the first hash value, the second hash value, etc. If a third hash value or more hash values ​​are generated, these hash values ​​are inserted into the hash chain and the hash chain is iterated.

[0071] The tracing back device 14 is used to trace back to the original data in the data lake by using the position, hash chain and reversible hash function when the original data needs to be retrieved.

[0072] After completing multiple dispatches of the original data, the hash chain and reversible hash function are used to trace back to the original data according to the hash values ​​in the data after multiple dispatches, and the original data is used to correct the data after multiple dispatches to eliminate data drift.

[0073] like Figure 2 As shown, the microservice device 11 is used to collect original data in the data lake, and the hash device 12 is used to hash the identifier in the original data, and then the hash chain construction device 13 is used to monitor the original data, and after monitoring that the original data is scheduled, the hash value in the scheduled original data is hashed again.

[0074] like Figure 3 As shown, when data backtracking is required, the hash chain construction device 13 is used to backtrack against the hash chain, and the hash device 12 and the reversible hash function are used to locate the original data in the data lake, and the original data is used to correct the scheduled data in the backtracking device 14.

[0075] In Example 2, Figure 4 The structure diagram of the microservice device in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the microservice device 11 includes:

[0076] The communication module 111 is used to define the microservice interface, build a transmission link based on an asynchronous communication mechanism, and determine the corresponding relationship between the transmission link and the microservice item;

[0077] Define and clarify the interface contract, including input parameters, output results and exception handling, etc. Use the gateway to manage the access entrance of all microservices, unify the message format and interface between all microservices, build a communication mechanism, and select the appropriate message queue middleware to ensure that data can be reliably transmitted and asynchronously processed between microservers; build a transmission link between microservers for data transmission. Each microserver has its corresponding transmission link, that is, each microserver has a corresponding relationship with the transmission link.

[0078] A transmission module 112 is used to transmit the original data to the data lake by using the first microservice item and its corresponding transmission link;

[0079] The first microservice item and its corresponding transmission link are used to transmit the collected raw data to the data lake. A multi-cloud data lake can be built here to realize the storage of raw data of different formats and types.

[0080] Figure 4 The structure diagram of the microservice device in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the microservice device 11 also includes:

[0081] A summary module 113 is used to generate a summary of the original data and convert the summary into a vector, at least obtaining a vector A and a vector B;

[0082] A similarity calculation module 114 is used to calculate the similarity between vector A and vector B according to the cosine similarity formula;

[0083] After the original data is collected, a summary of the original data is generated, and the summary is converted into a vector using a word embedding model or a word set model. For example, at this time, there are two original data A and B stored in the data lake. By generating summaries of A and B and inputting the summaries into the word embedding model, the vectors of A and B are obtained, and vector A and vector B are remembered.

[0084] The cosine similarity formula is:

[0085]

[0086] Where A·B is the dot product of two vectors (the sum of corresponding elements after multiplication), ||A||·||||B|| is the dot product of the moduli of two vectors;

[0087] The similarity between vector A and vector B is calculated. If the calculated value is larger, the similarity between vector A and vector B is higher.

[0088] The classification module 115 is used to classify the original data corresponding to the vector A and the vector B into the same management directory when the similarity is greater than a preset threshold, wherein the management directory has been pre-established.

[0089] If the similarity between vector A and vector B is greater than the preset threshold, the original data A and B are classified into the same management directory. Through further expansion, if the similarity between the collected data C and D in a management directory in the data lake is greater than the preset threshold, the data C can be classified into the management directory of D. In other words, by calculating the similarity of the original data, the original data can be classified. The preset threshold should be set in advance. The higher the threshold, the more similar the original data under the same management directory are. The management directory should be formulated in advance according to the specific requirements of the original data collection.

[0090] In Example 3, Figure 4 The structure diagram of the microservice device in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the microservice device 11 also includes:

[0091] A generating module 116, configured to obtain an identification principle, integrate the management directory and the identification principle, and generate an identifier;

[0092] The inserting module 117 is used to insert the identifier into the original data based on the identification principle and mark the position of the identifier.

[0093] Obtaining an identification principle, and generating an identifier according to the identification principle and the management directory, wherein the identification principle may be inserting a fixed-length character segment generated based on the management directory into the header or tail of the original data, and using the character segment as the identifier;

[0094] According to the identification principle, the identifier is inserted into the specified position in the original data, where the specified position may be the header or tail mentioned above, and the position of the identifier is marked.

[0095] In Example 4, Figure 5 The present invention shows a block diagram of the composition structure of a hash device in a data security collection and scheduling system based on big data provided by an embodiment of the present invention. The hash device 12 includes:

[0096] The salt adding module 121 is used to locate the identifier in the extracted data and add "salt" to the end of the identifier;

[0097] The storage module 122 is used to hash the identifier and the "salt" and save the generated "salt" value and the first hash value.

[0098] After the second microservice item extracts the original data from the data lake, it locates the identifier and adds a "salt" to the end of the identifier. The "salt" is random data. The "salt" is added to the identifier for hashing. The purpose of this is to avoid illegal data backtracking. Only when the hash value of the "salt" is obtained can the reversible hash function be used to backtrack the original data. At the same time, if the "salt" value is changed, the hash value will also be changed. By checking the consistency of the hash value and the "salt" value, it can be determined whether the original data has changed.

[0099] In Example 5, Figure 6 The structure diagram of the hash chain construction device in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown. The hash chain construction device 13 includes:

[0100] A construction module 131, capable of constructing a hash chain based on the identifier and the first hash value;

[0101] The description module 132 is used to insert descriptive information into all data nodes of the hash chain, and the descriptive information at least includes a data stamp and a source identifier.

[0102] Data nodes are created using an identifier, a first hash value, a second hash value, or even more hash values, where the original data and the identifier are the first data node, the original data after the first scheduling and the first hash value are the second data node, and so on. By building a hash chain, by recording each scheduling of the original data, and using a reversible hash function and a hash chain, the original data can be traced back. For example, after N schedulings of certain original data, a hash chain consisting of a first data node, a second data node, and up to the Nth data node is built, and the original data in the first data node is used to correct the data in the Nth data node to prevent the original data from changing and eliminate data drift.

[0103] The descriptive information includes data stamp, source identifier and other attribute data. The data stamp is the timestamp when the original data is generated, which is used to indicate the time when the original data was generated. The source identifier is used to identify the source of the original data and improve the traceability efficiency of the original data.

[0104] In Example 6, Figure 6 The structure diagram of the hash chain construction device in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the hash chain construction device 13 also includes:

[0105] An introduction module 133 is used to introduce a change detection mechanism based on differential comparison into the trigger;

[0106] The judgment module 134 is used to judge whether there is a difference between the original data before and after scheduling. If yes, the hash value and "salt" in the original data after scheduling are hashed again. If no, the trigger is not started.

[0107] A change detection mechanism is introduced into the trigger. By utilizing the change detection mechanism, it can be determined whether the original data has changed. The core of the change detection mechanism is differential comparison. It should be noted that the role of the trigger here is to monitor whether the data is scheduled for transmission, conversion, etc., not to monitor whether the data has changed. When the original data is monitored to be scheduled, the hash value and "salt" in the original data after scheduling are hashed again. If the original data is not scheduled, the trigger will not be started.

[0108] In Example 7, Figure 7 The structure block diagram of the transmission module in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the transmission module 112 includes:

[0109] A retrieving unit 1121 is used to integrate the transmission link and the second microservice item, traverse the management directory, and retrieve the original data from the data lake;

[0110] The sending unit 1122 is used to send the original data to the target terminal.

[0111] When the original data needs to be called, the second microservice item and its corresponding transmission link are used to retrieve the original data from the data lake through the management directory, and the transmission link is used to send the original data to the target terminal, where the target terminal can be a big data processing center or other data processing terminals.

[0112] In Example 8, Figure 8 The structure diagram of the description module in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the description module 132 includes:

[0113] A copy unit 1321, used to generate a copy of the descriptive information;

[0114] The superimposing unit 1322 is used to build an additional set consisting of copies and superimpose the additional set into the management directory.

[0115] Create a copy of the descriptive information in all the original data and form an additional set of the copies. By analyzing the additional set, you can better classify the original data and overlay the additional set into the management directory. By searching the additional set, you can quickly locate the required original data.

[0116] In Example 9, Fig. 9The structure diagram of the judgment module in the data security collection and scheduling system based on big data provided by an embodiment of the present invention is shown, and the judgment module 134 includes:

[0117] The difference calculation unit 1341 is used to calculate the mean square of the data at the corresponding position in the original data and the scheduled original data by using the mean square formula;

[0118] The mean square formula is:

[0119]

[0120] Where M is the mean square, A i is the sample data at location i, and n is the number of samples;

[0121] Calculate the mean square of the data at the corresponding positions in the original data and the scheduled data, where the corresponding positions should be set in advance. By calculating the mean square, the difference between the two data can be quantified. If the two data are very close, their mean square value will be small. If they are very different, the mean square value will be relatively large.

[0122] A substitution unit 1342 is used to substitute the calculated mean square into the difference percentage formula to calculate the difference percentage before and after the original data changes, and use the difference percentage to determine whether there is a difference between the original data before and after the change;

[0123] The difference percentage formula is:

[0124]

[0125] Where P is the difference percentage, M1 is the mean square of the original data, and M2 is the mean square of the original data after scheduling.

[0126] Substitute the calculated mean square into the difference percentage formula, and obtain the calculation result after calculation. This can more intuitively show the difference between the original data and the original data after scheduling. If the difference percentage is greater than the preset threshold, it means that the data has changed. When processing big data, it is necessary to trace back the original data and correct the original data after scheduling to ensure the accuracy of the original data during the big data processing. If the difference percentage is less than or equal to the preset threshold or is extremely close to zero, it proves that the original data has not changed significantly during the scheduling process, and big data processing can be performed directly without correction.

[0127] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data security collection and scheduling system based on big data, characterized in that: Including microservice equipment, hash equipment, hash chain construction equipment and backtracking equipment, The microservice device is used to create a microservice framework and integrate microservice items, wherein the microservice items include at least a first microservice item and a second microservice item, and the first microservice item is used to transmit the collected raw data to the constructed data lake, insert an identifier into the raw data, and mark the location of the identifier; The hash device is used to select a reversible hash function to hash the identifier after the second microservice item extracts data from the data lake, and to overwrite the identifier in the original data with the obtained first hash value; The hash device comprises: A salting module is used to locate the identifier in the extracted data and add "salt" to the end of the identifier; A storage module, used to hash the identifier and the "salt", and save the generated "salt" value and the first hash value; The hash chain construction device is used to use the identifier as the first data node and the first hash value as the second data node to construct a hash chain, create a trigger for monitoring changes in the original data, and when the trigger monitors that the original data is scheduled, continue to hash, and iterate the hash chain according to the hash result; The hash chain construction device includes: A construction module capable of constructing a hash chain based on the identifier and the first hash value; A description module, used to insert descriptive information into all data nodes of the hash chain, wherein the descriptive information at least includes a data stamp and a source identifier; The hash chain construction device also includes: An introduction module is used to introduce a change detection mechanism based on differential comparison into triggers; The judgment module is used to judge whether there is a difference between the original data before and after scheduling. If yes, the hash value and "salt" in the original data after scheduling are hashed again. If no, the trigger is not started; The judging module comprises: A difference calculation unit, used to calculate the mean square of data at corresponding positions in the original data and the scheduled original data using a mean square formula; The mean square formula is: Where M is the mean square, A i is the sample data at location i, and n is the number of samples; A substitution unit, used to substitute the calculated mean square into the difference percentage formula, calculate the difference percentage before and after the original data is scheduled, and use the difference percentage as a benchmark to determine whether there is a difference between the original data before and after the scheduling; The difference percentage formula is: Where P is the difference percentage, M1 is the mean square of the original data, and M2 is the mean square of the original data after scheduling; The backtracking device is used to use the position, hash chain and reversible hash function to trace back to the original data in the data lake when the original data needs to be retrieved.

2. The system according to claim 1, characterized in that The microservice device includes: The communication module is used to define the microservice interface, build a transmission link based on the asynchronous communication mechanism, and determine the corresponding relationship between the transmission link and the microservice item; The transmission module is used to transmit the original data to the data lake by using the first microservice item and its corresponding transmission link.

3. The system according to claim 2, characterized in that The microservice device also includes: A summary module, used for generating a summary of the original data and converting the summary into a vector, obtaining at least a vector A and a vector B; A similarity calculation module is used to calculate the similarity between vector A and vector B according to the cosine similarity formula; The cosine similarity formula is: Where S is the similarity, A·B is the dot product of two vectors, and A·B is the dot product of the moduli of two vectors; The classification module is used to classify the original data corresponding to vector A and vector B into the same management directory when the similarity is greater than a preset threshold, wherein the management directory has been pre-established.

4. The system according to claim 3, characterized in that The microservice device also includes: A generation module, used to obtain identification principles, integrate the management directory and identification principles, and generate an identifier; The inserting module is used to insert the identifier into the original data based on the identification principle and mark the position of the identifier.

5. The system according to claim 4, characterized in that The transmission module comprises: A retrieving unit, used to integrate the transmission link and the second microservice item, traverse the management directory, and retrieve original data from the data lake; The sending unit is used to send the original data to a target terminal.

6. The system according to claim 5, characterized in that The description module includes: A copy unit, used for generating a copy of the descriptive information; The superposition unit is used to build an additional set consisting of the copies and superimpose the additional set into the management directory.

Images