A data storage management system and method based on big data platform

By using the intermediate migration library to standardize data processing and automated conflict mediation during the data integration process, data conflicts and inconsistencies are solved, the accuracy and efficiency of data integration are improved, and the number of errors is reduced.

CN118760671BActive Publication Date: 2024-12-06JIANGSU LUOLING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411238454.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-12-06
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

During the data integration process, multiple users or systems operate on the same data, resulting in data conflicts and inconsistencies, resulting in data updates not being synchronized in time, resulting in multiple errors reported, seriously affecting the data process.

Method used

By building connections between data platforms, using intermediate migration libraries to standardize data processing and conflict mediation, realize automated mediation and priority conflict resolution, and reduce the number of errors reported.

Benefits of technology

Effectively build a full-process service mechanism for data reflow, realize accurate docking of data needs, reduce error-reported data, and improve data integration effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118760671B_ABST
    Figure CN118760671B_ABST
Patent Text Reader

Abstract

The present invention discloses a data storage management system and method based on a big data platform, which belongs to the field of data storage management technology. The system includes: a historical data migration module, a data platform construction module, a data reflux analysis module and a data storage management module; the output end of the historical data migration module is connected to the input end of the data platform construction module; the output end of the data platform construction module and the data reflux analysis module are connected to the input end of the data storage management module; the output end of the data storage management module is connected to the input end of the historical data migration module, the data platform construction module and the data reflux analysis module. In terms of data integration, the present invention proposes a set of data standard specification systems with unified standards and consistent calibers, which can meet various data errors in the data storage management process, reduce error data, and improve data integration effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data storage management, and in particular to a data storage management system and method based on a big data platform. Background Art

[0002] With the development of the data industry, in the process of electronic business data, the consolidated management of data assets has gradually become an important part of the data management system. It mainly includes: migrating and uniformly storing data of various business systems developed and applied, building a set of data standard specification system with unified standards and consistent caliber, realizing the management of data quality, metadata, and data resource directory and data asset visualization services, and formulating a data return sharing mechanism to quickly respond to project data needs, ensuring efficient data return and effective sharing and utilization, and actively building a full-process service work mechanism for data return, realizing accurate docking of data needs, and ensuring orderly data return.

[0003] In the process of integrating data assets, various data problems are also very likely to occur. For example, during the data synchronization process, multiple users or multiple systems operate on the same data at the same time, resulting in data conflicts, or different processing processes of different users, resulting in data inconsistency, which leads to data updates not being synchronized in time, resulting in data conflicts, and ultimately leading to multiple errors in the data integration and storage process, seriously affecting the data process. Summary of the invention

[0004] The purpose of the present invention is to provide a data storage management system and method based on a big data platform to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a data storage management method based on a big data platform, the method comprising the following steps:

[0006] S1. Build connections between data platforms, call the original data to the intermediate migration library, and calculate the resource capacity of the target migration database to determine whether to start data migration.

[0007] S2, the intermediate migration library is connected to the data platform construction module to standardize the data, including unified definition, format specification and value range constraints, and also feed back real-time data during the standardization process;

[0008] S3: The original data is transferred to the target migration database via the intermediate migration library. The target migration database is connected to an administrator access port. When receiving an access request from the administrator access port, it is determined whether a data conflict occurs.

[0009] S4. Construct a data storage management module, and realize automatic mediation of data conflicts based on the data storage management module, form automatic instructions, control each module to run according to the instructions, and reduce the number of errors.

[0010] According to the above technical solution, in step S1, it also includes:

[0011] Get the original data capacity value and calculate the data growth value per unit time;

[0012] Calculate the estimated migration time based on the average migration speed of the original data capacity value ;

[0013] Calculate the estimated target migration database resource capacity:

[0014]

[0015] in, Refers to the estimated target migration database resource capacity; Refers to the original data capacity value; Refers to the data growth value per unit time;

[0016] If the measured target migration database resource capacity is lower than the estimated target migration database resource capacity, an alarm signal is generated to the administrator access port and data migration is not performed.

[0017] According to the above technical solution, in step S2, it also includes:

[0018] The unified definition refers to providing a unified data standard definition and platform logic model for the data platform, and realizing a unified definition of the data in the intermediate migration library;

[0019] The format specification refers to the use of the same business standard specification, technical standard specification and management standard specification to standardize the data; the business standard specification refers to the association between data and business, and the verification of the scope and content of data; the technical standard specification refers to the standardization of the encoding rules and naming rules of data after the business standard specification work is completed, including data sources, data items, logical model standards, physical model standards, metadata, and public code value standards; the management standard specification refers to the clear application, approval, change, and sharing process of each standard data;

[0020] The range constraint refers to the constraint that the attribute value of the data satisfies the defined enumeration value.

[0021] According to the above technical solution, in step S3, it also includes:

[0022] The administrator accesses the port to send a data call access request, instructing to call the data in the target migration database. If the called data is standardized in the intermediate migration database or the called data changes in real time, it is defined as a data conflict.

[0023] According to the above technical solution, in step S4, it also includes:

[0024] Build a data conflict resolution mechanism, with priority conflict resolution as the only resolution mechanism;

[0025] Priority conflict resolution refers to setting priorities for different data during data synchronization, and synchronizing data with higher priorities first. Compared with timestamp conflict resolution and manual intervention conflict resolution, it can reduce the manpower and time costs.

[0026] If in real-time state, the administrator accesses the port to send a data call access request, and the called data is standardized in the intermediate migration library, then the data call access is higher than the standardized processing;

[0027] If the data changes in real time and the same data is standardized in the intermediate migration database, the data change is higher than the standardization;

[0028] If the administrator accesses the port to send a data call access request in real-time state, the called data is standardized in the intermediate migration library, and at the same time the data change state occurs, the data standardization process has the lowest priority, and the others are processed concurrently;

[0029] The concurrent processing refers to the real-time state where the administrator accesses the port to send a data call access request, and the called data simultaneously changes to a data state;

[0030] Build a data storage management module to automate concurrent processing instructions:

[0031] Perform change data analysis based on historical big data. In the historical big data, obtain the interval time and change number of change data to form data combinations with different interval time and change number.

[0032] The number of changes and the interval between each change of data changes under concurrent processing are obtained, recorded as data combination A, and data combination A is identified and included in the data combination in the historical big data to form a replacement combination of data combination A:

[0033] Set the data combination in the historical big data to H category, and define the misjudgment probability of data combination A as , It refers to the probability of misclassifying data combination A into a certain category j in the data combination in historical big data. Data combination A and j must not belong to the same category. j represents other data combinations in the H-category data combination that are not of the same category as data combination A.

[0034] Then we have:

[0035]

[0036] in, Represents the probability of misjudgment loss of data combination A under the data combination in historical big data; represents fixed misjudgment loss;

[0037] Then the misjudgment loss of data combination A is:

[0038]

[0039] in, is the misjudgment loss of data combination A; Represents the prior probability of category j under the data combination in historical big data;

[0040] The misjudgment loss is solved using the Mahalanobis distance, and the minimum value among all solutions is taken as the misjudgment loss of data combination A. Its corresponding category j is recorded as the priority category. At this time, the probability value of priority category j under the data combination of data combination A misjudged into the historical big data is the smallest, so the misjudgment loss is the smallest, which means that priority category j and data combination A are most similar. The mode of the number of changes in priority category j is taken as the output value;

[0041] The difference solution is performed based on the output value and the number of data changes under concurrent processing. If the difference solution is less than the system preset quantity threshold, the data call access priority is higher than the data change status. Conversely, the data change status priority is higher than the data call access.

[0042] A data storage management system based on a big data platform, the system comprises: a historical data migration module, a data platform construction module, a data reflux analysis module and a data storage management module;

[0043] The historical data migration module is used to back up the original data, import it into the intermediate migration library, and measure the resource capacity of the target migration database to determine whether to start data migration; the data platform construction module is used to standardize the data, including unified definition, format specification and value range constraints, to ensure the consistency, accuracy and interoperability of the data, and to feedback the real-time data in the standardization process; the data reflux analysis module is used to call the original data in the target migration database, and to feedback the call information to the data storage management module; the data storage management module is used to receive the data processing information data transmitted by each module, mediate data conflicts, form automatic instructions, control each module to run according to the instructions, and reduce the number of errors;

[0044] The output end of the historical data migration module is connected to the input end of the data platform construction module; the output end of the data platform construction module and the data backflow analysis module are connected to the input end of the data storage management module; the output end of the data storage management module is connected to the input end of the historical data migration module, the data platform construction module and the data backflow analysis module.

[0045] According to the above technical solution, the historical data migration module includes a data backup unit and a judgment unit;

[0046] The data backup unit is used to back up the original data and import it into the intermediate migration library; the judgment unit is used to measure the resource capacity of the target migration database and judge whether to start data migration;

[0047] The output end of the data backup unit is connected to the input end of the judgment unit.

[0048] According to the above technical solution, the data platform construction module includes a unified definition unit, a format specification unit and a value range constraint unit;

[0049] The unified definition unit is used to provide a unified data standard definition and platform logic model for the data platform, and realize a unified definition of the data in the intermediate migration library;

[0050] The format specification unit is used to standardize the data using the same business standard specification, technical standard specification and management standard specification; the business standard specification refers to the association between data and business, and the verification of the scope and content of data; the technical standard specification refers to the standardization of the encoding rules and naming rules of data after the business standard specification work is completed, including data sources, data items, logical model standards, physical model standards, metadata, and public code value standards; the management standard specification refers to the process of clarifying the application, approval, change, and sharing of each standard data;

[0051] The range constraint unit is used to ensure that the attribute value of the data satisfies the constraints of the defined enumeration value.

[0052] According to the above technical solution, the data backflow analysis module includes a node authentication unit and a task authorization unit;

[0053] The node authentication unit is used to perform node communication authentication before data exchange between nodes when the administrator accesses the port to send a data call access request; the task authorization unit is used to synchronize access requests accurately to the second level. Before the synchronization access request is executed, the sender and the receiver need to confirm, otherwise synchronization cannot be achieved;

[0054] The output end of the node authentication unit is connected to the input end of the task authorization unit.

[0055] According to the above technical solution, the data storage management module includes an automatic adjustment unit and an instruction unit;

[0056] The automatic mediation unit is used to receive data processing information data transmitted by each module, mediate data conflicts, and form automatic instructions; the instruction unit is used to control each module to run according to the instructions, complete the data storage task, and reduce the number of errors;

[0057] The output end of the automatic adjustment unit is connected to the input end of the instruction unit.

[0058] Compared with the prior art, the beneficial effects achieved by the present invention are: in the direction of data integration, the present invention proposes a set of data standard specification system with unified standards and consistent caliber, conducts intelligent analysis on changes and reflux in the process of data integration, effectively constructs a full-process service working mechanism for data reflux, realizes precise docking of data needs, can meet various types of data errors in the process of data storage and management, reduce error data, and improve data integration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 It is a flow chart of a data storage management system and method based on a big data platform of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1 ,In the first embodiment of the present invention: a data storage management method based on a big data platform is provided, the method comprising: building a connection between data platforms, calling original data to an intermediate migration library, and at the same time calculating the resource capacity of a target migration database, and determining whether to start data migration;

[0063] Get the original data capacity value and calculate the data growth value per unit time;

[0064] Calculate the estimated migration time based on the average migration speed of the original data capacity value ;

[0065] Calculate the estimated target migration database resource capacity:

[0066]

[0067] in, Refers to the estimated target migration database resource capacity; Refers to the original data capacity value; Refers to the data growth value per unit time;

[0068] If the measured target migration database resource capacity is lower than the estimated target migration database resource capacity, an alarm signal is generated to the administrator access port and data migration is not performed.

[0069] The intermediate migration library is connected to the data platform construction module to standardize the data, including unified definition, format specification and value range constraints, and also feed back real-time data during the standardization process;

[0070] The unified definition refers to providing a unified data standard definition and platform logic model for the data platform, and realizing a unified definition of the data in the intermediate migration library;

[0071] The format specification refers to the use of the same business standard specification, technical standard specification and management standard specification to standardize the data; the business standard specification refers to the association between data and business, and the verification of the scope and content of data; the technical standard specification refers to the standardization of the encoding rules and naming rules of data after the business standard specification work is completed, including data sources, data items, logical model standards, physical model standards, metadata, and public code value standards; the management standard specification refers to the clear application, approval, change, and sharing process of each standard data;

[0072] The range constraint refers to the constraint that the attribute value of the data satisfies the defined enumeration value.

[0073] The original data is transferred to the target migration database through the intermediate migration library. The target migration database is connected to the administrator access port. When receiving the access request from the administrator access port, it is determined whether a data conflict occurs.

[0074] The administrator accesses the port to send a data call access request, instructing to call the data in the target migration database. If the called data is standardized in the intermediate migration database or the called data changes in real time, it is defined as a data conflict.

[0075] Build a data conflict resolution mechanism, with priority conflict resolution as the only resolution mechanism;

[0076] If in real-time state, the administrator accesses the port to send a data call access request, and the called data is standardized in the intermediate migration library, then the data call access is higher than the standardized processing;

[0077] If the data changes in real time and the same data is standardized in the intermediate migration database, the data change is higher than the standardization;

[0078] If the administrator accesses the port to send a data call access request in real-time state, the called data is standardized in the intermediate migration library, and at the same time the data change state occurs, the data standardization process has the lowest priority, and the others are processed concurrently;

[0079] The concurrent processing refers to the real-time state where the administrator accesses the port to send a data call access request, and the called data simultaneously changes to a data state;

[0080] Build a data storage management module to automate concurrent processing instructions:

[0081] Perform change data analysis based on historical big data. In the historical big data, obtain the interval time and change number of change data to form data combinations with different interval time and change number.

[0082] The number of changes and the interval between each change of data changes under concurrent processing are obtained, recorded as data combination A, and data combination A is identified and included in the data combination in the historical big data to form a replacement combination of data combination A:

[0083] Set the data combination in the historical big data to H category, and define the misjudgment probability of data combination A as , It refers to the probability of misclassifying data combination A into a certain category j in the data combination in historical big data. Data combination A and j must not belong to the same category. j represents other data combinations in the H-category data combination that are not of the same category as data combination A.

[0084] Then we have:

[0085]

[0086] in, Represents the probability of misjudgment loss of data combination A under the data combination in historical big data; represents fixed misjudgment loss;

[0087] Then the misjudgment loss of data combination A is:

[0088]

[0089] in, is the misjudgment loss of data combination A; Represents the prior probability of category j under the data combination in historical big data;

[0090] The misjudgment loss is solved using the Mahalanobis distance, and the minimum value among all solutions is taken as the misjudgment loss of data combination A. Its corresponding category j is recorded as the priority category. At this time, the probability value of priority category j under the data combination of data combination A misjudged into the historical big data is the smallest, so the misjudgment loss is the smallest, which means that priority category j and data combination A are most similar. The mode of the number of changes in priority category j is taken as the output value;

[0091] The difference solution is performed based on the output value and the number of data changes under concurrent processing. If the difference solution is less than the system preset quantity threshold, the data call access priority is higher than the data change status. Conversely, the data change status priority is higher than the data call access.

[0092] In the second embodiment, a data storage management system based on a big data platform is provided, the system comprising: a historical data migration module, a data platform construction module, a data reflux analysis module and a data storage management module;

[0093] The historical data migration module is used to back up the original data, import it into the intermediate migration library, and measure the resource capacity of the target migration database to determine whether to start data migration; the data platform construction module is used to standardize the data, including unified definition, format specification and value range constraints, to ensure the consistency, accuracy and interoperability of the data, and to feedback the real-time data in the standardization process; the data reflux analysis module is used to call the original data in the target migration database, and to feedback the call information to the data storage management module; the data storage management module is used to receive the data processing information data transmitted by each module, mediate data conflicts, form automatic instructions, control each module to run according to the instructions, and reduce the number of errors;

[0094] The output end of the historical data migration module is connected to the input end of the data platform construction module; the output end of the data platform construction module and the data backflow analysis module are connected to the input end of the data storage management module; the output end of the data storage management module is connected to the input end of the historical data migration module, the data platform construction module and the data backflow analysis module.

[0095] The historical data migration module includes a data backup unit and a judgment unit;

[0096] The data backup unit is used to back up the original data and import it into the intermediate migration library; the judgment unit is used to measure the resource capacity of the target migration database and judge whether to start data migration;

[0097] The output end of the data backup unit is connected to the input end of the judgment unit.

[0098] The data platform construction module includes a unified definition unit, a format specification unit and a value range constraint unit;

[0099] The unified definition unit is used to provide a unified data standard definition and platform logic model for the data platform, and realize a unified definition of the data in the intermediate migration library;

[0100] The format specification unit is used to standardize the data using the same business standard specification, technical standard specification and management standard specification; the business standard specification refers to the association between data and business, and the verification of the scope and content of data; the technical standard specification refers to the standardization of the encoding rules and naming rules of data after the business standard specification work is completed, including data sources, data items, logical model standards, physical model standards, metadata, and public code value standards; the management standard specification refers to the process of clarifying the application, approval, change, and sharing of each standard data;

[0101] The range constraint unit is used to ensure that the attribute value of the data satisfies the constraints of the defined enumeration value.

[0102] The data backflow analysis module includes a node authentication unit and a task authorization unit;

[0103] The node authentication unit is used to perform node communication authentication before data exchange between nodes when the administrator accesses the port to send a data call access request; the task authorization unit is used to synchronize access requests accurately to the second level. Before the synchronization access request is executed, the sender and the receiver need to confirm, otherwise synchronization cannot be achieved;

[0104] The output end of the node authentication unit is connected to the input end of the task authorization unit.

[0105] The data storage management module includes an automatic adjustment unit and an instruction unit;

[0106] The automatic mediation unit is used to receive data processing information data transmitted by each module, mediate data conflicts, and form automatic instructions; the instruction unit is used to control each module to run according to the instructions, complete the data storage task, and reduce the number of errors;

[0107] The output end of the automatic adjustment unit is connected to the input end of the instruction unit.

[0108] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0109] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data storage management method based on a big data platform, characterized in that: The method comprises the following steps: S1. Build connections between data platforms, call the original data to the intermediate migration library, and calculate the resource capacity of the target migration database to determine whether to start data migration. S2, the intermediate migration library is connected to the data platform construction module to standardize the data, including unified definition, format specification and value range constraints, and also feed back real-time data during the standardization process; S3: The original data is transferred to the target migration database via the intermediate migration library. The target migration database is connected to an administrator access port. When receiving an access request from the administrator access port, it is determined whether a data conflict occurs. S4. Build a data storage management module, realize automatic mediation of data conflicts based on the data storage management module, form automatic instructions, control each module to run according to the instructions, and reduce the number of errors; In step S3, it also includes: The administrator accesses the port to send a data call access request, instructing to call the data in the target migration database. If the called data is standardized in the intermediate migration database or the called data is changed in real time, it is defined as a data conflict. In step S4, it also includes: Build a data conflict resolution mechanism, with priority conflict resolution as the only resolution mechanism; If in real-time state, the administrator accesses the port to send a data call access request, and the called data is standardized in the intermediate migration library, then the data call access is higher than the standardized processing; If the data changes in real time and the same data is standardized in the intermediate migration database, the data change is higher than the standardization; If the administrator accesses the port to send a data call access request in real-time state, the called data is standardized in the intermediate migration library, and at the same time the data change state occurs, the data standardization process has the lowest priority, and the others are processed concurrently; The concurrent processing refers to the real-time state where the administrator accesses the port to send a data call access request, and the called data simultaneously changes to a data state; Build a data storage management module to automate concurrent processing instructions: Perform change data analysis based on historical big data. In the historical big data, obtain the interval time and number of changes of the change data to form data combinations with different interval time and number of changes. The number of changes and the interval between each change of data changes under concurrent processing are obtained, recorded as data combination A, and data combination A is identified and included in the data combination in the historical big data to form a replacement combination of data combination A: Set the data combination in the historical big data to H category, and define the misjudgment probability of data combination A as , It refers to the probability of misclassifying data combination A into a certain category j in the data combination in historical big data. Data combination A and j must not belong to the same category. j represents other data combinations in the H-category data combination that are not of the same category as data combination A. Then we have: , in, Represents the probability of misjudgment loss of data combination A under the data combination in historical big data; represents fixed misjudgment loss; Then the misjudgment loss of data combination A is: , in, is the misjudgment loss of data combination A; Represents the prior probability of category j under the data combination in historical big data; The misjudgment loss is solved using the Mahalanobis distance, and the minimum value among all solutions is taken as the misjudgment loss of data combination A. Its corresponding category j is recorded as the priority category. At this time, the probability value of priority category j under the data combination of data combination A misjudged into the historical big data is the smallest, so the misjudgment loss is the smallest, which means that priority category j and data combination A are most similar. The mode of the number of changes in priority category j is taken as the output value; The difference solution is performed based on the output value and the number of data changes under concurrent processing. If the difference solution is less than the system preset quantity threshold, the data call access priority is higher than the data change status. Conversely, the data change status priority is higher than the data call access.

2. The data storage management method based on a big data platform according to claim 1, characterized in that: In step S1, it also includes: Get the original data capacity value and calculate the data growth value per unit time; Calculate the estimated migration time based on the average migration speed of the original data capacity value ; Calculate the estimated target migration database resource capacity: , in, Refers to the estimated target migration database resource capacity; Refers to the original data capacity value; Refers to the data growth value per unit time; If the measured target migration database resource capacity is lower than the estimated target migration database resource capacity, an alarm signal is generated to the administrator access port and data migration is not performed.

3. The data storage management method based on a big data platform according to claim 1, characterized in that: In step S2, it also includes: The unified definition refers to providing a unified data standard definition and platform logic model for the data platform, and realizing a unified definition of the data in the intermediate migration library; The format specification refers to the use of the same business standard specification, technical standard specification and management standard specification to standardize the data; the business standard specification refers to the association between data and business, and the verification of the scope and content of data; the technical standard specification refers to the standardization of the encoding rules and naming rules of data after the business standard specification work is completed, including data sources, data items, logical model standards, physical model standards, metadata, and public code value standards; the management standard specification refers to the clear application, approval, change, and sharing process of each standard data; The range constraint refers to the constraint that the attribute value of the data satisfies the defined enumeration value.

4. A data storage management system based on a big data platform, using the data storage management method based on a big data platform as claimed in claim 1, characterized in that: The system includes: historical data migration module, data platform construction module, data backflow analysis module and data storage management module; The historical data migration module is used to back up the original data, import it into the intermediate migration library, and measure the resource capacity of the target migration database to determine whether to start data migration; the data platform construction module is used to standardize the data, including unified definition, format specification and value range constraints, to ensure the consistency, accuracy and interoperability of the data, and to feedback the real-time data in the standardization process; the data reflux analysis module is used to call the original data in the target migration database, and to feedback the call information to the data storage management module; the data storage management module is used to receive the data processing information data transmitted by each module, mediate data conflicts, form automatic instructions, control each module to run according to the instructions, and reduce the number of errors; The output end of the historical data migration module is connected to the input end of the data platform construction module; the output end of the data platform construction module and the data backflow analysis module are connected to the input end of the data storage management module; the output end of the data storage management module is connected to the input end of the historical data migration module, the data platform construction module and the data backflow analysis module.

5. The data storage management system based on the big data platform according to claim 4, characterized in that: The historical data migration module includes a data backup unit and a judgment unit; The data backup unit is used to back up the original data and import it into the intermediate migration library; the judgment unit is used to measure the resource capacity of the target migration database and judge whether to start data migration; The output end of the data backup unit is connected to the input end of the judgment unit.

6. The data storage management system based on the big data platform according to claim 4, characterized in that: The data platform construction module includes a unified definition unit, a format specification unit and a value range constraint unit; The unified definition unit is used to provide a unified data standard definition and platform logic model for the data platform, and realize a unified definition of the data in the intermediate migration library; The format specification unit is used to standardize the data using the same business standard specification, technical standard specification and management standard specification; the business standard specification refers to the association between data and business, and the verification of the scope and content of data; the technical standard specification refers to the standardization of the encoding rules and naming rules of data after the business standard specification work is completed, including data sources, data items, logical model standards, physical model standards, metadata, and public code value standards; the management standard specification refers to the process of clarifying the application, approval, change, and sharing of each standard data; The range constraint unit is used to ensure that the attribute value of the data satisfies the constraints of the defined enumeration value.

7. The data storage management system based on the big data platform according to claim 4, characterized in that: The data backflow analysis module includes a node authentication unit and a task authorization unit; The node authentication unit is used to perform node communication authentication before data exchange between nodes when the administrator accesses the port to send a data call access request; the task authorization unit is used to synchronize access requests accurately to the second level. Before the synchronization access request is executed, the sender and the receiver need to confirm, otherwise synchronization cannot be achieved; The output end of the node authentication unit is connected to the input end of the task authorization unit.

8. The data storage management system based on a big data platform according to claim 4, characterized in that: The data storage management module includes an automatic adjustment unit and an instruction unit; The automatic mediation unit is used to receive data processing information data transmitted by each module, mediate data conflicts, and form automatic instructions; the instruction unit is used to control each module to run according to the instructions, complete the data storage task, and reduce the number of errors; The output end of the automatic adjustment unit is connected to the input end of the instruction unit.