A data management system and method based on a heterogeneous platform

Abstract

The application discloses a data management system and method based on a heterogeneous platform, and belongs to the technical field of data management of a big data service platform.The method first collects protocol characteristics and task metadata of a multi-source heterogeneous platform in the big data service platform, constructs a dynamic knowledge graph according to the interaction relationship between platforms, analyzes uploaded data by using the knowledge graph, classifies and marks real-time levels according to data formats and task priorities, formulates a priority scheduling strategy, dynamically adjusts data transmission paths and resource allocation by monitoring platform data flow, processor and memory usage in real time, combining data real-time and the scheduling strategy, on the basis of which, abnormal data in transmission is monitored and identified, abnormal characteristics are extracted, an abnormal data monitoring model is constructed, and abnormal information fed back by the model is used for dynamically updating the knowledge graph, continuously optimizing data classification and the scheduling strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management of big data service platform, and particularly relates to a data management system and method based on a heterogeneous platform. BACKGROUND

[0002] In the modern big data service platform architecture, each platform usually integrates nodes of multiple sources and types, including data collection nodes, computing nodes, storage nodes, etc., which interact with each other through different protocols (such as REST API, RPC, message queue, etc.) and have various data formats (such as JSON, Parquet, image / video stream, etc.), which leads to many challenges in processing multi-source heterogeneous data in the prior art:

[0003] Firstly, due to the significant differences in data formats, transmission protocols and communication mechanisms of different nodes, it takes a long time and is prone to errors to parse and integrate cross-node data, resulting in low data integration efficiency, for example: the processing flow of real-time data stream and batch data is quite different, and the traditional static configuration method is difficult to dynamically adapt; secondly, when high-priority tasks and low-priority tasks share limited computing and network resources, static scheduling strategies can easily cause delays in critical tasks, for example: real-time analysis tasks may not meet the timeliness requirements due to network congestion or computing node overload; in addition, the prior art usually relies on manual configuration of routing paths and resource allocation strategies, which is difficult to cope with dynamic changes in node load and data anomalies, and when the computing node is overloaded, data may be lost or delayed due to not being routed to an idle node in time. SUMMARY

[0004] The present application aims to provide a data management system and method based on a heterogeneous platform to solve the problems raised in the background.

[0005] To solve the above technical problems, the present application provides the following technical solution: a data management method based on a heterogeneous platform, characterized in that the method comprises:

[0006] Step S100: Collect the protocol characteristics and task metadata of multi-source heterogeneous platforms in the big data service platform, and construct a dynamic knowledge graph according to the interaction relationship between each heterogeneous platform;

[0007] Step S200: Analyze the data uploaded by each heterogeneous platform according to the knowledge graph, classify and label the data according to the data format and task priority, and develop a priority scheduling strategy for different categories of data; in the scheduling process, a dynamic weight adjustment mechanism is introduced to dynamically adjust the weights of task priority and real-time performance in the scheduling algorithm, so that high real-time performance and high priority data are processed first;

[0008] Step S300: Real-time monitoring of data flow, processor usage and memory occupancy of each heterogeneous platform to calculate the load index of each platform, and dynamically adjusting the data transmission path combined with the data real-time level and priority scheduling strategy, and adjusting the resource allocation according to the load condition; when the load of the platform is too high and the transmitted data is high real-time data, the platform with the lowest load is selected as the target of the alternative route by calculating the load index;

[0009] Step S400: Based on the adjustment of the transmission path and resource allocation, real-time monitoring and identifying abnormal data appearing in each heterogeneous platform during transmission, setting an abnormal monitoring threshold based on the protocol characteristics and task metadata, and determining abnormal data when the data index exceeds the threshold; extracting various abnormal features from the identified abnormal data, constructing an abnormal data monitoring model, and dynamically updating the knowledge graph according to the abnormal information feedback by the model, thereby continuously optimizing the data classification and scheduling strategy.

[0010] Further, the step S100 comprises:

[0011] Step S101: Set the set of heterogeneous platforms as P={P1, P2,..., Pi}, where Pi represents the ith heterogeneous platform, and each heterogeneous platform includes a data interface, a processing module and a communication module; For the ith heterogeneous platform Pi, collect the corresponding protocol characteristics Fc_i and task metadata Fr_i; The protocol characteristics include data format Ff_i, transmission rate Fv_i and communication protocol type Ft_i; The task metadata includes sampling frequency Fl_i, task priority Fp_i and signal strength Fs_i;

[0012] The data format Ff_i represents the organization structure of the data of the ith heterogeneous platform in the transmission process; The transmission rate Fv_i represents the data transmission speed of the ith heterogeneous platform under different loads; The communication protocol type Ft_i represents the communication protocol used by the ith heterogeneous platform and the corresponding communication mechanism; The sampling frequency Fl_i represents the number of times each sensor in the ith heterogeneous platform collects data per second; The task priority Fp_i represents the priority level of the tasks in the ith heterogeneous platform, including high priority, medium priority and low priority; The signal strength Fs_i is for the communication module of the ith heterogeneous platform, which is used to quantitatively represent the strength of the signal in the communication link;

[0013] Step S102: select Neo4j graph database to construct knowledge graph G(V, E), wherein the node V represents platform type, protocol feature and task metadata, and the edge E represents the association relationship between the nodes; for the i-th heterogeneous platform Pi, create the corresponding node vi in the knowledge graph G, and store the corresponding protocol feature and task metadata as the attribute of the node, the attribute set of the node vi is represented as Ai={Fc_i, Fr_i, Ff_i, Fv_i, Ft_i, Fl_i, Fp_i, Fs_i}; according to the data transmission between the platforms, the edge eij connecting the nodes vi and vj is established, wherein vj represents the node corresponding to the j-th heterogeneous platform Pj, the weight Wij of the edge is set to represent the data interaction frequency between the platform Pi and the platform Pj, and the data interaction frequency is calculated by counting the data amount transmitted between the two platforms within a certain time window: set the time window T, count the data amount Dij transmitted from the platform Pi to the platform Pj and the data amount Dji transmitted from the platform Pj to the platform Pi within the time window T, and the weight Wij of the edge eij is represented as: Wij=(Dij+Dji) / T; the sampling frequency Fl_i of the platform Pi is monitored in real time, and when the sampling frequency changes, the new sampling frequency value is updated to the attribute Ai of the node vi.

[0014] In the above technical solution, the dispersed platform information in the big data service platform is integrated, the knowledge system of the service platform is constructed, and data basis and association basis are provided for subsequent data classification, scheduling and the like;

[0015] Further, the step S200 comprises:

[0016] Step S201: from the dynamic knowledge graph G(V, E), the protocol features and task metadata represented by the node V are used to identify the same class of data; for the node vi corresponding to each heterogeneous platform Pi and the attribute set Ai thereof, the data category and the real-time level are determined according to the data format and the task priority; a category mapping function C is set, for the data uploaded by the heterogeneous platform Pi, represented as Di, the mapping function C is used to obtain the category C(Di) to which the data belongs; by traversing all the data uploaded by the platforms, the data with the same value is classified into the same class; the identified same class of data is collected into a unified data buffer area B, for the m-th class of data, the data uploaded by the platform Pi is represented as Dim, and all Dim are collected into the corresponding m-th sub-buffer area Bm in the buffer area B; at the same time, the data is marked with the real-time level Ri according to the task priority, wherein the high real-time data is marked as Ri=1, and the low real-time data is marked as Ri=0;

[0017] Step S202: According to the real-time level Ri of the data, set a priority scheduling strategy for each category data m; In the data buffer area B, the data in each sub-buffer area Bm is prioritized, high real-time data is processed and transmitted first, and low real-time data is processed in order; A priority scheduling algorithm is used: Q(Dim)=a*Ri+β*Fp_i; Where Q(Dim) represents the priority value of the data Dim, a and β are the real-time and priority weight coefficients set respectively, and a+β=1.

[0018] In the above technical solution, on the basis of the knowledge graph, the data is classified according to the data characteristics, and the resources are allocated in combination with the real-time and task priority, the data requirements of different tasks are balanced, and the basis is provided for subsequent data transmission path optimization;

[0019] The step S300 includes:

[0020] Step S301: Real-time monitoring of the load conditions of each platform, including data flow, processor usage and memory occupancy; Calculate the load index Li of each platform: Li=w1*Fi+w2*Ui+w3*Ni; Where Fi represents the data flow of the i-th heterogeneous platform, Ui represents the processor usage of the i-th heterogeneous platform, and Ni represents the memory occupancy of the i-th platform; w1, w2, and w3 represent the weight coefficients of data flow, processor usage and memory occupancy respectively, and w1+w2+w3=1;

[0021] Step S302: According to the load index Li and the data real-time level, dynamically adjust the transmission path of the data, when the load of the platform Pi is too high and the transmitted data is high real-time data, select the platform with the lowest load as the candidate routing target by calculating the load index Li: R(Dim)=argmin Pj (Lj+γ*(1-Ri)); Where R(Dim) represents the selected routing target of the data Dim transmitted from the platform Pi to the platform Pj, Lj represents the load index of the j-th heterogeneous platform Pj, and γ is an adjustment coefficient. By adjusting the value of γ, the platform load and data real-time are considered comprehensively to find the next routing target platform.

[0022] In the above technical solution, the data flow is decided in real time according to the dynamically changing load of each platform, which improves the self-adaptability of the system and the reliability of high real-time data transmission;

[0023] Further, the step S400 includes:

[0024] Step S401: After determining the scheduling strategy and transmission route, real-time monitoring of data transmitted by each heterogeneous platform is performed, and based on the data classification in step S200, monitoring indicators Mm={Mm1, Mm2,..., Mmn} are set for each category of data based on protocol features and task metadata, where Mmn represents the nth monitoring indicator of the mth category of data; in the data buffer area B, real-time monitoring of data in each sub-buffer area is performed, and for data Dim uploaded to Bm by the platform Pi, the corresponding protocol features and task metadata values are extracted and compared with the monitoring indicators Mm; if a feature value of the data Dim exceeds the corresponding monitoring indicator range, the data is determined to be abnormal data;

[0025] Step S402: Extracting abnormal features from all abnormal data, for each category of data m, all abnormal features of abnormal data are summarized to form an abnormal feature set Em={Em1, Em2,..., Ems}, where s is the number of abnormal data, and Ems represents the s th abnormal data feature of the m th category of abnormal data; statistical analysis is performed on the abnormal feature set Em to calculate the frequency of each feature; let Nmk be the number of times feature k appears in the abnormal feature set Em, then the frequency Ymk of feature k is Nmk / s; set a filtering threshold μ, sort all features in the abnormal feature set Em from high to low according to the frequency, and filter out the features with Ymk>μ as key abnormal features, denoted as Km={Km1, Km2,..., Kmt}, where t is the number of key abnormal features, and Kmt represents the t th feature in the key feature set;

[0026] Step S403: Using the key abnormal feature set Km obtained in step S402 as the input feature of the model; at the same time, using the abnormal data and normal data marked in step S401 as training samples; for each data sample, the corresponding key abnormal feature value is extracted and data normalization operation is performed, and the preprocessed data is divided into training set and test set according to the ratio of 8:2; an abnormal monitoring model is constructed through random forest algorithm, and the training set is used for training; through multiple random sampling and feature selection of training data, multiple decision trees are constructed, and in the training process, the number of model decision trees, the maximum depth of each decision tree, and the final determination of the hyperparameter combination are adjusted to complete the training of the model; real-time reception of newly transmitted data input into the model, when abnormal data is detected, the corresponding abnormal position and abnormal type are output; the abnormal position is located to a specific heterogeneous platform Pi; the abnormal type is data in the protocol feature and task metadata; according to the abnormal information output by the model, return to step S102 to update the attribute set Ai of the corresponding node vi in the knowledge graph and the corresponding edge weight data.

[0027] In the above technical solution, the data in the data transmission process is closely related to all previous steps, and the data is monitored for abnormalities, from data monitoring, abnormality analysis to knowledge graph correction, to ensure that the system continuously improves itself according to the actual situation during long-time operation, and maintains the accuracy and effectiveness of data management.

[0028] A data management system based on a heterogeneous platform, the system comprising: a heterogeneous platform data acquisition module, a data priority scheduling module, a transmission path dynamic adjustment module, a data anomaly monitoring module;

[0029] The heterogeneous platform data acquisition module is used to acquire the protocol characteristics and task metadata of multiple source heterogeneous platforms in a big data service platform, and to construct a dynamic knowledge graph according to the interaction relationship between the heterogeneous platforms;

[0030] The data priority scheduling module analyzes the data uploaded by each heterogeneous platform according to the knowledge graph, classifies and labels the data according to the data format and task priority, and formulates a priority scheduling strategy for different categories of data; in the scheduling process, a dynamic weight adjustment mechanism is introduced to dynamically adjust the weight of task priority and real-time performance in the scheduling algorithm, so that high real-time performance and high priority data are processed first;

[0031] The transmission path dynamic adjustment module calculates the load index of each platform by real-time monitoring of the data flow, processor usage and memory occupancy of each heterogeneous platform, and dynamically adjusts the data transmission path in combination with the data real-time performance level and priority scheduling strategy, and adjusts the resource allocation according to the load condition; when the load of the platform is too high and the transmitted data is high real-time performance data, the platform with the lowest load is selected as the target of the alternative route by calculating the load index;

[0032] The data anomaly monitoring module monitors and identifies abnormal data of each heterogeneous platform in the transmission process based on the adjustment of the transmission path and resource allocation, sets an abnormal monitoring threshold based on the protocol characteristics and task metadata, and determines the abnormal data when the data index exceeds the threshold; abnormal features of various types are extracted from the identified abnormal data, an abnormal data monitoring model is constructed, and the knowledge graph is dynamically updated according to the abnormal information fed back by the model, so as to continuously optimize the data classification and scheduling strategy.

[0033] Compared with the prior art, the beneficial effects achieved by the present application are:

[0034] The present application realizes the unified management and dynamic update of the data of multiple source heterogeneous platforms in a big data service platform by constructing a dynamic knowledge graph, and ensures the priority transmission and processing of high real-time performance data through data classification and priority scheduling strategy, thereby improving the efficiency and accuracy of data processing;

[0035] The application can effectively avoid data congestion and resource waste by monitoring the load of each heterogeneous platform in real time, dynamically adjusting the data transmission path and resource allocation, and ensuring the smoothness and stability of data transmission; meanwhile, the overall performance and resource utilization of the system are further improved by reasonably allocating resources according to the load condition.

[0036] The application can accurately identify abnormal data based on protocol characteristics and task metadata, extract abnormal features to construct an abnormal data monitoring model, realize real-time monitoring and accurate positioning of abnormal data, and dynamically update the knowledge graph according to the abnormal information fed back by the model, so as to realize the continuous optimization and upgrading of the platform. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0038] Figure 1 A method flow chart of a data management method based on a heterogeneous platform. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0040] Please refer to Figure 1 The application provides a technical solution: a data management method based on a heterogeneous platform, characterized in that the method comprises:

[0041] Step S100: Collecting protocol characteristics and task metadata of multiple source heterogeneous platforms in a big data service platform, and constructing a dynamic knowledge graph according to the interaction relationship between each heterogeneous platform;

[0042] Step S200: Analyzing the data uploaded by each heterogeneous platform according to the knowledge graph, classifying and marking the data according to the data format and task priority, and marking the real-time level, and formulating a priority scheduling strategy for different categories of data; in the scheduling process, a dynamic weight adjustment mechanism is introduced to dynamically adjust the weight of task priority and real-time in the scheduling algorithm, so that high real-time and high priority data are processed first;

[0043] Step S300: Real-time monitoring of data flow, processor usage and memory occupancy of each heterogeneous platform to calculate the load index of each platform, and dynamically adjusting the data transmission path combined with the data real-time level and priority scheduling strategy, and adjusting the resource allocation according to the load condition; when the load of the platform is too high and the transmitted data is high real-time data, the platform with the lowest load is selected as the target of the alternative route by calculating the load index;

[0044] Step S400: Based on the adjustment of the transmission path and resource allocation, real-time monitoring and identifying abnormal data appearing in each heterogeneous platform during transmission, setting an abnormal monitoring threshold based on the protocol characteristics and task metadata, and determining abnormal data when the data index exceeds the threshold; extracting various abnormal features from the identified abnormal data, constructing an abnormal data monitoring model, and dynamically updating the knowledge graph according to the abnormal information feedback by the model, thereby continuously optimizing the data classification and scheduling strategy.

[0045] Further, the step S100 includes:

[0046] Step S101: Set the set of heterogeneous platforms as P={P1, P2,..., Pi}, where Pi represents the ith heterogeneous platform, and each heterogeneous platform includes a data interface, a processing module and a communication module; For the ith heterogeneous platform Pi, collect the corresponding protocol characteristics Fc_i and task metadata Fr_i; The protocol characteristics include data format Ff_i, transmission rate Fv_i and communication protocol type Ft_i; The task metadata includes sampling frequency Fl_i, task priority Fp_i and signal strength Fs_i;

[0047] The data format Ff_i represents the organization structure of the data of the ith heterogeneous platform in the transmission process; The transmission rate Fv_i represents the data transmission speed of the ith heterogeneous platform under different loads; The communication protocol type Ft_i represents the communication protocol used by the ith heterogeneous platform and the corresponding communication mechanism; The sampling frequency Fl_i represents the number of times each sensor in the ith heterogeneous platform collects data per second; The task priority Fp_i represents the priority level of the tasks in the ith heterogeneous platform, including high priority, medium priority and low priority; The signal strength Fs_i is for the communication module of the ith heterogeneous platform, which is used to quantitatively represent the strength of the signal in the communication link;

[0048] Step S102: select Neo4j graph database to construct a knowledge graph G(V, E), wherein the node V represents the platform type, the protocol feature and the task metadata, and the edge E represents the association relationship between the nodes; for the i-th heterogeneous platform Pi, a corresponding node vi is created in the knowledge graph G, and the corresponding protocol feature and task metadata are stored as the attributes of the node, and the attribute set of the node vi is represented as Ai={Fc_i, Fr_i, Ff_i, Fv_i, Ft_i, Fl_i, Fp_i, Fs_i}; according to the data transmission between the platforms, an edge eij is established to connect the nodes vi and vj, wherein vj represents the node corresponding to the j-th heterogeneous platform Pj, the weight Wij of the edge is set to represent the data interaction frequency between the platform Pi and the platform Pj, and the data interaction frequency is calculated by counting the data amount transmitted between the two platforms within a certain time window: the time window T is set, the data amount Dij transmitted from the platform Pi to the platform Pj and the data amount Dji transmitted from the platform Pj to the platform Pi are counted within the time window T, and the weight Wij of the edge eij is represented as: Wij=(Dij+Dji) / T; the sampling frequency Fl_i of the platform Pi is monitored in real time, and when the sampling frequency changes, the new sampling frequency value is updated to the attribute Ai of the node vi.

[0049] In the above technical solution, the dispersed platform information in the big data service platform is integrated, the knowledge system of the service platform is constructed, and data basis and association basis are provided for subsequent data classification, scheduling and the like;

[0050] Further, the step S200 comprises:

[0051] Step S201: from the dynamic knowledge graph G(V, E), the protocol feature and the task metadata represented by the node V are used to identify the same type of data; for the node vi corresponding to each heterogeneous platform Pi and the attribute set Ai thereof, the data category and the real-time level are determined according to the data format and the task priority; a category mapping function C is set, for the data uploaded by the heterogeneous platform Pi, represented as Di, the data category C(Di) to which the data belongs is obtained through the mapping function C; by traversing all the data uploaded by the platforms, the data with the same value is classified into the same category; the identified same type of data is collected into a unified data buffer area B, for the m-th type of data, the data uploaded by the platform Pi is represented as Dim, and all Dim are collected into the corresponding m-th sub-buffer area Bm in the buffer area B; at the same time, the data is marked with the real-time level Ri according to the task priority, wherein the high real-time data is marked as Ri=1, and the low real-time data is marked as Ri=0;

[0052] Step S202: According to the real-time level Ri of the data, set a priority scheduling strategy for each category data m; in the data buffer area B, prioritize the data in each sub-buffer area Bm, and process and transmit the high real-time data first, and process the low real-time data in sequence; use a priority scheduling algorithm: Q(Dim)=a*Ri+β*Fp_i; wherein Q(Dim) represents the priority value of the data Dim, a and β are respectively the real-time and priority weight coefficients set, and a+β=1.

[0053] In the above technical solution, on the basis of the knowledge graph, the data is classified according to the data characteristics, and the resources are allocated in combination with the real-time and task priority, the data requirements of different tasks are balanced, and the basis is provided for subsequent data transmission path optimization;

[0054] The step S300 includes:

[0055] Step S301: Real-time monitoring of the load conditions of each platform, including data flow, processor usage and memory occupancy; calculating the load index Li of each platform: Li=w1*Fi+w2*Ui+w3*Ni; wherein Fi represents the data flow of the i-th heterogeneous platform, Ui represents the processor usage of the i-th heterogeneous platform, and Ni represents the memory occupancy of the i-th platform; w1, w2 and w3 represent the weight coefficients of the data flow, the processor usage and the memory occupancy respectively, and w1+w2+w3=1;

[0056] Step S302: According to the load index Li and the data real-time level, dynamically adjust the transmission path of the data, when the load of the platform Pi is too high and the transmitted data is high real-time data, select the platform with the lowest load as the candidate routing target by calculating the load index Li: R(Dim)=argmin Pj (Lj+γ*(1-Ri)); wherein R(Dim) represents the selected routing target of the data Dim transmitted from the platform Pi to the platform Pj, Lj represents the load index of the j-th heterogeneous platform Pj, and γ is an adjustment coefficient. By adjusting the value of γ, the platform load and the data real-time are considered comprehensively to find the next routing target platform.

[0057] In the above technical solution, the data flow is decided in real time according to the dynamically changing load of each platform, and the self-adaptability of the system and the reliability of high real-time data transmission are improved.

[0058] Further, the step S400 includes:

[0059] Step S401: After determining the scheduling strategy and transmission route, real-time monitoring of data transmitted by each heterogeneous platform is performed, and based on the data classification in step S200, monitoring indicators Mm={Mm1, Mm2,..., Mmn} are set for each category of data based on protocol features and task metadata, where Mmn represents the nth monitoring indicator of the mth category of data; in the data buffer area B, real-time monitoring of data in each sub-buffer area is performed, and for data Dim uploaded to Bm by the platform Pi, the corresponding protocol features and task metadata values are extracted and compared with the monitoring indicators Mm; if a feature value of the data Dim exceeds the corresponding monitoring indicator range, the data is determined to be abnormal data;

[0060] Step S402: Extracting abnormal features from all abnormal data, for each category of data m, all abnormal features of abnormal data are summarized to form an abnormal feature set Em={Em1, Em2,..., Ems}, where s is the number of abnormal data, and Ems represents the s th abnormal data feature of the m th category of abnormal data; statistical analysis is performed on the abnormal feature set Em to calculate the frequency of each feature; let Nmk be the number of times that feature k appears in the abnormal feature set Em, then the frequency Ymk of feature k is Nmk / s; set a filtering threshold μ, sort all features in the abnormal feature set Em from high to low according to the frequency, and filter out the features with Ymk>μ as key abnormal features, denoted as Km={Km1, Km2,..., Kmt}, where t is the number of key abnormal features, and Kmt represents the t th feature in the key feature set;

[0061] Step S403: The key abnormal feature set Km obtained in step S402 is used as the input feature of the model; at the same time, the abnormal data and normal data marked in step S401 are used as training samples; for each data sample, the corresponding key abnormal feature value is extracted, and data normalization operation is performed; the preprocessed data is divided into training set and test set according to the ratio of 8:2; an abnormal monitoring model is constructed through random forest algorithm, and the training set is used for training; through multiple random sampling and feature selection of training data, multiple decision trees are constructed, and in the training process, the number of model decision trees, the maximum depth of each decision tree, and the final determination of the hyperparameter combination are adjusted to complete the training of the model; real-time reception of newly transmitted data input into the model, when abnormal data is detected, the corresponding abnormal position and abnormal type are output; the abnormal position is located to a specific heterogeneous platform Pi; the abnormal type is data in the protocol feature and task metadata; according to the abnormal information output by the model, return to step S102 to update the attribute set Ai of the corresponding node vi in the knowledge graph and the corresponding edge weight data.

[0062] In the above technical solution, the data in the data transmission process is closely related to all previous steps, and the data is monitored for abnormalities, from data monitoring, abnormality analysis to knowledge graph correction, to ensure that the system continuously improves itself according to the actual situation during long-time operation, and maintains the accuracy and effectiveness of data management;

[0063] A data management system based on a heterogeneous platform, the system comprising: a heterogeneous platform data acquisition module, a data priority scheduling module, a transmission path dynamic adjustment module, a data anomaly monitoring module;

[0064] The heterogeneous platform data acquisition module is used for acquiring protocol characteristics and task metadata of multiple source heterogeneous platforms in a big data service platform, and constructing a dynamic knowledge graph according to the interaction relationship between the heterogeneous platforms;

[0065] The data priority scheduling module analyzes the data uploaded by each heterogeneous platform according to the knowledge graph, classifies and labels the data according to the data format and task priority, and formulates a priority scheduling strategy for different categories of data; in the scheduling process, a dynamic weight adjustment mechanism is introduced to dynamically adjust the weight of task priority and real-time performance in the scheduling algorithm, so that high real-time performance and high priority data are processed first;

[0066] The transmission path dynamic adjustment module calculates the load index of each platform by real-time monitoring of the data flow, processor usage and memory occupancy of each heterogeneous platform, and dynamically adjusts the data transmission path in combination with the data real-time performance level and priority scheduling strategy, and adjusts the resource allocation according to the load condition; when the load of the platform is too high and the transmitted data is high real-time performance data, the platform with the lowest load is selected as the target of the alternative route by calculating the load index;

[0067] The data anomaly monitoring module monitors and identifies abnormal data of each heterogeneous platform in the transmission process based on the adjustment of the transmission path and resource allocation, sets an abnormal monitoring threshold based on the protocol characteristics and task metadata, and determines the abnormal data when the data index exceeds the threshold; abnormal features of various types are extracted from the identified abnormal data, an abnormal data monitoring model is constructed, and the knowledge graph is dynamically updated according to the abnormal information fed back by the model, so as to continuously optimize the data classification and scheduling strategy.

[0068] Embodiments of the present application:

[0069] Step S100: set the heterogeneous platform set of the big data service platform as P={P1, P2, P3}, wherein P1 represents a data acquisition platform, P2 represents a data calculation platform, and P3 represents a data storage platform;

[0070] For the data acquisition platform P1:

[0071] Protocol features: ① Data format Ff_1: JSON format is adopted, and the data contains timestamp, data source identification and specific data value; ② Transmission rate Fv_1: when the load is low in the initial stage of data collection, the transmission rate is 100 Mbps; when a large amount of data is collected concurrently, the transmission rate is increased to 1 Gbps; ③ Communication protocol type Ft_1: REST API protocol, data transmission is carried out through HTTP request, and GET, POST and other methods are supported;

[0072] Task metadata: ① Sampling frequency Fl_1: for real-time sensor data, sampling is carried out every 1 ms; for non-real-time data, sampling is carried out every 1 hour; ② Task priority Fp_1: real-time sensor data collection is high priority; historical data supplement is low priority; ③ Signal strength Fs_1: if wireless communication mode is adopted, the signal strength is stable at about -70 dBm in a good indoor environment;

[0073] Data computing platform P2:

[0074] Protocol features: ① Data format Ff_2: Parquet format is adopted, which is suitable for large-scale data storage and efficient computing, and has columnar storage and compression functions; ② Transmission rate Fv_2: when routine computing tasks are carried out, the transmission rate is 500 Mbps; when complex batch computing is carried out, it can be increased to 5 Gbps; ③ Communication protocol type Ft_2: RPC protocol is used for efficient remote procedure call to realize communication between different computing nodes;

[0075] Task metadata: ① Sampling frequency Fl_2: according to the requirements of computing tasks, the computing state is sampled every 100 ms; ② Task priority Fp_2: real-time data analysis task is high priority; batch data processing task is medium priority; data cleaning and archiving task is low priority; ③ Signal strength Fs_2: wired network is adopted for internal communication, and the signal strength is stable;

[0076] Data storage platform P3:

[0077] Protocol features: ① Data format Ff_3: HDFS block storage format is adopted, and data is segmented into fixed-size blocks for distributed storage; ② Transmission rate Fv_3: when data is written, the transmission rate is 200 Mbps; when data is read, according to the concurrent reading situation, it can reach 10 Gbps at most; ③ Communication protocol type Ft_3: HDFS protocol is used for interaction with Hadoop distributed file system;

[0078] Task metadata: ①Sampling frequency Fl_3: sample the storage status every 5 minutes, including disk usage, file quantity, etc.; ②Task priority Fp_3: data backup task is high priority; data migration task is medium priority; data deletion task is low priority; ③Signal strength Fs_3: internal communication uses wired network, signal strength is stable;

[0079] Select Neo4j graph database to construct knowledge graph G(V, E), where node V represents platform type, protocol feature and task metadata, and edge E represents the association relationship between nodes; take P1 as an example, A1={Fc_1, Fr_1, Ff_1, Fv_1, Ft_1, Fl_1, Fp_1, Fs_1};

[0080] Calculate edge weight: set time window T=60s, in this minute, statistics to P1 transmits 500MB data D12=500MB to P2, P2 transmits 100MB data D21=100MB to P1, then the weight of edge e12 is W12=[(500+100)*8] / (60*100)≈0.08Mbps; calculate other edge weights in the same way to construct the association relationship between platforms; real-time monitoring sampling frequency: if the real-time sensor data sampling frequency of P1 is temporarily increased from every 1ms to every 5ms due to large data volume, immediately update the new data to the attribute Fl1 of node v1;

[0081] Step S200: analysis from dynamic knowledge graph G(V, E), for P1 uploaded data: if the data format is JSON real-time sensor data and the task priority is high, it is judged as real-time key data class through category mapping function C, marked as the first class, and the real-time level R1=1; if it is non-real-time historical data and the priority is low, it is classified as ordinary data class, marked as the second class, R2=0; classify the data of P2 and P3 in the same way, and store each class of data in data buffer area B, and store the real-time key data D11 of P1 in the first sub-buffer area B1 of buffer area B;

[0082] Set real-time weight coefficient α=0.6 and priority weight coefficient β=0.4; in buffer area B, for high real-time real-time key data in B1, calculate according to algorithm Q(D11)=0.6*1+0.4*Fp1; where Fp_1 is the corresponding value 3 of high priority, the priority value is high, and it is processed and transmitted preferentially; while the ordinary data in B2 waits for processing in order, ensuring fast response of key data;

[0083] Step S300: Real-time monitoring of the load of each platform: the data collection platform P1 data flow F1 = 800 Mbps, processor usage U1 = 70%, memory occupancy N1 = 60%, set the data flow, processor usage, memory occupancy weight coefficient w1 = 0.4, w2 = 0.4, w3 = 0.2, then the load index L1 = 0.4 * 800 + 0.4 * 70 + 0.2 * 60 = 72; similarly, the load indexes of P2 and P3 are calculated; at this time, P1 is overloaded and transmits high real-time critical data R1 = 1, and the adjustment coefficient γ is dynamically set according to the network state: γ = 0.5 when the network is normal, and γ = 2 when the network is congested; in this embodiment, the adjustment coefficient γ = 2; calculate the load indexes of other platforms, L2 = 50, L3 = 30, and select P3 as the target of the backup route through the formula R(D1j) = argmin Pj (Lj+γ*(1-R1)= argmin Pj (Lj), filter out P3 as the target of the backup route, and forward the data originally transmitted directly from P1 through P3 to avoid the high load of P1; at the same time, since P1 is overloaded, its non-critical task resource allocation is appropriately reduced to prioritize real-time critical data processing.

[0084] Step S400: After determining the scheduling and transmission route, set monitoring indicators for each type of data based on data classification;

[0085] For real-time critical data (type 1): set the normal range of sensor data values [0, 1000] in the data format as the monitoring indicator M11; monitor the data D11 uploaded to B1 on P1, and the sensor data value is 1200, which is outside the range of M11, and is determined as abnormal data; collect all abnormal data characteristics, collect 100 abnormal data in a period of time, and statistically find that the sensor data value abnormality occurs 60 times; set the screening threshold μ = 0.4, sort by frequency, and the sensor data abnormality characteristic frequency Y11 = 60 / 100 = 0.6 > μ, and the key abnormality characteristic K1 = {K11} is screened; the abnormality monitoring model takes K1 as the input characteristic, combines the labeled abnormal and normal real-time critical data as the training sample, extracts the characteristic value and normalizes it, and divides the training set and the test set according to 8:2; use the random forest algorithm to build the model, adjust the number of decision trees to 100 during training, and the maximum depth is 8 layers, and the training is completed; real-time receive new data, detect abnormality, and output the abnormal platform: P1 and the abnormal type: sensor data value abnormality; according to this information, return to step S102 to update the attribute set A1 of the corresponding node v1 in the knowledge graph and the corresponding edge weight data.

[0086] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalent ones. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data management method based on a heterogeneous platform, characterized by: The method comprises: Step S100: Collect protocol features and task metadata from multiple heterogeneous platforms in the big data service platform, and construct a dynamic knowledge graph based on the interaction relationships between the heterogeneous platforms; Step S200: Analyze the data uploaded by various heterogeneous platforms based on the knowledge graph, classify the data according to the data format and task priority, and mark the real-time level. Develop priority scheduling strategies for different data categories. During the scheduling process, introduce a dynamic weight adjustment mechanism to dynamically adjust the weights of task priority and real-time performance in the scheduling algorithm, so that high-real-time and high-priority data are processed first. Step S300: Monitor the data traffic, processor usage, and memory occupancy of each heterogeneous platform in real time to calculate the load index of each platform. Dynamically adjust the data transmission path based on the data real-time level and priority scheduling strategy, and adjust resource allocation based on the load status. When the platform load is too high and the transmitted data is high-real-time data, the load index is calculated to select the platform with the lowest load as the alternative routing target. Step S400: Based on the adjustment of transmission paths and resource allocation, abnormal data appearing on various heterogeneous platforms during the transmission process are monitored and identified in real time. Based on protocol characteristics and task metadata, an abnormal monitoring threshold is set. When data indicators exceed the threshold, they are determined to be abnormal data. Various abnormal features are extracted from the identified abnormal data, and an abnormal data monitoring model is constructed. The knowledge graph is dynamically updated according to the abnormal information fed back by the model, thereby continuously optimizing data classification and scheduling strategies.

2. The data management method based on a heterogeneous platform according to claim 1, characterized in that: The step S100 includes: Step S101: Set the heterogeneous platform set to P = {P1, P2, ..., Pi}, where Pi represents the i-th heterogeneous platform. Each heterogeneous platform includes a data interface, a processing module, and a communication module. For the i-th heterogeneous platform Pi, collect the corresponding protocol features Fc_i and task metadata Fr_i. The protocol features include data format Ff_i, transmission rate Fv_i, and communication protocol type Ft_i. The task metadata includes sampling frequency Fl_i, task priority Fp_i, and signal strength Fs_i. The data format Ff_i represents the organizational structure of the data of the i-th heterogeneous platform during transmission; the transmission rate Fv_i represents the data transmission speed of the i-th heterogeneous platform under different loads; the communication protocol type Ft_i represents the communication protocol used by the i-th heterogeneous platform and the corresponding communication mechanism; the sampling frequency Fl_i represents the number of times each sensor in the i-th heterogeneous platform collects data per second; the task priority Fp_i represents the priority level of the task in the i-th heterogeneous platform, including high priority, medium priority and low priority; the signal strength Fs_i is for the communication module of the i-th heterogeneous platform, and is used to quantify the strength of the signal in the communication link; Step S102: Use Neo4j graph database to build knowledge graph G(V,E), where node V represents platform type, protocol features and task metadata, and edge E represents the relationship between nodes; for the i-th heterogeneous platform Pi, create a corresponding node vi in ​​the knowledge graph G, and store the corresponding protocol features and task metadata as node attributes. The attribute set of node vi is represented as Ai={Fc_i, Fr_i, Ff_i, Fv_i, Ft_i, Fl_i, Fp_i, Fs_i}; according to the data transmission between platforms, establish edge eij to connect nodes vi and vj, where vj represents the j-th For the node corresponding to the heterogeneous platform Pj, the edge weight Wij is set to represent the frequency of data interaction between the platform Pi and the platform Pj. The data interaction frequency is calculated by counting the amount of data transmitted between the two platforms within a certain time window: set the time window T, and within the time window T, count the amount of data Dij transmitted from the platform Pi to the platform Pj, and the amount of data Dji transmitted from the platform Pj to the platform Pi. The weight Wij of the edge eij is expressed as: Wij=(Dij+Dji) / T; monitor the sampling frequency Fl_i of the platform Pi in real time. When the sampling frequency changes, update the new sampling frequency value to the attribute Ai of the node vi.

3. The data management method based on a heterogeneous platform according to claim 2, characterized in that: The step S200 includes: Step S201: From the dynamic knowledge graph G(V,E), identify the same type of data through the protocol features and task metadata represented by the node V; for each node vi and its attribute set Ai corresponding to the heterogeneous platform Pi, determine the data category and mark the real-time level according to the data format and task priority; set the category mapping function C, for the data uploaded by the heterogeneous platform Pi, it is represented as Di, and the category C(Di) to which the data belongs is obtained through the mapping function C; by traversing the data uploaded by all platforms, the data with the same value are classified into the same category; the identified data of the same category are aggregated into a unified data cache B, for the mth type of data, the data uploaded on the platform Pi is recorded as Dim, and all Dims are aggregated into the corresponding mth sub-cache Bm in the cache B; at the same time, the real-time level Ri is marked for the data according to the task priority, where high real-time data is marked as Ri=1 and low real-time data is marked as Ri=0; Step S202: According to the real-time level Ri of the data, a priority scheduling strategy is set for each category of data m; in the data cache area B, the data in each sub-cache area Bm is prioritized, high real-time data is processed and transmitted first, and low real-time data is processed in sequence; a priority scheduling algorithm is adopted: Q(Dim)=α*Ri+β*Fp_i; where Q(Dim) represents the priority value of the data Dim, α and β are the set real-time and priority weight coefficients respectively, and α+β=1.

4. The data management method based on a heterogeneous platform according to claim 3, characterized in that: The step S300 includes: Step S301: Monitor the load of each platform in real time, including data traffic, processor usage, and memory occupancy; calculate the load index Li of each platform: Li=w1*Fi+w2*Ui+w3*Ni; where Fi represents the data traffic of the i-th heterogeneous platform, Ui represents the processor usage of the i-th heterogeneous platform, and Ni represents the memory occupancy of the i-th platform; w1, w2, and w3 represent the weight coefficients of data traffic, processor usage, and memory occupancy, respectively, and w1+w2+w3=1; Step S302 dynamically adjusts the data transmission path according to the load index Li and the data real-time level. When the load of platform Pi is too high and the transmitted data is high-real-time data, the platform with the lowest load is selected as the alternative routing target by calculating the load index Li: R(Dim)=argmin Pj (Lj+γ*(1-Ri)); where R(Dim) represents the routing target selected by the data Dim transmitted from platform Pi to platform Pj, Lj represents the load index of the j-th heterogeneous platform Pj, and γ is the adjustment coefficient. By adjusting the value of γ, the next routing target platform is found by comprehensively considering the platform load and data real-time performance.

5. The data management method based on a heterogeneous platform according to claim 4, characterized in that: The step S400 includes: Step S401: After determining the scheduling strategy and transmission route, the data transmitted by each heterogeneous platform is monitored in real time. Based on the data classification in step S200, a monitoring indicator Mm={Mm1,Mm2,...,Mmn} is set for each category of data based on protocol characteristics and task metadata, where Mmn represents the nth monitoring indicator of the mth category of data. In data buffer area B, the data in each sub-buffer area is monitored in real time. For the data Dim uploaded by platform Pi to Bm, its corresponding protocol characteristics and task metadata values ​​are extracted and compared with the monitoring indicator Mm. If a feature value of the data Dim exceeds the corresponding monitoring indicator range, the data is determined to be abnormal data. Step S402: Extract abnormal features from all data marked as abnormal. For each category of data m, summarize the abnormal features of all abnormal data to form an abnormal feature set Em = {Em1, Em2, ..., Ems}, where s is the number of abnormal data, and Ems represents the sth abnormal data feature of the mth category abnormal data; perform statistical analysis on the abnormal feature set Em and calculate the frequency of occurrence of each feature; let the number of times feature k appears in the abnormal feature set Em be Nmk, then the frequency of occurrence of feature k is Ymk = Nmk / s; set a screening threshold μ, sort all features in the abnormal feature set Em from high to low according to the frequency of occurrence, and screen out features with Ymk>μ as key abnormal features, expressed as Km = {Km1, Km2, ..., Kmt}, where t is the number of key abnormal features, and Kmt represents the tth feature in the key feature set; Step S403: Use the key abnormal feature set Km obtained in step S402 as the input feature of the model; at the same time, use the abnormal data and normal data marked in step S401 as training samples; for each data sample, extract its corresponding key abnormal feature value, and perform data normalization operation, and divide the preprocessed data into training set and test set in a ratio of 8:2; construct an abnormal monitoring model through the random forest algorithm, and use the training set for training; construct multiple decision trees by performing multiple random sampling and feature selection on the training data. During the training process, adjust the number of model decision trees and the maximum depth of each decision tree, and finally determine the hyperparameter combination to complete the model training; receive the newly transmitted data input into the model in real time, and when abnormal data is detected, output the corresponding abnormal position and abnormal type; the abnormal position is located on the specific heterogeneous platform Pi; the abnormal type is the data in the protocol characteristics and task metadata; according to the abnormal information output by the model, return to step S102 to update the attribute set Ai of the corresponding node vi in ​​the knowledge graph and the corresponding edge weight data.

6. A data management system based on a heterogeneous platform, characterized by: The system includes: a heterogeneous platform data acquisition module, a data priority scheduling module, a transmission path dynamic adjustment module, and a data anomaly monitoring module; The heterogeneous platform data acquisition module is used to collect protocol features and task metadata of multi-source heterogeneous platforms in the big data service platform, and build a dynamic knowledge graph based on the interaction relationship between the heterogeneous platforms; The data priority scheduling module analyzes the data uploaded by various heterogeneous platforms based on the knowledge graph, classifies the data according to the data format and task priority, and marks the real-time level, and formulates priority scheduling strategies for different categories of data; during the scheduling process, a dynamic weight adjustment mechanism is introduced to dynamically adjust the weights of task priority and real-time performance in the scheduling algorithm, so that high real-time and high-priority data are processed first; The transmission path dynamic adjustment module calculates the load index of each platform by monitoring the data flow, processor usage, and memory occupancy of each heterogeneous platform in real time. It dynamically adjusts the data transmission path based on the data real-time level and priority scheduling strategy, and adjusts resource allocation according to the load status. When the load of a platform is too high and the transmitted data is high-real-time data, the load index is calculated to select the platform with the lowest load as the alternative routing target; The data anomaly monitoring module monitors and identifies abnormal data appearing on various heterogeneous platforms during the transmission process in real time based on the adjustment of transmission paths and resource allocation. It sets an anomaly monitoring threshold based on protocol characteristics and task metadata, and determines that data is abnormal when the data indicator exceeds the threshold. It extracts various types of abnormal features from the identified abnormal data, constructs an abnormal data monitoring model, and dynamically updates the knowledge graph based on the abnormal information fed back by the model, thereby continuously optimizing data classification and scheduling strategies.

7. The data management system based on a heterogeneous platform according to claim 6, characterized in that: The heterogeneous platform data acquisition module includes a platform information acquisition unit and a knowledge graph construction unit; The platform information collection unit is used to set a set of heterogeneous platforms and collect protocol characteristics of each heterogeneous platform, including data format, transmission rate, and communication protocol type; and task metadata, including sampling frequency, task priority, and signal strength; The knowledge graph construction unit uses the Neo4j graph database to construct a knowledge graph, stores the collected data as node attributes, and establishes edges and assigns weights based on data transmission between platforms.

8. The data management system based on a heterogeneous platform according to claim 6, characterized in that: The data priority scheduling module includes a data classification unit and a priority scheduling unit; the data classification unit determines the data category according to the data format and task priority, aggregates the data of the same category into a unified data buffer area, and marks the real-time level; The priority scheduling unit sets a priority scheduling strategy according to the real-time level of the data and the task priority, and sorts and processes the data.

9. The data management system based on a heterogeneous platform according to claim 6, characterized in that: The transmission path dynamic adjustment module includes a load monitoring unit and a path adjustment decision unit; the load monitoring unit is used to monitor and calculate the load index of each platform, including data traffic, processor usage and memory occupancy; the path adjustment decision unit selects the platform with the lowest load as an alternative routing target based on the load index and data real-time level, and adjusts the data transmission path.

10. The data management system based on a heterogeneous platform according to claim 6, characterized in that: The data anomaly monitoring module includes a data monitoring unit, an anomaly analysis unit, and an anomaly monitoring model building unit; the data monitoring unit sets monitoring indicators for each category of data, extracts the protocol characteristics and task metadata values ​​of the data, and compares them with the monitoring indicators to determine abnormal data; the anomaly analysis unit extracts abnormal features from the abnormal data, performs statistical analysis, and screens out key abnormal features; The anomaly monitoring model construction unit uses key anomaly features as input features to construct a random forest anomaly monitoring model, perform model training and real-time anomaly detection.

Images