Multi-element big data sharing method and system

Abstract

The invention discloses a multivariate big data sharing method and system, relates to the field of data sharing, and solves the problem of low sharing efficiency of an existing big data sharing mode, and the method comprises the following steps: step F1, respectively obtaining a plurality of monitoring index data to obtain multivariate shared data; f2, analyzing the multivariate shared data, and dividing the multiple types of data into first type of shared data and second type of shared data to obtain shared type divided data; f3, performing big data sharing on the first type of shared data according to the multivariate shared data and the shared type division data; f4, performing big data sharing on the second type of shared data according to the multivariate shared data and the shared type division data; according to the data sharing method and device, targeted sharing modes are adopted for different types of data, the data sharing efficiency can be effectively guaranteed, and the risk of data loss is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of data sharing and relates to big data technology, and specifically to a multivariate big data sharing method and system. Background Art

[0002] The existing multivariate big data sharing methods have the following defects:

[0003] 1. Existing multivariate big data sharing methods usually use a centralized data management system, where all data is stored on a single platform for unified management and sharing, which can easily lead to system paralysis due to a single location failure;

[0004] 2. The existing multivariate big data sharing method does not distinguish between different data generated on the same platform, resulting in a lack of pertinence in the sharing method, which can easily lead to low sharing efficiency and waste of resources;

[0005] To this end, we propose a multivariate big data sharing method and system. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention aims to provide a multivariate big data sharing method and system. The present invention is based on dividing the data generated by the target data platform into multiple types of data, respectively obtaining the historical access frequency data, the generated timestamp data, the real-time data volume, the data period change volume and the data update frequency corresponding to each type of data, obtaining multiple monitoring indicator data, and defining them as multivariate shared data. According to the multivariate shared data, the data type division coefficients corresponding to different types of data are calculated, and the data type division coefficient threshold is obtained to perform numerical comparison on each data type division coefficient, and the different types of data are divided into a first type of shared data and a second type of shared data to obtain the shared type division data. According to the multivariate shared data, a data sharing sorting sequence corresponding to each first type of shared data is obtained, and the target data platform shares the first type of shared data to the data receiving terminal according to the data sharing sorting sequence, and divides the second type of shared data into the first terminal type shared data and the second terminal type shared data according to the number of the data receiving terminals. The target data platform synchronously shares the first terminal type shared data to multiple data receiving terminals through a star connection, obtains the receiving hierarchical data corresponding to the second terminal type shared data, and the target data platform shares the second terminal type shared data to multiple data receiving terminals according to the receiving hierarchical data.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: A multivariate big data sharing method comprises the following specific steps:

[0008] Step F1: Divide the data generated by the target data platform into multiple types of data, obtain the historical access frequency data, generation timestamp data, real-time data volume, data period change volume and data update frequency corresponding to each type of data, obtain multiple monitoring indicator data, and define them as multivariate shared data;

[0009] Step F2: Calculate the data type division coefficients corresponding to different types of data according to the multivariate shared data, obtain the data type division coefficient threshold, perform numerical comparison on each data type division coefficient, divide the different types of data into the first type of shared data and the second type of shared data, and obtain the shared type division data;

[0010] Step F3: obtaining a data sharing ranking sequence corresponding to each first type of shared data according to the multivariate shared data analysis, and the target data platform shares the first type of shared data to the data receiving terminal according to the data sharing ranking sequence;

[0011] Step F4: Divide the second type of shared data into first terminal type shared data and second terminal type shared data according to the number of data receiving terminals. The target data platform synchronously shares the first terminal type shared data to multiple data receiving terminals through a star connection, obtains the receiving layered data corresponding to the second terminal type shared data, and the target data platform shares the second terminal type shared data to multiple data receiving terminals according to the receiving layered data.

[0012] Furthermore, the step F1 further includes the following specific steps:

[0013] Step F11: Divide various types of data generated by the target data platform into several types of data, and name them as type data k1 to kk respectively;

[0014] Step F12: Perform data generation monitoring on the k1th type of data to obtain the k1th monitoring indicator data;

[0015] Step F13: Obtain the historical access frequency data, generation timestamp data, real-time data volume, data time period change volume and data update frequency corresponding to the k2 to kk types of data, and obtain the k2 to kk monitoring indicator data;

[0016] Step F14: Define the monitoring indicator data from k1 to kk as multivariate shared data.

[0017] Furthermore, the step F12 further includes the following specific steps:

[0018] In the process of monitoring the k1-th type of data, the generation time of each sub-data in the k1-th type of data is obtained to obtain multiple sub-data generation timestamps, which are named as generation timestamp data;

[0019] Acquire the historical access frequency corresponding to each sub-data in the k1-th type of data, obtain multiple sub-data historical access frequencies, and collectively name them as historical access frequency data;

[0020] Within the time range for monitoring the k1th type of data, multiple characteristic monitoring periods are marked, and the time interval and monitoring duration of each two consecutive characteristic monitoring periods are the same, and they are named the first to jth characteristic monitoring periods respectively, and the duration of the characteristic monitoring period is obtained to obtain the characteristic monitoring duration value;

[0021] Obtain the cumulative update times of the k1th type data corresponding to the first to jth characteristic monitoring periods, obtain the cumulative update times of the first to jth data, calculate the average of the cumulative update times of the first to jth data, and then calculate the ratio of the obtained average to the characteristic monitoring duration value to obtain the data update frequency;

[0022] Performing data volume detection on the k1th type of data at the first characteristic monitoring moment to obtain a first monitoring change data volume;

[0023] The details are as follows:

[0024] Mark the time value corresponding to the start time of the first characteristic monitoring period as the first monitoring time value, and mark the time value corresponding to the end time of the first characteristic monitoring period as the second monitoring time value;

[0025] Respectively obtain the data amount of the k1th type data corresponding to the first monitoring time value and the second monitoring time value, and name them as the first monitoring data amount and the second monitoring data amount respectively;

[0026] Calculate the difference between the second monitoring data volume and the first monitoring data volume, and then calculate the ratio of the obtained difference to the characteristic monitoring duration value to obtain the first monitoring change data volume;

[0027] Acquire the monitoring change data amounts corresponding to the second to j-th characteristic monitoring periods respectively, and obtain the second to j-th monitoring data change amounts;

[0028] Calculate the average value of the changes in the first to jth monitoring data to obtain the change in the data period;

[0029] Obtain the data volume corresponding to the k1th type of data in the target data platform in real time to obtain the real-time data volume;

[0030] The historical access frequency data, generation timestamp data, real-time data volume, data time period change volume and data update frequency corresponding to the k1th type of data are defined as the k1th monitoring indicator data.

[0031] Furthermore, the step F2 further includes the following specific steps:

[0032] Step F21: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0033] Step F22: acquiring the real-time data volume, data period change volume and data update frequency corresponding to the k1th type of data according to the k1th monitoring indicator data;

[0034] Step F23: Calculate the data type division coefficient corresponding to the k1th type of data by using the real-time data volume, the data time period change volume and the data update frequency;

[0035] Calculate the data type division coefficient corresponding to the k1th type of data. The specific formula configuration is as follows:

[0036] Slx=Sjl+Sbl+Sgp*a1;

[0037] Among them, Slx is the data type division coefficient corresponding to the k1th type of data, Sjl is the real-time data volume, Sbl is the data period change volume, Sgp is the data update frequency, a1 is the set proportional coefficient and a1 is greater than 0;

[0038] Step F24: obtaining data type division coefficients corresponding to the k2th to kkth type data respectively according to the k2th to kkth monitoring indicator data;

[0039] Step F25: Obtain the data type division coefficient threshold, perform numerical comparison between the data type division coefficients corresponding to the k1 to kk type data and the data type division coefficient threshold, and obtain the shared type division data.

[0040] Furthermore, the step F25 further includes the following specific steps:

[0041] Step F251: respectively obtaining a real-time data volume threshold, a data time period change volume threshold, and a data update frequency threshold;

[0042] Step F252: Calculate the data type division coefficient threshold by using the real-time data volume threshold, the data time period change threshold, and the data update frequency threshold;

[0043] Step F253: when the data type division coefficient is greater than or equal to the data type division coefficient threshold, determining that the corresponding type data is the first type shared data;

[0044] Step F254: When the data type division coefficient is less than the data type division coefficient threshold, the corresponding type data is determined to be the second type shared data.

[0045] Furthermore, the step F3 further includes the following specific steps:

[0046] Step F31: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0047] Step F32: Acquire sharing type classification data;

[0048] Step F33: when the ki-th type of data is the first type of shared data, data sharing is performed on the ki-th type of data according to the ki-th monitoring indicator data;

[0049] The step F33 further includes the following specific steps:

[0050] Step F331: naming the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data respectively;

[0051] Step F332: construct a distributed sharing framework through a distributed file management platform, and select multiple data nodes in the HDFS library in the distributed sharing framework;

[0052] Step F333: Obtain the i-th data sharing sorting sequence;

[0053] Step F334: Divide the i1th to iith sub-data into ii data blocks, and copy them to multiple data nodes respectively to obtain the i1th to iith sub-nodes, and perform node sorting on the i1th to iith sub-nodes according to the i-th data shared sorting sequence;

[0054] Step F335: Acquire the APIs corresponding to the i1th to iith sub-nodes respectively to obtain the sub-node API data;

[0055] Step F336: A Kafka group is established through a distributed sharing framework, and the target data platform publishes the sub-node API data in the Kafka group. The Kafka group pushes the sub-node API data to the data receiving terminal according to the i-th data sharing sorting sequence;

[0056] Step F337: Repeat the sharing process for the i1th sub-data, and share each sub-data in each first type shared data respectively.

[0057] Furthermore, the step F333 further includes the following specific steps:

[0058] According to the ki monitoring indicator, obtain the sub-data historical access frequency and sub-data generation timestamp corresponding to each data item in the ki type data;

[0059] Name the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data, respectively, mark the historical access frequencies of the sub-data corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th access frequencies, respectively, and mark the timestamps of the sub-data generation corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th timestamps;

[0060] Get the time value corresponding to the current moment and get the current time value;

[0061] The data sharing ranking coefficient corresponding to the i1th sub-data is obtained by calculating the i1th access frequency, the i1th timestamp and the current time value, and is named the i1th ranking coefficient;

[0062] Calculate the data sharing ranking coefficient corresponding to the i1th sub-data. The specific formula configuration is as follows:

[0063] Spx = |Sc-Sd|*Fc;

[0064] Wherein, Spx is the data sharing ranking coefficient corresponding to the i1th sub-data, Sc is the time value corresponding to the i1th timestamp, Sd is the current time value, and Fc is the i1th access frequency;

[0065] Calculate the data sharing ranking coefficients corresponding to the i2th to iith sub-data respectively according to the i2th to iith access frequencies, the i2th to iith timestamps and the current time value to obtain the i2th to iith ranking coefficients;

[0066] Arrange the i1th to iith sorting coefficients in descending order according to their numerical values ​​to obtain the ith data sharing sorting sequence.

[0067] Furthermore, the step F4 further includes the following specific steps:

[0068] Step F41: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0069] Step F42: Acquire sharing type division data;

[0070] Step F42: when the kq-th type data is the second type shared data, data sharing is performed on the ki-th type data according to the kq-th monitoring indicator data;

[0071] Step F43: Acquire the number of data receiving terminals corresponding to the kq-th type of data to obtain the value of the number of shared terminals;

[0072] Step F44: when the number of data receiving terminals is greater than p, the kq-th type of data is classified as first terminal type shared data;

[0073] Step F45: when the number of data receiving terminals is less than or equal to p, the kq-th type of data is classified as second terminal type shared data;

[0074] Step F46: when the kq-th type data is the first terminal type shared data, data sharing is performed on the kq-th type data;

[0075] The details are as follows:

[0076] Step F461: With the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and multiple branch nodes are connected in a star shape, and the main node synchronously shares multiple sub-data in the kq-th type data to multiple branch nodes in the order of the time stamps of the sub-data generation;

[0077] Step F47: when the kq-th type data is the second terminal type shared data, data sharing is performed on the kq-th type data;

[0078] The details are as follows:

[0079] Step F471: Obtain received hierarchical data;

[0080] Step F472: With the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and the multiple branch nodes are hierarchically connected according to the received hierarchical data;

[0081] Step F473: the master node synchronously shares the multiple sub-data in the kq-th type data to the branch node corresponding to the second-layer receiving end hierarchy in the order of the time when the sub-data are generated. The branch node corresponding to the second-layer receiving end hierarchy synchronously shares the multiple sub-data in the kq-th type data to the branch node corresponding to the third-layer receiving end hierarchy in the order of the time when the sub-data are generated. Repeat this sharing process until the multiple sub-data in the kq-th type data are shared to the branch node corresponding to the e+1-th layer receiving end hierarchy.

[0082] Step F48: Repeat the sharing process for the kq-th type data, and share each sub-data in each second type shared data respectively.

[0083] Furthermore, the step F471 further includes the following specific steps:

[0084] Name the multiple data receiving terminals as first and z receiving terminals respectively;

[0085] Performing terminal monitoring on the first receiving terminal to obtain a terminal monitoring coefficient corresponding to the first receiving terminal;

[0086] The details are as follows:

[0087] The unit time length before the current moment is taken as the terminal monitoring cycle;

[0088] Obtaining the total number of accesses to the kq-th type of data by the first receiving terminal within a terminal monitoring period, to obtain a first total number of accesses;

[0089] Acquire the number of periodic online users of the first receiving terminal in the terminal monitoring period to obtain the first number of online users;

[0090] Acquire a data receiving delay of a first receiving terminal to obtain a first receiving delay;

[0091] The details are as follows:

[0092] Sending test data to the first receiving terminal through the upper level sending terminal of the first receiving terminal, and acquiring the time value corresponding to the sending time to obtain a first time value;

[0093] The first receiving terminal receives the test data, and acquires the time value corresponding to the receiving moment to obtain a second time value;

[0094] Calculate the difference between the second time value and the first time value to obtain a first receiving delay;

[0095] The terminal monitoring coefficient corresponding to the first receiving terminal is obtained by calculating the first total number of visits, the first number of online users, and the first receiving delay;

[0096] The terminal monitoring coefficient corresponding to the first receiving terminal is calculated, and the specific formula is as follows:

[0097] Zjx1=(Fws1×Sy1)+Yhs1;

[0098] Wherein Zjx1 is the terminal monitoring coefficient corresponding to the first receiving terminal, Fws1 is the first total number of accesses, Sy1 is the first receiving delay, and Yhs1 is the first number of online users;

[0099] Acquire the terminal monitoring coefficient corresponding to each receiving terminal respectively;

[0100] Acquire multiple terminal monitoring coefficient thresholds respectively, and arrange them in descending order as the first to e-th terminal monitoring coefficient thresholds, and numerically compare the terminal monitoring coefficient corresponding to each receiving terminal with the first to e-th terminal monitoring coefficient thresholds respectively, to obtain the first to e+1th layer receiving terminal classification, and name it as receiving layered data.

[0101] A multivariate big data sharing system, comprising:

[0102] Data acquisition module: used to obtain multiple monitoring indicator data and obtain multi-dimensional shared data;

[0103] Data analysis module: used to analyze multi-type shared data, divide multiple types of data into first type shared data and second type shared data, and obtain shared type divided data;

[0104] The first sharing module is used to share the first type of shared data according to the multi-shared data and the shared type classification data.

[0105] The second sharing module is used to perform big data sharing of the second type of shared data according to the multi-shared data and the shared type division data.

[0106] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0107] 1. The present invention divides the data to be shared into first-type shared data and second-type shared data by obtaining the data type division coefficient, and obtains the data sharing sorting sequence corresponding to the first-type shared data for distributed data sharing, which can effectively ensure data sharing efficiency and reduce the risk of data loss;

[0108] 2. The present invention adopts different sharing modes for the second type of shared data according to the number of data receiving terminals, which can effectively improve the pertinence of data sharing and the efficiency of using shared resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0110] Figure 1 It is a diagram of the implementation steps of the present invention;

[0111] Figure 2 is the overall system block diagram of the present invention;

[0112] Figure 3 It is a star connection schematic diagram of the present invention;

[0113] Figure 4 It is a schematic diagram of the hierarchical connection of the present invention. DETAILED DESCRIPTION

[0114] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0115] Embodiment 1

[0116] See also Figure 2 , the present invention provides a technical solution: a multivariate big data sharing system, comprising a data acquisition module, a data analysis module, a first sharing module, a second sharing module and a server, wherein the data acquisition module, the data analysis module, the first sharing module and the second sharing module are respectively connected to the server, and the server controls the data acquisition module, the data analysis module, the first sharing module and the second sharing module respectively;

[0117] The data acquisition module acquires multiple shared data;

[0118] Divide various types of data generated by the target data platform into several types of data, and name them as type data from k1 to kk respectively;

[0119] It should be noted here that:

[0120] In the present invention, k mentioned here specifically refers to the quantity value corresponding to the type data, and k is an integer greater than 0;

[0121] In the present invention, the type data needs to be specifically determined according to the specific data platform type;

[0122] For example, if the target data platform is an online shopping platform, the k1th type of data may be product price data, the k2th type of data may be product descriptions, and the k3th type of data may be sales records corresponding to the product;

[0123] Perform data generation monitoring on the k1th type of data to obtain the k1th monitoring indicator data;

[0124] In the process of monitoring the k1-th type of data, the generation time of each sub-data in the k1-th type of data is obtained to obtain multiple sub-data generation timestamps, which are named as generation timestamp data;

[0125] Acquire the historical access frequency corresponding to each sub-data in the k1-th type of data, obtain multiple sub-data historical access frequencies, and collectively name them as historical access frequency data;

[0126] It should be noted here that:

[0127] In the present invention, the various data involved include existing versions and historical versions, and the historical access frequencies here are the access frequencies corresponding to the historical versions of various data;

[0128] For example, suppose that a certain data in the k1th type of data is the price of a commodity. Since the price of a commodity fluctuates, the historical price is the historical version of this data, and the current price is the current version of this data.

[0129] Within the time range for monitoring the k1th type of data, multiple characteristic monitoring periods are marked, and the time interval and monitoring duration of each two consecutive characteristic monitoring periods are the same, and they are named the first to jth characteristic monitoring periods respectively;

[0130] Obtain the duration of the characteristic monitoring period to obtain the characteristic monitoring duration value;

[0131] Obtain the cumulative update times of the k1th type data corresponding to the first to jth characteristic monitoring periods, obtain the cumulative update times of the first to jth data, calculate the average of the cumulative update times of the first to jth data, and then calculate the ratio of the obtained average to the characteristic monitoring duration value to obtain the data update frequency;

[0132] It should be noted here that:

[0133] In the present invention, j is the quantity value corresponding to the characteristic monitoring period, and j is an integer greater than 0;

[0134] Performing data volume detection on the k1th type of data at the first characteristic monitoring moment to obtain a first monitoring change data volume;

[0135] The details are as follows:

[0136] Mark the time value corresponding to the start time of the first characteristic monitoring period as the first monitoring time value, and mark the time value corresponding to the end time of the first characteristic monitoring period as the second monitoring time value;

[0137] Respectively obtain the data amount of the k1th type data corresponding to the first monitoring time value and the second monitoring time value, and name them as the first monitoring data amount and the second monitoring data amount respectively;

[0138] Calculate the difference between the second monitoring data volume and the first monitoring data volume, and then calculate the ratio of the obtained difference to the characteristic monitoring duration value to obtain the first monitoring change data volume;

[0139] Acquire the monitoring change data amounts corresponding to the second to j-th characteristic monitoring periods respectively, and obtain the second to j-th monitoring data change amounts;

[0140] Calculate the average value of the changes in the first to jth monitoring data to obtain the change in the data period;

[0141] Obtain the data volume corresponding to the k1th type of data in the target data platform in real time to obtain the real-time data volume;

[0142] It should be noted here that:

[0143] In the present invention, the amount of data involved specifically refers to the size or capacity of the data contained in the storage device or transmission pipeline;

[0144] The historical access frequency data, generation timestamp data, real-time data volume, data period change volume and data update frequency corresponding to the k1th type of data are defined as the k1th monitoring indicator data;

[0145] Obtain the historical access frequency data, generation timestamp data, real-time data volume, data period change volume, and data update frequency corresponding to the k2 to kk types of data, and obtain the k2 to kk monitoring indicator data;

[0146] The monitoring indicator data from k1 to kk are defined as multivariate shared data;

[0147] The data acquisition module acquires the multi-shared data and transmits it to the data analysis module, the first sharing module and the second sharing module;

[0148] The data analysis module analyzes the multivariate shared data to obtain the shared type classification data;

[0149] Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0150] According to the k1th monitoring indicator data, obtain the real-time data volume, data period change volume and data update frequency corresponding to the k1th type of data;

[0151] The data type division coefficient corresponding to the k1th type of data is obtained by calculating the real-time data volume, the data period change volume and the data update frequency;

[0152] Calculate the data type division coefficient corresponding to the k1th type of data. The specific formula configuration is as follows:

[0153] Slx=Sjl+Sbl+Sgp*a1;

[0154] Among them, Slx is the data type division coefficient corresponding to the k1th type of data, Sjl is the real-time data volume, Sbl is the data period change volume, Sgp is the data update frequency, a1 is the set proportional coefficient and a1 is greater than 0;

[0155] Here are some things to note:

[0156] The a1 involved here adjusts the specific value of the data update frequency to a value range that matches the real-time data volume and the data period change by proportional adjustment. In practical applications, it is necessary to make corresponding settings according to the size relationship and weight relationship between different values;

[0157] Obtain data type division coefficients corresponding to k2 to kk type data respectively according to k2 to kk monitoring indicator data;

[0158] Obtain a data type division coefficient threshold, compare the data type division coefficients corresponding to the k1-kk type data with the data type division coefficient threshold, and obtain shared type division data;

[0159] The details are as follows:

[0160] Obtain the real-time data volume threshold, data period change threshold, and data update frequency threshold respectively;

[0161] The data type division coefficient threshold is obtained by calculating the real-time data volume threshold, the data period change threshold and the data update frequency threshold;

[0162] Calculate the data type division coefficient threshold. The specific formula configuration is as follows:

[0163] Slxy=Sjly+Sbly+Sgpy*a1;

[0164] Among them, Slxy is the data type classification coefficient threshold, Sjly is the real-time data volume threshold, Sbly is the data period change threshold, Sgpy is the data update frequency threshold, a1 is the set proportional coefficient and a1 is greater than 0;

[0165] The numerical comparison process is as follows:

[0166] When the data type division coefficient is greater than or equal to the data type division coefficient threshold, determining that the corresponding type data is the first type shared data;

[0167] When the data type division coefficient is less than the data type division coefficient threshold, determining that the corresponding type data is the second type shared data;

[0168] It should be noted here that:

[0169] In the present invention, the data update quantity, real-time data volume and data generation speed corresponding to the first type of shared data are greater than the data update quantity, real-time data volume and data generation speed corresponding to the second type of shared data;

[0170] The data analysis module obtains the sharing type classification data and transmits it to the first sharing module and the second sharing module;

[0171] The first sharing module performs big data sharing of the first type of shared data according to the multi-shared data and the shared type division data;

[0172] The first sharing module includes a target data platform and a plurality of data receiving terminals;

[0173] Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0174] Get the shared type classification data;

[0175] When the ki-th type of data is the first type of shared data, the ki-th type of data is shared according to the ki-th monitoring indicator data;

[0176] The details are as follows:

[0177] Name the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data respectively;

[0178] A distributed sharing framework is built through a distributed file management platform, and multiple data nodes are selected from the HDFS library in the distributed sharing framework;

[0179] Divide the i1th to iith sub-data into ii data blocks, and copy them to multiple data nodes respectively to obtain i1th to iith sub-nodes, and sort the i1th to iith sub-nodes according to the i-th data shared sorting sequence;

[0180] It should be noted here that:

[0181] In the present invention, ii is the quantity value corresponding to the sub-data in the ki-th type of data, and ii is an integer greater than 0;

[0182] Acquire the i-th data sharing sorting sequence;

[0183] The details are as follows:

[0184] According to the ki monitoring indicator, obtain the sub-data historical access frequency and sub-data generation timestamp corresponding to each data item in the ki type data;

[0185] Name the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data, respectively, mark the historical access frequencies of the sub-data corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th access frequencies, respectively, and mark the timestamps of the sub-data generation corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th timestamps;

[0186] Get the time value corresponding to the current moment and get the current time value;

[0187] The data sharing ranking coefficient corresponding to the i1th sub-data is obtained by calculating the i1th access frequency, the i1th timestamp and the current time value, and is named the i1th ranking coefficient;

[0188] Calculate the data sharing ranking coefficient corresponding to the i1th sub-data. The specific formula configuration is as follows:

[0189] Spx = |Sc-Sd|*Fc;

[0190] Wherein, Spx is the data sharing ranking coefficient corresponding to the i1th sub-data, Sc is the time value corresponding to the i1th timestamp, Sd is the current time value, and Fc is the i1th access frequency;

[0191] Calculate the data sharing ranking coefficients corresponding to the i2th to iith sub-data respectively according to the i2th to iith access frequencies, the i2th to iith timestamps and the current time value to obtain the i2th to iith ranking coefficients;

[0192] Arrange the i1th to iith sorting coefficients in descending order according to their numerical values ​​to obtain the ith data sharing sorting sequence;

[0193] The HDFS tool obtains the APIs corresponding to the child nodes i1 to ii, and obtains the child node API data;

[0194] A Kafka group is established through a distributed sharing framework, the target data platform is set as the "publisher", and multiple data receiving terminals are set as "consumers". The target data platform publishes the sub-node API data in the Kafka group, and the Kafka group pushes the sub-node API data to the data receiving terminal according to the i-th data sharing sorting sequence;

[0195] Repeat the sharing process for the i1th sub-data, and share each sub-data in each first type of shared data respectively;

[0196] It should be noted here that:

[0197] Kafka provides a "publish-subscribe" model, where producers can publish messages to topics, and consumers can subscribe to messages from these topics. In this invention, the target data platform is the "publisher" and multiple data receiving terminals are set as "consumers";

[0198] In the present invention, the storage space of each data block needs to be larger than the space for storing the corresponding sub-data;

[0199] For example: if the i1th sub-data is stored in the i1th data block, and the storage space of the i1th data block needs to be larger than the data volume of the i1th sub-data;

[0200] The second sharing module performs big data sharing of the second type of shared data according to the multi-shared data and the shared type division data;

[0201] The second data contribution module includes a target data platform and a plurality of data receiving terminals;

[0202] Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0203] Get the shared type classification data;

[0204] When the kq-th type of data is the second type of shared data, the ki-th type of data is shared according to the kq-th monitoring indicator data;

[0205] Acquire the number of data receiving terminals corresponding to the kq-th type of data to obtain the value of the number of shared terminals;

[0206] When the value of the number of data sharing terminals is greater than p, the kq-th type of data is classified as the first terminal type shared data;

[0207] When the number of data sharing terminals is less than or equal to p, the kq-th type of data is classified as the second terminal type shared data;

[0208] It should be noted here that:

[0209] In this application, p is a division index corresponding to the number of data sharing terminals. In practical applications, the specific data of p needs to be divided according to the specific type of transmitted data;

[0210] When the kq-th type data is the first terminal type shared data;

[0211] See also Figure 3 , with the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and multiple branch nodes are connected in a star shape, and the main node synchronously shares multiple sub-data in the kq-th type data to multiple branch nodes in the order of the time stamps of the sub-data;

[0212] When the kq-th type data is the second terminal type shared data;

[0213] See also Figure 4 , with the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and the multiple branch nodes are hierarchically connected according to the received hierarchical data, the main node synchronously shares the multiple sub-data in the kq-th type data to the branch nodes corresponding to the second-layer receiving terminal hierarchy in the order of the time when the sub-data are generated. The branch nodes corresponding to the second-layer receiving terminal hierarchy synchronously share the multiple sub-data in the kq-th type data to the branch nodes corresponding to the third-layer receiving terminal hierarchy in the order of the time when the sub-data are generated. Repeat this sharing process until the multiple sub-data in the kq-th type data are shared to the branch nodes corresponding to the e+1-th layer receiving terminal hierarchy;

[0214] Acquiring received hierarchical data;

[0215] The details are as follows:

[0216] Name the multiple data receiving terminals as first and z receiving terminals respectively;

[0217] It should be noted here that:

[0218] In this application, z is the number value corresponding to the receiving terminal, and z is an integer greater than 0;

[0219] Performing terminal monitoring on the first receiving terminal to obtain a terminal monitoring coefficient corresponding to the first receiving terminal;

[0220] The details are as follows:

[0221] The unit time length before the current moment is taken as the terminal monitoring cycle;

[0222] It should be noted here that:

[0223] In this application, the unit time length is specifically 10 seconds;

[0224] Obtaining the total number of accesses to the kq-th type of data by the first receiving terminal within a terminal monitoring period, to obtain a first total number of accesses;

[0225] Acquire the number of periodic online users of the first receiving terminal in the terminal monitoring period to obtain the first number of online users;

[0226] Acquire a data receiving delay of a first receiving terminal to obtain a first receiving delay;

[0227] The details are as follows:

[0228] Sending test data to the first receiving terminal through the upper level sending terminal of the first receiving terminal, and acquiring the time value corresponding to the sending time to obtain a first time value;

[0229] The first receiving terminal receives the test data, and acquires the time value corresponding to the receiving moment to obtain a second time value;

[0230] Calculate the difference between the second time value and the first time value to obtain a first receiving delay;

[0231] The terminal monitoring coefficient corresponding to the first receiving terminal is obtained by calculating the first total number of visits, the first number of online users, and the first receiving delay;

[0232] The terminal monitoring coefficient corresponding to the first receiving terminal is calculated, and the specific formula is as follows:

[0233] Zjx1=(Fws1×Sy1)+Yhs1;

[0234] Wherein Zjx1 is the terminal monitoring coefficient corresponding to the first receiving terminal, Fws1 is the first total number of accesses, Sy1 is the first receiving delay, and Yhs1 is the first number of online users;

[0235] Acquire the terminal monitoring coefficient corresponding to each receiving terminal respectively;

[0236] Acquire multiple terminal monitoring coefficient thresholds respectively, and arrange them in descending order as first to e-th terminal monitoring coefficient thresholds, compare the terminal monitoring coefficient corresponding to each receiving terminal with the first to e-th terminal monitoring coefficient thresholds respectively, obtain the first to e+1th layer receiving terminal classification, and name it as receiving layered data;

[0237] It should be noted here that:

[0238] In this application, e is a numerical value corresponding to the terminal monitoring coefficient threshold, and e is an integer greater than 0;

[0239] The first terminal monitoring coefficient threshold is greater than the second terminal monitoring coefficient threshold, the second terminal monitoring coefficient threshold is greater than the third terminal monitoring coefficient threshold, and so on, the e-1th terminal monitoring coefficient threshold is greater than the eth terminal monitoring coefficient threshold;

[0240] For example:

[0241] If the terminal monitoring coefficient corresponding to the f-th receiving end is greater than the second terminal monitoring coefficient threshold and less than or equal to the first terminal monitoring coefficient threshold, then the f-th receiving end is classified as a second-layer receiving end;

[0242] In this application, the number of terminal monitoring coefficient thresholds needs to be specifically determined according to the number of receiving terminal classifications, and the specific value of each specific terminal monitoring coefficient threshold needs to be determined accordingly according to the data type and data volume;

[0243] Repeat the sharing process for the kq-th type of data, and share each sub-data in each second type of shared data respectively.

[0244] In this application, if corresponding calculation formulas appear, the above calculation formulas are all dimensionless and take their numerical calculations. The weight coefficients, proportional coefficients and other coefficients in the formulas are set to a result value obtained by quantifying each parameter. The size of the weight coefficient and the proportional coefficient can be determined as long as it does not affect the proportional relationship between the parameter and the result value.

[0245] Embodiment 2

[0246] See also Figure 1 Based on another concept of the same invention, a multivariate big data sharing method is now proposed, comprising the following steps:

[0247] Step F1: Obtain multi-shared data;

[0248] Step F11: Divide various types of data generated by the target data platform into several types of data, and name them as type data k1 to kk respectively;

[0249] Step F12: Perform data generation monitoring on the k1th type of data to obtain the k1th monitoring indicator data;

[0250] The step F12 further includes the following specific steps:

[0251] Step F121: in the process of monitoring the k1-th type of data, the generation time of each sub-data in the k1-th type of data is acquired to obtain multiple sub-data generation timestamps, which are named as generation timestamp data;

[0252] Step F122: acquiring the historical access frequency corresponding to each sub-data item in the k1-th type of data, obtaining multiple sub-data historical access frequencies, and collectively naming them as historical access frequency data;

[0253] Step F123: within the time range for monitoring the k1-th type of data, mark multiple characteristic monitoring time periods, and the time interval and monitoring duration of each two consecutive characteristic monitoring time periods are the same, and name them as the first to j-th characteristic monitoring time periods, respectively, obtain the duration of the characteristic monitoring time periods, and obtain the characteristic monitoring duration values;

[0254] Step F124: Obtain the cumulative update times of the k1th type data corresponding to the first to jth characteristic monitoring periods, obtain the cumulative update times of the first to jth data, calculate the average of the cumulative update times of the first to jth data, and then calculate the ratio of the obtained average to the characteristic monitoring duration value to obtain the data update frequency;

[0255] Step F125: performing data volume detection on the k1th type of data at the first characteristic monitoring moment to obtain a first monitoring change data volume;

[0256] The details are as follows:

[0257] Step F1251: marking the time value corresponding to the start time of the first characteristic monitoring period as the first monitoring time value, and marking the time value corresponding to the end time of the first characteristic monitoring period as the second monitoring time value;

[0258] Step F1252: respectively obtaining the data amount of the k1-th type data corresponding to the first monitoring time value and the second monitoring time value, and naming them as the first monitoring data amount and the second monitoring data amount respectively;

[0259] Step F1253: Calculate the difference between the second monitoring data volume and the first monitoring data volume, and then calculate the ratio of the obtained difference to the characteristic monitoring duration value to obtain the first monitoring change data volume;

[0260] Step F126: acquiring the monitoring change data amounts corresponding to the second to j-th characteristic monitoring periods respectively, and obtaining the second to j-th monitoring data change amounts;

[0261] Step F127: Calculate the average value of the changes in the first to jth monitoring data to obtain the change in the data period;

[0262] Step F128: acquiring the data volume corresponding to the k1th type of data in the target data platform in real time to obtain the real-time data volume;

[0263] Step F129: defining the historical access frequency data, generation timestamp data, real-time data volume, data time period change volume and data update frequency corresponding to the k1th type of data as the k1th monitoring indicator data;

[0264] Step F13: Obtain the historical access frequency data, generation timestamp data, real-time data volume, data time period change volume and data update frequency corresponding to the k2 to kk types of data, and obtain the k2 to kk monitoring indicator data;

[0265] Step F14: define the monitoring indicator data from k1 to kk as multivariate shared data;

[0266] Step F2: Analyze the multivariate sharing data to obtain sharing type classification data;

[0267] Step F21: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0268] Step F22: acquiring the real-time data volume, data period change volume and data update frequency corresponding to the k1th type of data according to the k1th monitoring indicator data;

[0269] Step F23: Calculate the data type division coefficient corresponding to the k1th type of data by using the real-time data volume, the data time period change volume and the data update frequency;

[0270] Calculate the data type division coefficient corresponding to the k1th type of data. The specific formula configuration is as follows:

[0271] Slx=Sjl+Sbl+Sgp*a1;

[0272] Among them, Slx is the data type division coefficient corresponding to the k1th type of data, Sjl is the real-time data volume, Sbl is the data period change volume, Sgp is the data update frequency, a1 is the set proportional coefficient and a1 is greater than 0;

[0273] Step F24: obtaining data type division coefficients corresponding to the k2th to kkth type data respectively according to the k2th to kkth monitoring indicator data;

[0274] Step F25: obtaining a data type division coefficient threshold, comparing the data type division coefficients corresponding to the k1-kk type data respectively with the data type division coefficient threshold, and obtaining shared type division data;

[0275] The details are as follows:

[0276] Step F251: respectively obtaining a real-time data volume threshold, a data time period change volume threshold, and a data update frequency threshold;

[0277] Step F252: Calculate the data type division coefficient threshold by using the real-time data volume threshold, the data time period change threshold, and the data update frequency threshold;

[0278] Step F253: when the data type division coefficient is greater than or equal to the data type division coefficient threshold, determining that the corresponding type data is the first type shared data;

[0279] Step F254: when the data type division coefficient is less than the data type division coefficient threshold, determining that the corresponding type data is the second type shared data;

[0280] Step F3: performing big data sharing on the first type of shared data according to the multi-shared data and the shared type division data;

[0281] The first sharing module includes a target data platform and a plurality of data receiving terminals;

[0282] Step F31: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0283] Step F32: Acquire sharing type classification data;

[0284] Step F33: when the ki-th type of data is the first type of shared data, data sharing is performed on the ki-th type of data according to the ki-th monitoring indicator data;

[0285] The step F33 further includes the following specific steps:

[0286] Step F331: naming the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data respectively;

[0287] Step F332: construct a distributed sharing framework through a distributed file management platform, and select multiple data nodes in the HDFS library in the distributed sharing framework;

[0288] Step F333: Obtain the i-th data sharing sorting sequence;

[0289] Step F334: Divide the i1th to iith sub-data into ii data blocks, and copy them to multiple data nodes respectively to obtain the i1th to iith sub-nodes, and perform node sorting on the i1th to iith sub-nodes according to the i-th data shared sorting sequence;

[0290] Step F335: Acquire the APIs corresponding to the i1th to iith sub-nodes respectively to obtain the sub-node API data;

[0291] Step F336: A Kafka group is established through a distributed sharing framework, and the target data platform publishes the sub-node API data in the Kafka group. The Kafka group pushes the sub-node API data to the data receiving terminal according to the i-th data sharing sorting sequence;

[0292] Step F337: repeating the sharing process for the i1th sub-data, and sharing each sub-data in each first type of shared data respectively;

[0293] The step F333 further includes the following specific steps:

[0294] Step F3331: Obtain the sub-data historical access frequency and sub-data generation timestamp corresponding to each item of data in the ki-th type of data according to the ki-th monitoring indicator;

[0295] Step F3332: Name the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data, respectively, mark the historical access frequencies of the sub-data corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th access frequencies, respectively, and mark the generation timestamps of the sub-data corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th timestamps;

[0296] Step F3333: Obtain the time value corresponding to the current moment to obtain the current time value;

[0297] Step F3334: Calculate the data sharing ranking coefficient corresponding to the i1th sub-data by using the i1th access frequency, the i1th timestamp and the current time value, and name it the i1th ranking coefficient;

[0298] Calculate the data sharing ranking coefficient corresponding to the i1th sub-data. The specific formula configuration is as follows:

[0299] Spx = |Sc-Sd|*Fc;

[0300] Wherein, Spx is the data sharing ranking coefficient corresponding to the i1th sub-data, Sc is the time value corresponding to the i1th timestamp, Sd is the current time value, and Fc is the i1th access frequency;

[0301] Step F3335: Calculate the data sharing ranking coefficients corresponding to the i2th to iith sub-data respectively according to the i2th to iith access frequencies, the i2th to iith timestamps and the current time value to obtain the i2th to iith ranking coefficients;

[0302] Step F3336: Arrange the i1th to iith sorting coefficients in descending order according to their numerical values ​​to obtain the ith data sharing sorting sequence;

[0303] Step F4: performing big data sharing on the second type of shared data according to the multi-shared data and the shared type division data;

[0304] Step F41: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data;

[0305] Step F42: Acquire sharing type division data;

[0306] Step F42: when the kq-th type data is the second type shared data, data sharing is performed on the ki-th type data according to the kq-th monitoring indicator data;

[0307] Step F43: Acquire the number of data receiving terminals corresponding to the kq-th type of data to obtain the value of the number of shared terminals;

[0308] Step F44: when the value of the number of data sharing terminals is greater than p, the kq-th type of data is classified as first terminal type shared data;

[0309] Step F45: when the value of the number of data sharing terminals is less than or equal to p, the kq-th type of data is classified as the second terminal type shared data;

[0310] Step F46: when the kq-th type data is the first terminal type shared data, data sharing is performed on the kq-th type data;

[0311] The details are as follows:

[0312] Step F461: With the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and multiple branch nodes are connected in a star shape, and the main node synchronously shares multiple sub-data in the kq-th type data to multiple branch nodes in the order of the time stamps of the sub-data generation;

[0313] Step F47: when the kq-th type data is the second terminal type shared data, data sharing is performed on the kq-th type data;

[0314] The details are as follows:

[0315] Step F471: Obtain received hierarchical data;

[0316] Step F472: With the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and the multiple branch nodes are hierarchically connected according to the received hierarchical data;

[0317] Step F473: the master node synchronously shares the multiple sub-data in the kq-th type data to the branch node corresponding to the second-layer receiving end hierarchy in the order of the time when the sub-data are generated. The branch node corresponding to the second-layer receiving end hierarchy synchronously shares the multiple sub-data in the kq-th type data to the branch node corresponding to the third-layer receiving end hierarchy in the order of the time when the sub-data are generated. Repeat this sharing process until the multiple sub-data in the kq-th type data are shared to the branch node corresponding to the e+1-th layer receiving end hierarchy.

[0318] The step F471 further includes the following specific steps:

[0319] Step F4711: naming multiple data receiving terminals as first and zth receiving terminals respectively;

[0320] Step F4712: performing terminal monitoring on the first receiving terminal to obtain a terminal monitoring coefficient corresponding to the first receiving terminal;

[0321] The details are as follows:

[0322] Step F47121: taking the unit time length before the current moment as the terminal monitoring cycle;

[0323] Step F47122: Obtain the total number of accesses to the kq-th type of data by the first receiving terminal within the terminal monitoring period, and obtain a first total number of accesses;

[0324] Step F47123: Acquire the number of periodic online users of the first receiving terminal in the terminal monitoring period to obtain the number of first online users;

[0325] Step F47124: Acquire the data receiving delay of the first receiving terminal to obtain a first receiving delay;

[0326] The details are as follows:

[0327] Step F471241: Send test data to the first receiving terminal through the upper level sending terminal of the first receiving terminal, and obtain the time value corresponding to the sending time to obtain a first time value;

[0328] Step F471242: The first receiving terminal receives the test data, and acquires the time value corresponding to the receiving moment to obtain a second time value;

[0329] Step F471243: Calculate the difference between the second time value and the first time value to obtain a first receiving delay;

[0330] Step F47125: obtaining a terminal monitoring coefficient corresponding to the first receiving terminal by calculating the first total number of visits, the first number of online users, and the first receiving delay;

[0331] The terminal monitoring coefficient corresponding to the first receiving terminal is calculated, and the specific formula is as follows:

[0332] Zjx1=(Fws1×Sy1)+Yhs1;

[0333] Wherein Zjx1 is the terminal monitoring coefficient corresponding to the first receiving terminal, Fws1 is the first total number of accesses, Sy1 is the first receiving delay, and Yhs1 is the first number of online users;

[0334] Step F47126: Acquire the terminal monitoring coefficient corresponding to each receiving terminal respectively;

[0335] Step F47127: Acquire multiple terminal monitoring coefficient thresholds respectively, and arrange them in descending order as first to e-th terminal monitoring coefficient thresholds, compare the terminal monitoring coefficient corresponding to each receiving terminal with the first to e-th terminal monitoring coefficient thresholds respectively, obtain the first to e+1th layer receiving terminal classification, and name it as receiving layered data;

[0336] Step F48: Repeat the sharing process for the kq-th type data, and share each sub-data in each second type shared data respectively.

[0337] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multivariate big data sharing method, characterized in that: include: Step F1: Divide the data generated by the target data platform into multiple types of data, obtain the historical access frequency data, generation timestamp data, real-time data volume, data period change volume and data update frequency corresponding to each type of data, obtain multiple monitoring indicator data, and define them as multivariate shared data; Step F2: Calculate the data type division coefficients corresponding to different types of data according to the multivariate shared data, obtain the data type division coefficient threshold, perform numerical comparison on each data type division coefficient, divide the different types of data into the first type of shared data and the second type of shared data, and obtain the shared type division data; Step F3: obtaining a data sharing ranking sequence corresponding to each first type of shared data according to the multivariate shared data analysis, and the target data platform shares the first type of shared data to the data receiving terminal according to the data sharing ranking sequence; Step F4: Divide the second type of shared data into first terminal type shared data and second terminal type shared data according to the number of data receiving terminals. The target data platform synchronously shares the first terminal type shared data to multiple data receiving terminals through a star connection, obtains the receiving layered data corresponding to the second terminal type shared data, and the target data platform shares the second terminal type shared data to multiple data receiving terminals according to the receiving layered data.

2. A multivariate big data sharing method according to claim 1, characterized in that: The step F1 further includes the following specific steps: Step F11: Divide various types of data generated by the target data platform into several types of data, and name them as type data k1 to kk respectively; Step F12: Perform data generation monitoring on the k1th type of data to obtain the k1th monitoring indicator data; Step F13: Obtain the historical access frequency data, generation timestamp data, real-time data volume, data time period change volume and data update frequency corresponding to the k2 to kk types of data, and obtain the k2 to kk monitoring indicator data; Step F14: Define the monitoring indicator data from k1 to kk as multivariate shared data.

3. A multivariate big data sharing method according to claim 2, characterized in that: The step F12 further includes the following specific steps: In the process of monitoring the k1-th type of data, the generation time of each sub-data in the k1-th type of data is obtained to obtain multiple sub-data generation timestamps, which are named as generation timestamp data; Acquire the historical access frequency corresponding to each sub-data in the k1-th type of data to obtain multiple sub-data historical access frequencies, and name them as historical access frequency data; Within the time range for monitoring the k1th type of data, multiple characteristic monitoring periods are marked, and the time interval and monitoring duration of each two consecutive characteristic monitoring periods are the same, and they are named the first to jth characteristic monitoring periods respectively, and the duration of the characteristic monitoring period is obtained to obtain the characteristic monitoring duration value; Obtain the cumulative update times of the k1th type data corresponding to the first to jth characteristic monitoring periods, obtain the cumulative update times of the first to jth data, calculate the average of the cumulative update times of the first to jth data, and then calculate the ratio of the obtained average to the characteristic monitoring duration value to obtain the data update frequency; Performing data volume detection on the k1th type of data at the first characteristic monitoring moment to obtain a first monitoring change data volume; The details are as follows: Mark the time value corresponding to the start time of the first characteristic monitoring period as the first monitoring time value, and mark the time value corresponding to the end time of the first characteristic monitoring period as the second monitoring time value; Respectively obtain the data amount of the k1th type data corresponding to the first monitoring time value and the second monitoring time value, and name them as the first monitoring data amount and the second monitoring data amount respectively; Calculate the difference between the second monitoring data volume and the first monitoring data volume, and then calculate the ratio of the obtained difference to the characteristic monitoring duration value to obtain the first monitoring change data volume; Acquire the monitoring change data amounts corresponding to the second to j-th characteristic monitoring periods respectively, and obtain the second to j-th monitoring data change amounts; Calculate the average value of the changes in the first to jth monitoring data to obtain the change in the data period; Obtain the data volume corresponding to the k1th type of data in the target data platform in real time to obtain the real-time data volume; The historical access frequency data, generation timestamp data, real-time data volume, data time period change volume and data update frequency corresponding to the k1th type of data are defined as the k1th monitoring indicator data.

4. A multivariate big data sharing method according to claim 1, characterized in that: The step F2 further includes the following specific steps: Step F21: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data; Step F22: acquiring the real-time data volume, data period change volume and data update frequency corresponding to the k1th type of data according to the k1th monitoring indicator data; Step F23: Calculate the data type division coefficient corresponding to the k1th type of data by using the real-time data volume, the data time period change volume and the data update frequency; Calculate the data type division coefficient corresponding to the k1th type of data. The specific formula configuration is as follows: Slx=Sjl+Sbl+Sgp*a1; Among them, Slx is the data type division coefficient corresponding to the k1th type of data, Sjl is the real-time data volume, Sbl is the data period change volume, Sgp is the data update frequency, a1 is the set proportional coefficient and a1 is greater than 0; Step F24: obtaining data type division coefficients corresponding to k2-th to kk-th type data according to k2-th to kk-th monitoring indicator data respectively; Step F25: Obtain the data type division coefficient threshold, perform numerical comparison between the data type division coefficients corresponding to the k1th to kkth type data and the data type division coefficient threshold, and obtain the shared type division data.

5. A multivariate big data sharing method according to claim 4, characterized in that: The step F25 further includes the following specific steps: Step F251: obtaining a real-time data volume threshold, a data time period change volume threshold, and a data update frequency threshold; Step F252: Calculate the data type division coefficient threshold by using the real-time data volume threshold, the data time period change threshold, and the data update frequency threshold; Step F253: when the data type division coefficient is greater than or equal to the data type division coefficient threshold, determining that the corresponding type data is the first type shared data; Step F254: When the data type division coefficient is less than the data type division coefficient threshold, the corresponding type data is determined to be the second type shared data.

6. A multivariate big data sharing method according to claim 1, characterized in that: The step F3 further includes the following specific steps: Step F31: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data; Step F32: Acquire sharing type classification data; Step F33: when the ki-th type of data is the first type of shared data, data sharing is performed on the ki-th type of data according to the ki-th monitoring indicator data; The step F33 further includes the following specific steps: Step F331: naming the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data respectively; Step F332: construct a distributed sharing framework through a distributed file management platform, and select multiple data nodes in the HDFS library in the distributed sharing framework; Step F333: Obtain the i-th data sharing sorting sequence; Step F334: Divide the i1th to iith sub-data into ii data blocks, copy them to multiple data nodes respectively, obtain the i1th to iith sub-nodes, and sort the i1th to iith sub-nodes according to the i-th data shared sorting sequence; Step F335: Acquire the APIs corresponding to the i1th to iith child nodes respectively to obtain the child node API data; Step F336: A Kafka group is established through a distributed sharing framework, and the target data platform publishes the sub-node API data in the Kafka group. The Kafka group pushes the sub-node API data to the data receiving terminal according to the i-th data sharing sorting sequence; Step F337: Share each sub-data in each first type shared data respectively.

7. A multivariate big data sharing method according to claim 6, characterized in that: The step F333 further includes the following specific steps: According to the ki monitoring indicator, obtain the sub-data historical access frequency and sub-data generation timestamp corresponding to each data item in the ki type data; Name the multiple sub-data in the ki-th type of data as the i1-th to ii-th sub-data, respectively, mark the historical access frequencies of the sub-data corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th access frequencies, respectively, and mark the timestamps of the sub-data generation corresponding to the i1-th to ii-th sub-data as the i1-th to ii-th timestamps; Get the time value corresponding to the current moment and get the current time value; The data sharing ranking coefficient corresponding to the i1th sub-data is obtained by calculating the i1th access frequency, the i1th timestamp and the current time value, and is named the i1th ranking coefficient; Calculate the data sharing ranking coefficient corresponding to the i1th sub-data. The specific formula configuration is as follows: Spx = |Sc-Sd|*Fc; Wherein, Spx is the data sharing ranking coefficient corresponding to the i1th sub-data, Sc is the time value corresponding to the i1th timestamp, Sd is the current time value, and Fc is the i1th access frequency; According to the access frequencies from i2 to ii, the timestamps from i2 to ii, and the current time value, the data sharing ranking coefficients corresponding to the i2 to ii sub-data are calculated to obtain the i2 to ii ranking coefficients; Arrange the i1th to iith sorting coefficients in descending order according to their numerical values ​​to obtain the ith data sharing sorting sequence.

8. A multivariate big data sharing method according to claim 1, characterized in that: The step F4 further includes the following specific steps: Step F41: Acquire multivariate shared data, and acquire k1 to kk monitoring indicator data according to the multivariate shared data; Step F42: Acquire sharing type division data; Step F42: when the kq-th type data is the second type shared data, data sharing is performed on the ki-th type data according to the kq-th monitoring indicator data; Step F43: Acquire the number of data receiving terminals corresponding to the kq-th type of data to obtain the value of the number of shared terminals; Step F44: when the number of data receiving terminals is greater than p, the kq-th type of data is classified as first terminal type shared data; Step F45: when the number of data receiving terminals is less than or equal to p, the kq-th type of data is classified as second terminal type shared data; Step F46: when the kq-th type data is the first terminal type shared data, data sharing is performed on the kq-th type data; The details are as follows: Step F461: With the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and multiple branch nodes are connected in a star shape, and the main node synchronously shares multiple sub-data in the kq-th type data to multiple branch nodes in the order of the time stamps of the sub-data generation; Step F47: when the kq-th type data is the second terminal type shared data, data sharing is performed on the kq-th type data; The details are as follows: Step F471: Obtain received hierarchical data; Step F472: With the target data platform as the main node and multiple data receiving terminals as branch nodes, the main node and the multiple branch nodes are hierarchically connected according to the received hierarchical data; Step F473: the master node synchronously shares the multiple sub-data in the kq-th type data to the branch node corresponding to the second-layer receiving end hierarchy in the order of the time when the sub-data are generated. The branch node corresponding to the second-layer receiving end hierarchy synchronously shares the multiple sub-data in the kq-th type data to the branch node corresponding to the third-layer receiving end hierarchy in the order of the time when the sub-data are generated. Repeat this sharing process until the multiple sub-data in the kq-th type data are shared to the branch node corresponding to the e+1-th layer receiving end hierarchy. Step F48: Repeat the sharing process for the kq-th type of data, and share each sub-data in each second type of shared data respectively.

9. A multivariate big data sharing method according to claim 8, characterized in that: The step F471 further includes the following specific steps: Name the multiple data receiving terminals as first and z receiving terminals respectively; Performing terminal monitoring on the first receiving terminal to obtain a terminal monitoring coefficient corresponding to the first receiving terminal; The details are as follows: The unit time length before the current moment is taken as the terminal monitoring cycle; Obtaining the total number of accesses to the kq-th type of data by the first receiving terminal within a terminal monitoring period, to obtain a first total number of accesses; Acquire the number of periodic online users of the first receiving terminal in the terminal monitoring period to obtain the first number of online users; Acquire a data receiving delay of a first receiving terminal to obtain a first receiving delay; The details are as follows: Sending test data to the first receiving terminal through the upper level sending terminal of the first receiving terminal, and acquiring the time value corresponding to the sending time to obtain a first time value; The first receiving terminal receives the test data, and acquires the time value corresponding to the receiving moment to obtain a second time value; Calculate the difference between the second time value and the first time value to obtain a first receiving delay; The first total number of visits, the first number of online users, and the first receiving delay are calculated to obtain a terminal monitoring coefficient corresponding to the first receiving terminal; The terminal monitoring coefficient corresponding to the first receiving terminal is calculated, and the specific formula is as follows: Zjx1=(Fws1×Sy1)+Yhs1; Wherein Zjx1 is the terminal monitoring coefficient corresponding to the first receiving terminal, Fws1 is the first total number of accesses, Sy1 is the first receiving delay, and Yhs1 is the first number of online users; Acquire the terminal monitoring coefficient corresponding to each receiving terminal respectively; Acquire multiple terminal monitoring coefficient thresholds respectively, and arrange them in descending order as the first to e-th terminal monitoring coefficient thresholds, and numerically compare the terminal monitoring coefficient corresponding to each receiving terminal with the first to e-th terminal monitoring coefficient thresholds respectively, to obtain the first to e+1th layer receiving terminal classification, and name it as receiving layered data.

10. A multivariate big data sharing system, applicable to a multivariate big data sharing method according to any one of claims 1 to 9, characterized in that: The sharing system comprises: Data acquisition module: used to obtain multiple monitoring indicator data and obtain multi-dimensional shared data; Data analysis module: used to analyze multi-type shared data, divide multiple types of data into first type shared data and second type shared data, and obtain shared type divided data; The first sharing module is used to share the first type of shared data according to the multi-shared data and the shared type classification data. The second sharing module is used to perform big data sharing of the second type of shared data according to the multi-shared data and the shared type division data.

Images