A multi-source beidou space-time big data whole life cycle integration management method

Abstract

The application relates to the technical field of data management, and discloses a multi-source Beidou space-time big data whole life cycle integrated management method, which comprises the following steps: collecting Beidou space-time big data of different sources, and processing the Beidou space-time big data according to a whole life cycle to form a processing flow of a plurality of theme products; obtaining nodes corresponding to each link of each theme product and generating a theme product analysis graph structure; quantifying the public weight of each node; obtaining an initial unique factor of each data attribute of each node; obtaining a plurality of comparison data attributes of each data attribute; quantifying the correlation degree change degree of the data attribute and each comparison data attribute thereof; obtaining a final unique factor of each data attribute of each theme product; and adjusting and distributing storage of a plurality of source data of each data attribute. The application aims to solve the problem that the data attribute correlation is not considered in the whole life cycle integration process of multi-source Beidou space-time big data, and the storage influences the calling efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a full life cycle integrated management method for multi-source Beidou spatiotemporal big data. Background Art

[0002] With the development of technologies such as Beidou navigation, satellite remote sensing, and the Internet of Things, massive spatiotemporal data have been generated in fields such as urban management, power engineering, and emergency monitoring. These data come from diverse sources (such as satellites, drones, and social media) and have different spatiotemporal scales, precision, and timeliness, resulting in decentralized data storage and difficulty in unified management, forming "data islands." In the full life cycle integration and management of multi-source Beidou spatiotemporal big data, it is necessary to set up a universal templated data access process to improve data management efficiency, and at the same time combine distributed storage to achieve efficient integration of cross-source heterogeneous data and provide support for intelligent decision-making.

[0003] The purpose of using multi-source Beidou spatiotemporal big data is to combine the advantages of multi-source data and comprehensively realize the efficient use of data for the use of subsequent thematic products (using positioning theme services, telemetry services, etc.). During the use process, the distributed storage method of data will determine the efficiency of the subsequent management method; however, due to the differences in attributes and forms of multi-source Beidou spatiotemporal big data, if it is stored only according to the source data category, the data call will have efficiency problems, and the joint storage of each source data type will result in low data storage efficiency. Summary of the Invention

[0004] The present invention provides a full life cycle integration management method for multi-source Beidou spatiotemporal big data to solve the problem that the existing multi-source Beidou spatiotemporal big data full life cycle integration process does not consider the data attribute association and stores it, which affects the call efficiency. The technical solution adopted is as follows:

[0005] The present invention proposes a full life cycle integrated management method for multi-source Beidou spatiotemporal big data, which includes the following steps:

[0006] Collect BeiDou spatiotemporal big data from different sources and construct a processing flow for several thematic products based on the entire life cycle;

[0007] Based on the process relationships of several links of each theme product, the nodes corresponding to each link of each theme product are obtained and the theme product analysis graph structure is generated; the connection relationship between each node in the theme product analysis graph structure is analyzed, and the common weight of each node is quantified; based on the distribution of the data attribute combination consisting of the corresponding node and the previous and next nodes of the data attribute displayed by the process relationship of the theme product in the theme product analysis graph structure, the initial unique factor of each data attribute of each node is obtained;

[0008] Analyze the intersection relationship between nodes corresponding to different data attributes to obtain several comparison data attributes of each data attribute; analyze the temporal correlation change relationship between the source data of the data attribute and the comparison data attribute, and quantify the degree of change in the correlation between the data attribute and its comparison data attributes; adjust the initial unique factor based on the degree of change in the correlation between the data attribute and all its comparison data attributes to obtain the final unique factor for each data attribute of each theme product;

[0009] Based on the final unique factors of each data attribute of each theme product, several source data of each data attribute are adjusted and distributedly stored.

[0010] Optionally, the specific method of obtaining the nodes corresponding to each link of each theme product and generating the theme product analysis graph structure includes:

[0011] Each theme product is regarded as a source node in the graph structure, and each link of each theme product is regarded as a child node. The link corresponding to the child node is linked to the theme product corresponding to the process relationship to which it belongs, and the links are linked according to the order of the links of each theme product in its process. The same links of different theme products are represented as the same child nodes in the graph structure; the corresponding child nodes of each link of each theme product in the graph structure are obtained, and the corresponding source nodes of the corresponding theme products are linked. The obtained graph structure is used as the theme product analysis graph structure.

[0012] Optionally, the public weight of each node is obtained by:

[0013] For any node in the topic analysis product graph structure, obtain the number of nodes linked to the node as the number of linked nodes of the node, and take the ratio of the number of linked nodes of the node to the maximum number of linked nodes of all nodes in the topic product analysis graph structure as the public weight of the node.

[0014] Optionally, the specific method of obtaining the initial unique factor of each data attribute of each node includes:

[0015] According to the distribution of the data attribute combination of each node and the data attributes of the previous and next nodes in the theme product analysis graph structure, the previous combination probability and the next combination probability of each data attribute of each node are obtained;

[0016] For any data attribute in any node, the public weight of the node is used as the weight of the pre-combination probability of the data attribute, and the difference obtained by subtracting the public weight of the node from 1 is used as the weight of the post-combination probability of the data attribute. The pre-combination probability and the post-combination probability are weighted and summed, and the result is used as the initial unique factor of the data attribute of the node.

[0017] Optionally, the obtaining of the pre-combination probability and the post-combination probability of each data attribute of each node includes the following specific methods:

[0018] For any data attribute in any node, obtain several previous nodes and several subsequent nodes of the node;

[0019] The combination of the data attribute and any data attribute in any previous node of the node is taken as a previous data attribute combination of the data attribute, the number of times the previous data attribute combination appears in the nodes connected in pairs in the subject product analysis graph structure is obtained as the occurrence frequency of the previous data attribute combination, and the ratio of the occurrence frequency of the previous data attribute combination to the number of edges that are not linked to the source node in the subject product analysis graph structure is taken as the occurrence frequency of the previous data attribute combination; the maximum value of the occurrence frequency of all previous data attribute combinations consisting of the data attribute and all data attributes in all previous nodes of the node is obtained as the previous combination probability of the data attribute;

[0020] Obtain a subsequent data attribute combination consisting of the data attribute and any data attribute in any subsequent node of the node, obtain its occurrence frequency, and then obtain the subsequent combination probability of the data attribute.

[0021] Optionally, the obtaining of several comparison data attributes of each data attribute includes the following specific methods:

[0022] For any data attribute, the nodes including this data attribute are used as the distribution nodes of this data attribute to form the distribution node set of this data attribute; the intersection-and-union ratio of the distribution node set of any other data attribute except this data attribute and the distribution node set of this data attribute is obtained. If the intersection-and-union ratio is greater than the comparison threshold, the other data attribute is used as the comparison data attribute of this data attribute to obtain several comparison data attributes of this data attribute.

[0023] Optionally, the degree of change in the correlation between the data attribute and each of its compared data attributes can be obtained by:

[0024] Monitor the temporal correlation changes of source data of the same subject products based on the data attributes and their comparison data attributes, and obtain the correlation changes between the data attributes and their comparison data attributes in each of the same subject products;

[0025] The average of the correlation changes between any data attribute and any of its comparison data attributes in all the same subject products is taken as the correlation change degree between the data attribute and the comparison data attribute.

[0026] Optionally, the method of obtaining the correlation change between the data attribute and the comparison data attribute in each product with the same subject includes:

[0027] Analyze the data attributes and their comparison data attributes to monitor the correlation changes of source data in several time periods of the same theme product, and obtain the correlation changes of the data attributes and their comparison data attributes in each time period of each same theme product;

[0028] The time lengths of several time periods obtained by monitoring any data attribute and any of its comparison data attributes for any of the same subject products are weighted normalized, and the results obtained are used as reference weights for each time period. The relevant changes corresponding to each time period are weighted and summed with the reference weights, and the results obtained are used as the associated changes between the data attribute and the comparison data attribute in the same subject product.

[0029] Optionally, the method of obtaining the relevant change amount between the data attribute and the comparison data attribute in each time period of each same theme product includes:

[0030] For any data attribute and any of its comparison data attributes, obtain source data for several time periods obtained by monitoring any of the same subject products with respect to the data attribute and the comparison data attribute;

[0031] Starting from the first moment in any time period, the first time window in the time period is obtained based on the initial time length, the source data of the data attribute in the first time window in the time period is obtained, and the Pearson correlation coefficient with the source data of the comparison data attribute in the first time window in the time period is obtained as the correlation coefficient between the data attribute and the comparison data attribute in the first time window of the time period; the time window is amplified according to the amplification time length to obtain a second time window, and the correlation coefficient corresponding to the second time window is obtained, and so on, until the time window includes the entire time period, and the correlation coefficients corresponding to several time windows are obtained;

[0032] The absolute value of the difference between the correlation coefficients of adjacent time windows is calculated, and the accumulated value of the absolute value of the difference between the correlation coefficients of all adjacent time windows is used as the correlation change between the data attribute and the comparison data attribute in the same subject product in the time period.

[0033] Optionally, the specific method for obtaining the final unique factor of each data attribute of each subject product includes:

[0034] The average of the degree of change in the correlation between any data attribute and all its compared data attributes is used as the final unique value adjustment weight of the data awakening; the sum of 1 plus the final unique value adjustment weight is multiplied by the initial unique factor of the data attribute at each node as the final unique factor of the data attribute of the subject product corresponding to each node.

[0035] The beneficial effects of the present invention are as follows: the present invention performs a uniqueness analysis of the source data of the multi-source data of the multi-theme products of the full life cycle of Beidou spatiotemporal big data, and preliminarily quantifies the initial unique factors of the data attributes in each link according to the distribution relationship between the links and data attributes in the processing flow of each theme product, and then adjusts the temporal correlation changes between the source data of the same theme product with different data attributes to obtain the final unique factors reflecting the unique information of each data attribute; wherein the theme product analysis graph structure is constructed by taking the links in the processing flow of each theme product as sub-nodes, integrating the same links between different theme products, and quantifying the common weights of the nodes corresponding to the links to reflect their commonality in different processing flows; and the data attributes corresponding to the source data in the links are used to compare the data attributes of adjacent nodes. Combined analysis, and then based on the distribution of data attribute combinations, quantify the initial unique factors of data attributes in the nodes to reflect some unique information of data attributes under the nodes; obtain data attributes by comparing data attributes based on node intersection, and based on the changes in temporal correlation between source data for the same theme product, quantify the degree of correlation change caused by the unique information of data attributes, and then obtain adjustment weights to adjust the initial unique factors, and obtain the final unique factor, which includes the uniqueness of data attributes in the link processing flow, as well as the uniqueness of the unique information contained in the theme product, so as to adjust the storage method of subsequent multi-source data, ensure the rapid call of multi-source data and rapid response of processing flow of Beidou spatiotemporal big data throughout the life cycle, and improve the efficiency of integrated management. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of a full life cycle integrated management method for multi-source Beidou spatiotemporal big data is provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] See also Figure 1 , which shows a flow chart of a full life cycle integrated management method for multi-source Beidou spatiotemporal big data provided by one embodiment of the present invention, the method comprising the following steps:

[0040] Step S001: Collect BeiDou spatiotemporal big data from different sources and construct a processing flow of several thematic products based on the entire life cycle.

[0041] The purpose of this embodiment is to integrate and manage Beidou spatiotemporal big data from multiple sources throughout their entire life cycle, that is, to store and manage big data. First, it is necessary to obtain Beidou spatiotemporal big data from multiple sources and construct the processing flow for each theme product based on the entire life cycle of each theme product.

[0042] Specifically, Beidou spatiotemporal big data from different sources are collected, including satellite navigation positioning data, geodetic data, remote sensing image data, basic geographic data, and meteorological and physical field data. Each source data contains different data forms, including but not limited to code pseudorange, carrier phase, Doppler frequency shift, carrier-to-noise ratio, navigation message, data length, CRC16-bit check bit, protocol number and equipment code and other message forms for satellite navigation positioning data; topographic surveying and mapping data of geodetic data; map data, place name data and three-dimensional building models of remote sensing image data and basic geographic data; meteorological observation data (temperature, humidity, air pressure), electromagnetic field and wind and cloud field of meteorological physical field data; each type of data is used as source data, and each source data contains several data attributes, such as collection time, coding style, working status code and base station status; the collection of each source data complies with existing data standards and specifications, such as the Beidou satellite navigation system meteorological information transmission specification, and thus obtains several source data and their corresponding data attributes.

[0043] Furthermore, in the multi-source Beidou spatiotemporal big data, the entire life cycle includes the processing flow of several thematic products. The processing flow contains several links. Each link needs to obtain several source data of the corresponding thematic products. The links correspond to the source data and their attributes, and then several links in the processing flow of each thematic product are obtained.

[0044] Step S002: Based on the process relationship of several links of each theme product, obtain the nodes corresponding to each link of each theme product and generate a theme product analysis graph structure; analyze the connection relationship between each node in the theme product analysis graph structure, and quantify the common weight of each node; based on the distribution of the data attribute combination composed of the corresponding node and the previous and next nodes represented by the process relationship of the theme product in the theme product analysis graph structure, obtain the initial unique factor of each data attribute of each node.

[0045] It should be noted that since the use of multi-source Beidou spatiotemporal big data is for product-based use, corresponding process processing is carried out for each thematic product, and different processing links may require joint analysis of different attributes of data from different sources. Therefore, when subsequent thematic products are produced, it is necessary to efficiently call the data involved, and the distributed storage method needs to be adjusted; therefore, by analyzing the use of the attributes of data from different sources between the processing links of different thematic products, the distributed storage data can be reasonably adjusted; the processing links of different thematic products are integrated, and the integration relationship of the processing links of different themes is represented by a graph structure.

[0046] Preferably, in one embodiment of the present invention, based on the process relationship of several links of each theme product, the nodes corresponding to each link of each theme product are obtained and the theme product analysis graph structure is generated, including the specific method of:

[0047] Each theme product is regarded as a source node in the graph structure, and each link of each theme product is regarded as a child node. The link corresponding to the child node is linked to the theme product corresponding to the process relationship to which it belongs, and the links of each theme product are linked in the order of its links in its process. For example, the theme product is divided into 5 links, the link 1 subnode is linked to the source node, the link 2 subnode is linked to the link 1 subnode, and the link 3 involves the output of link 1 and the output of link 2, then the link 3 subnode is linked to the link 1 subnode and the link 2 subnode; it should be noted that since the process relationships of different theme products contain the same links, for example, some theme products need to pre-process the original data before production. (including time format conversion and projection conversion, etc.), in order to express the integration relationship of the processing links, the same links need to be merged; the same links of different theme products are represented as the same child nodes in the graph structure, that is, there is one and only one child node of the same link in the graph structure, that is, there is the possibility of one child node linking multiple theme products; then the corresponding child nodes of each link of each theme product in the graph structure are obtained, and the source nodes corresponding to the corresponding theme products are linked. The obtained graph structure is used as the theme product analysis graph structure, wherein the theme product analysis graph structure is a directed graph, that is, the links between nodes are directed, which is the source node or the child node of the previous link, pointing to the child node of the next link.

[0048] It should be further explained that after obtaining the theme product analysis graph structure, each node in the graph structure represents a different processing link, and each processing link corresponds to a number of data attributes. The data attributes contain different information, such as data time and data format. If they belong to each source, they are public data attributes, and the data attributes that exist in each source data only in a few processing links or only in some source data are unique data attributes in these source data; different data attributes contain different information, and some public data attributes also contain some unique information that can be used for subsequent services, resulting in differences in the use of subsequent theme product production. In order to efficiently call data, it is necessary to reasonably allocate storage resources. Therefore, it is necessary to divide public data attributes and unique data attributes according to the theme product analysis graph structure to serve the subsequent adjustment of storage resources.

[0049] Preferably, in one embodiment of the present invention, the connection relationship between each node in the subject product analysis graph structure is analyzed and the common weight of each node is quantified, including the specific method of:

[0050] For any node in the topic analysis product graph structure (excluding the source node), obtain the number of nodes linked to the node (the child nodes or source nodes of the previous link that the node links to and points to, excluding the child nodes of the next link that the node links to and points to) as the number of linked nodes of the node, and take the ratio of the number of linked nodes of the node to the maximum value of the number of linked nodes of all nodes in the topic product analysis graph structure as the public weight of the node.

[0051] It should be noted that the more linked nodes a sub-node corresponds to, the more processes the sub-node is in, that is, the process relationships of different source nodes are linked to the sub-node, and it is in the processes of different theme products and corresponds to several previous links, then its public weight is greater.

[0052] It should be further explained that the common weight is a characteristic description of the process obtained based on the process relationship of the integrated theme product, and the data attributes involved in the process are an explanation of the distribution of each child node. Therefore, based on the structure of the theme product analysis graph, the distribution of data attributes can be analyzed to determine the impact of data attributes on child nodes; considering the production process sequence of the theme product, the attributes required by the current link are related to the previous link, and also determine the data attributes used in subsequent links. Therefore, it is necessary to obtain the data attribute combination between child nodes based on the theme product analysis graph structure, analyze the distribution of data attributes under each combination, and then obtain the initial unique factor of the data attribute.

[0053] Preferably, in one embodiment of the present invention, based on the distribution of the data attribute combination consisting of the node corresponding to the data attribute displayed by the process relationship of the subject product and the previous and next nodes in the subject product analysis graph structure, the initial unique factor of each data attribute of each node is obtained, including the specific method of:

[0054] For any data attribute in any node, obtain several previous nodes of the node (child nodes that the node links to and points to) and several subsequent nodes of the node (child nodes that the node links to and points to); take the combination of the data attribute and any data attribute in any previous node of the node as a previous data attribute combination of the data attribute, and obtain the number of times the previous data attribute combination appears in the nodes connected in pairs in the theme product analysis graph structure (the link order of the corresponding nodes needs to be consistent with the node link order corresponding to the previous data attribute combination. For example, if the data attribute a in the previous node A and the data attribute b in the node constitute the previous data attribute combination ab, then the data attribute with the number of occurrences is counted. The combination also needs to link the data attribute a in the previous node to the data attribute b in the next node), and take the frequency of occurrence of the previous data attribute combination as the ratio of the frequency of occurrence of the previous data attribute combination to the number of edges that do not link to the source node in the theme product analysis graph structure as the frequency of occurrence of the previous data attribute combination; obtain the maximum value of the frequency of occurrence of all previous data attribute combinations consisting of the data attribute and all data attributes in all previous nodes of the node as the previous combination probability of the data attribute; similarly, obtain the subsequent data attribute combination consisting of the data attribute and any data attribute in any subsequent node of the node, and obtain its frequency of occurrence, and then obtain the subsequent combination probability of the data attribute.

[0055] Furthermore, the public weight of the node is used as the weight of the pre-combination probability of the data attribute, and the difference obtained by subtracting the public weight of the node from 1 is used as the weight of the post-combination probability of the data attribute. The pre-combination probability and the post-combination probability are weighted and summed, and the result obtained is used as the initial unique factor of the data attribute of the node. It should be noted that if there is no post-node for the node, that is, the subsequent link has no child node, the calculation is only based on the pre-combination probability and its weight. Similarly, if there is no pre-node for the node, that is, it is directly linked to the source node, the calculation is only based on the post-combination probability and its weight.

[0056] What needs to be explained is that the reference weights of the front combination probability and the back combination probability are constructed based on the common weight. The larger the common weight, the more reference is needed to the distribution of the previous data attribute combination corresponding to the front combination probability. Based on the distribution of the data attribute combination, the unique distribution of the data attribute of the node is quantified to obtain the initial unique factor.

[0057] At this point, the theme product analysis graph structure is constructed by taking the links in the processing flow of each theme product as sub-nodes, integrating the same links between different theme products, and quantifying the common weights of the nodes corresponding to the links to reflect their commonality in different processing flows; and using this to perform data attribute combination analysis of adjacent nodes on the data attributes corresponding to the source data in the links, and then quantifying the initial unique factors of the data attributes in the nodes based on the distribution of the data attribute combinations, reflecting some unique information of the data attributes under the nodes.

[0058] Step S003: Analyze the intersection relationship between the nodes corresponding to different data attributes to obtain several comparison data attributes of each data attribute; analyze the temporal correlation change relationship of the source data between the data attribute and the comparison data attribute, and quantify the degree of change in the correlation between the data attribute and its comparison data attributes; adjust the initial unique factor based on the degree of change in the correlation between the data attribute and all its comparison data attributes to obtain the final unique factor of each data attribute of each theme product.

[0059] It should be noted that since multi-source data often describe the same thing from different angles, the changes in the attributes of data from different sources should be correlated. For example, the latitude and longitude coordinates and pseudo-range signals of the monitoring target obtained over a period of time can both represent the displacement process of the monitoring target during this period, and belong to different forms of representing the changes in the monitoring target. If the changes between the data attributes in data from different sources are irregular, it means that the data attribute of the source data contains other important information as it changes over time, and the corresponding unique value of the attribute should be larger. Therefore, based on the changes in the relationship between the data attributes of data from different sources for the same monitoring target over a period of time, the initial unique factor of the data attribute is adjusted to obtain the final unique factor.

[0060] Preferably, in one embodiment of the present invention, the intersection relationship between nodes corresponding to different data attributes is analyzed to obtain several comparison data attributes of each data attribute, including the following specific methods:

[0061] Since each data attribute corresponds to multiple nodes, for any data attribute, the node including the data attribute will be used as the distribution node of the data attribute, thereby forming a distribution node set of the data attribute; the intersection-and-union ratio of the distribution node set of any other data attribute except the data attribute and the distribution node set of the data attribute is obtained, and a comparison threshold is preset. In this embodiment, the comparison threshold is described as 0.6. If the intersection-and-union ratio is greater than the comparison threshold, the other data attribute is used as the comparison data attribute of the data attribute, and several comparison data attributes of the data attribute are obtained.

[0062] It's important to note that the presence of unique information is determined by comparing changes in data attributes over time. If there are discrepancies in the change relationships, this indicates the presence of unique information between the two data attributes from the two sources. Therefore, when expressing correlation, we consider the temporal changes in data—that is, the relationship between the changes in data collected over time from different sources for the same monitoring target.

[0063] Preferably, in one embodiment of the present invention, the temporal correlation change relationship between the source data of the data attribute and the comparison data attribute is analyzed, and the degree of change of the correlation between the data attribute and each of its comparison data attributes is quantified, including the specific method of:

[0064] For any data attribute and any of its comparison data attributes, obtain the source data of several time periods obtained by monitoring any same theme product with the data attribute and the comparison data attribute; preset the initial duration and the expansion duration. In this embodiment, the initial duration is described using 5 moments as an example, and the expansion duration is described using 5 moments as an example; starting from the first moment in any time period, obtain the first time window in the time period based on the initial duration, obtain the source data of the first time window of the data attribute in the time period, and compare it with the Pearson correlation coefficient of the source data of the first time window of the comparison data attribute in the time period (corresponding to the same theme product), as the correlation coefficient between the data attribute and the comparison data attribute in the first time window of the time period. The correlation coefficient of a time window; the time window is amplified according to the amplification time to obtain the second time window, that is, the first ten moments, and the correlation coefficient corresponding to the second time window is obtained, and so on, until the time window includes the entire time period, and the correlation coefficients corresponding to several time windows are obtained. It should be noted that if the last moment is less than five, the last time window directly amplifies all the remaining moments on the basis of the penultimate time window to form the last time window; the absolute value of the difference between the correlation coefficients of adjacent time windows is calculated, and the accumulated value of the absolute value of the difference between the correlation coefficients of all adjacent time windows is used as the correlation change between the data attribute and the comparison data attribute in the same subject product in this time period.

[0065] Furthermore, the lengths of the respective time periods obtained by monitoring any identical subject product for the data attribute and the comparison data attribute are weighted and normalized, and the obtained results are used as reference weights for each time period. The relevant changes corresponding to each time period are weighted and summed with the reference weights, and the obtained results are used as the correlation change between the data attribute and the comparison data attribute in the identical subject product. Since the data attribute and the comparison data attribute may jointly monitor multiple subject products to obtain source data, the average of the correlation change between the data attribute and the comparison data attribute in all identical subject products is used as the degree of change in the correlation between the data attribute and the comparison data attribute.

[0066] What needs to be explained is that by performing correlation analysis on the source data obtained from the same monitoring target for data attributes and their comparison data attributes, the change in the correlation between the data attributes and the comparison data attributes is analyzed based on the change in the temporal correlation between the source data in several time periods, and the weighted averaging is performed with the length of the time period as the weight to present the change in correlation caused by the unique information contained between the data attributes and the comparison data attributes, and then the degree of correlation change is obtained.

[0067] It should be noted that the same data attributes contained in the corresponding nodes of different links actually correspond to the same source data but are used in the processing and production processes of different theme products. The final unique factor is obtained based on the initial unique factor of the data attributes of each node, combined with the degree of change in the correlation corresponding to the comparison data attributes, and the comprehensive quantification of the unique information contained in the data attributes compared with other data attributes.

[0068] Preferably, in one embodiment of the present invention, the initial unique factor is adjusted based on the degree of change in the correlation between the data attribute and all its compared data attributes to obtain the final unique factor of each data attribute of each subject product, including the following specific methods:

[0069] The average of the degree of change in the correlation between any data attribute and all its compared data attributes is used as the final unique value adjustment weight of the data awakening; the sum obtained by adding 1 to the final unique value adjustment weight is multiplied by the initial unique factor of the data attribute at each node as the final unique factor of the data attribute of the subject product corresponding to each node (the subject product corresponding to the source node of the node link), that is, the initial unique factor of the data attribute of each node is multiplied by the sum to obtain the final unique factor of the data attribute of each node, and based on the subject product corresponding to the source node of the node link, the final unique factor of the data attribute of the subject product is obtained.

[0070] It should be noted that the degree of change in correlation reflects the unique information contained in the source data and the comparison data attributes through the change in the correlation of the source data. The content of the unique information that needs to be adjusted is further quantified by averaging to obtain the final unique value adjustment weight. At the same time, since the same data attributes of different nodes actually correspond to the same source data, the initial unique factors of the data attributes of different nodes are adjusted with the same adjustment weight to obtain the final unique factor.

[0071] At this point, by obtaining data attributes by comparing them based on node intersection, and based on the changes in temporal correlation between source data for the same theme product, the degree of correlation change caused by the unique information of the data attributes is quantified, and then the adjustment weight is obtained to adjust the initial unique factor, and the final unique factor is obtained, which includes the uniqueness of the data attributes in the link processing flow and the uniqueness of the unique information contained in the theme product, so as to adjust the storage method of subsequent multi-source data.

[0072] Step S004: Based on the final unique factors of each data attribute of each theme product, the distributed storage of several source data of each data attribute is adjusted to realize the integration and storage management of the full life cycle data of multi-source Beidou spatiotemporal big data.

[0073] It should be noted that the entire life cycle of multi-source Beidou spatiotemporal big data includes several source data of several subject product processing flows, and each source data corresponds to a data attribute. The final unique factor of the data attribute reflects whether it is called by other subject products in the subsequent management process under distributed storage conditions. In this way, distributed storage and centralized cloud storage of multi-source data throughout the entire life cycle are carried out, thereby realizing the integration of data throughout the entire life cycle of multi-source Beidou spatiotemporal big data.

[0074] Specifically, a unique threshold is preset. In this embodiment, the unique threshold is described as 0.7. If the final unique factor of any data attribute of any theme product is greater than the unique threshold, the data attribute is a unique data attribute of the theme product. Otherwise, if it is less than or equal to the unique threshold, it is a public data attribute. The unique data attributes of each theme product are distributedly stored, and the corresponding source data are stored in the servers of the distributed nodes. For the public data attributes of each theme product, the corresponding source data are centrally stored in the cloud server to facilitate the rapid call of the subsequent processing flow of other theme products, thereby realizing the storage management of multi-source Beidou spatiotemporal big data throughout the entire life cycle, and then realizing integrated management. It should be noted that if the same source data contains public data attributes and unique data attributes, they are stored according to the unique data attributes.

[0075] At this point, by analyzing the uniqueness of the source data of multi-source data of multi-theme products throughout the life cycle of Beidou spatiotemporal big data, and based on the distribution relationship between the links and data attributes in the processing flow of each theme product, the initial unique factors of data attributes in each link are preliminarily quantified. Then, through the changes in the temporal correlation between the source data of the same theme product with different data attributes, the final unique factors are adjusted to reflect the unique information of each data attribute, so as to facilitate the adjustment of the multi-source data storage method, ensure the rapid call of multi-source data and the rapid response of the processing flow of Beidou spatiotemporal big data throughout the life cycle, and improve the efficiency of integrated management.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A full life cycle integrated management method for multi-source BeiDou spatiotemporal big data, characterized by: The method comprises the following steps: Collect BeiDou spatiotemporal big data from different sources and construct a processing flow for several thematic products based on the entire life cycle; Based on the process relationships of several links of each theme product, the nodes corresponding to each link of each theme product are obtained and the theme product analysis graph structure is generated; the connection relationship between each node in the theme product analysis graph structure is analyzed, and the common weight of each node is quantified; based on the distribution of the data attribute combination consisting of the corresponding node and the previous and next nodes of the data attribute displayed by the process relationship of the theme product in the theme product analysis graph structure, the initial unique factor of each data attribute of each node is obtained; The public weight of each node is obtained in the following way: For any node in the topic analysis product graph structure, obtain the number of nodes linked to the node as the number of linked nodes of the node, and calculate the ratio of the number of linked nodes of the node to the maximum number of linked nodes of all nodes in the topic product analysis graph structure as the public weight of the node; The specific method for obtaining the initial unique factor of each data attribute of each node includes: According to the distribution of the data attribute combination of each node and the data attributes of the previous and next nodes in the theme product analysis graph structure, the previous combination probability and the next combination probability of each data attribute of each node are obtained; For any data attribute in any node, the public weight of the node is used as the weight of the pre-combination probability of the data attribute, and the difference obtained by subtracting the public weight of the node from 1 is used as the weight of the post-combination probability of the data attribute. The weighted sum of the pre-combination probability and the post-combination probability is used as the initial unique factor of the data attribute of the node; Analyze the intersection relationship between nodes corresponding to different data attributes to obtain several comparison data attributes of each data attribute; analyze the temporal correlation change relationship between the source data of the data attribute and the comparison data attribute, and quantify the degree of change in the correlation between the data attribute and its comparison data attributes; adjust the initial unique factor based on the degree of change in the correlation between the data attribute and all its comparison data attributes to obtain the final unique factor for each data attribute of each theme product; Based on the final unique factors of each data attribute of each theme product, several source data of each data attribute are adjusted and distributedly stored.

2. The full life cycle integrated management method of multi-source BeiDou spatiotemporal big data according to claim 1 is characterized in that: The specific method of obtaining the nodes corresponding to each link of each theme product and generating the theme product analysis graph structure includes: Each theme product is regarded as a source node in the graph structure, and each link of each theme product is regarded as a child node. The link corresponding to the child node is linked to the theme product corresponding to the process relationship to which it belongs, and the links are linked according to the order of the links of each theme product in its process. The same links of different theme products are represented as the same child nodes in the graph structure; the corresponding child nodes of each link of each theme product in the graph structure are obtained, and the corresponding source nodes of the corresponding theme products are linked. The obtained graph structure is used as the theme product analysis graph structure.

3. The full life cycle integrated management method of multi-source Beidou spatiotemporal big data according to claim 2 is characterized in that: The specific method of obtaining the pre-combination probability and post-combination probability of each data attribute of each node includes: For any data attribute in any node, obtain several previous nodes and several subsequent nodes of the node; The combination of the data attribute and any data attribute in any previous node of the node is taken as a previous data attribute combination of the data attribute, the number of times the previous data attribute combination appears in the nodes connected in pairs in the subject product analysis graph structure is obtained as the occurrence frequency of the previous data attribute combination, and the ratio of the occurrence frequency of the previous data attribute combination to the number of edges that are not linked to the source node in the subject product analysis graph structure is taken as the occurrence frequency of the previous data attribute combination; the maximum value of the occurrence frequency of all previous data attribute combinations consisting of the data attribute and all data attributes in all previous nodes of the node is obtained as the previous combination probability of the data attribute; Obtain a subsequent data attribute combination consisting of the data attribute and any data attribute in any subsequent node of the node, obtain its occurrence frequency, and then obtain the subsequent combination probability of the data attribute.

4. The full life cycle integrated management method of multi-source BeiDou spatiotemporal big data according to claim 1 is characterized in that: The specific method of obtaining several comparison data attributes of each data attribute includes: For any data attribute, the nodes including this data attribute are used as the distribution nodes of this data attribute to form the distribution node set of this data attribute; the intersection-and-union ratio of the distribution node set of any other data attribute except this data attribute and the distribution node set of this data attribute is obtained. If the intersection-and-union ratio is greater than the comparison threshold, the other data attribute is used as the comparison data attribute of this data attribute to obtain several comparison data attributes of this data attribute.

5. The full life cycle integrated management method of multi-source BeiDou spatiotemporal big data according to claim 1 is characterized in that: The degree of change in the correlation between the data attribute and its respective comparison data attributes is obtained in the following way: Monitor the temporal correlation changes of source data of the same subject products based on the data attributes and their comparison data attributes, and obtain the correlation changes between the data attributes and their comparison data attributes in each of the same subject products; The average of the correlation changes between any data attribute and any of its comparison data attributes in all the same subject products is taken as the correlation change degree between the data attribute and the comparison data attribute.

6. The full life cycle integrated management method of multi-source BeiDou spatiotemporal big data according to claim 5 is characterized in that: The specific method of obtaining the correlation change between the data attribute and the comparison data attribute in each product with the same subject matter is as follows: Analyze the data attributes and their comparison data attributes to monitor the correlation changes of source data in several time periods of the same theme product, and obtain the correlation changes of the data attributes and their comparison data attributes in each time period of each same theme product; The time lengths of several time periods obtained by monitoring any data attribute and any of its comparison data attributes for any of the same subject products are weighted normalized, and the results obtained are used as reference weights for each time period. The relevant changes corresponding to each time period are weighted and summed with the reference weights, and the results obtained are used as the associated changes between the data attribute and the comparison data attribute in the same subject product.

7. The full life cycle integrated management method of multi-source BeiDou spatiotemporal big data according to claim 6 is characterized in that: The specific method for obtaining the relevant change amount of the data attribute and the comparison data attribute in each time period of each same theme product includes: For any data attribute and any of its comparison data attributes, obtain source data for several time periods obtained by monitoring any of the same subject products with respect to the data attribute and the comparison data attribute; Starting from the first moment in any time period, the first time window in the time period is obtained based on the initial time length, the source data of the data attribute in the first time window in the time period is obtained, and the Pearson correlation coefficient with the source data of the comparison data attribute in the first time window in the time period is obtained as the correlation coefficient between the data attribute and the comparison data attribute in the first time window of the time period; the time window is amplified according to the amplification time length to obtain a second time window, and the correlation coefficient corresponding to the second time window is obtained, and so on, until the time window includes the entire time period, and the correlation coefficients corresponding to several time windows are obtained; The absolute value of the difference between the correlation coefficients of adjacent time windows is calculated, and the accumulated value of the absolute value of the difference between the correlation coefficients of all adjacent time windows is used as the correlation change between the data attribute and the comparison data attribute in the same subject product in the time period.

8. The full life cycle integrated management method of multi-source BeiDou spatiotemporal big data according to claim 1 is characterized in that: The specific method for obtaining the final unique factor of each data attribute of each subject product includes: The average of the degree of change in the correlation between any data attribute and all its compared data attributes is used as the final unique value adjustment weight of the data awakening; the sum of 1 plus the final unique value adjustment weight is multiplied by the initial unique factor of the data attribute at each node as the final unique factor of the data attribute of the subject product corresponding to each node.

Images