A platform data processing management method and system based on big data

Abstract

The present invention relates to the field of data management technology, and in particular to a platform data processing management method and system based on big data. A platform data processing management method based on big data, comprising the following steps: S1: using NLP to obtain a unique semantic label for each data item field in a data warehouse, and adjusting the usage quantity of each data item in all downstream systems according to the unique semantic label; S2: based on the adjusted usage quantity in the downstream system, using a comprehensive weight formula to obtain the comprehensive weight of each data item, using a weighted time calculation formula according to the comprehensive weight, obtaining a weighted average waiting time, and obtaining a cache adjustment formula according to the weighted average waiting time. The present invention adapts to the complex needs of different scenarios by adjusting the cache size and data management rules in real time according to the dynamic needs of the downstream system, while providing good support for future system expansion.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a platform data processing management method and system based on big data. Background Art

[0002] With the advancement of informatization, more and more enterprises and organizations rely on data warehouse systems to process and store large amounts of data in their daily operations. However, with the diversification of downstream applications, different applications have different requirements for real-time data. For example, some applications require real-time access and update of data, while others only rely on periodic data updates. This difference in requirements will cause the following problems: inconsistency between real-time data and scheduled updates. When applications that rely on real-time data fail to obtain data updates in a timely manner, they will read outdated information, making wrong decisions or affecting business processes; data synchronization delays. Differences in the frequency of data updates by downstream systems lead to data synchronization delays, which in turn affects the access accuracy of applications with strong real-time requirements. Summary of the invention

[0003] In order to overcome the disadvantage of data synchronization delay in data warehouse, the present invention provides a platform data processing management method and system based on big data.

[0004] The technical solution is: a platform data processing and management method based on big data, comprising the following steps:

[0005] S1: Use NLP to obtain a unique semantic label for each data item field in the data warehouse, and adjust the usage quantity of each data item in all downstream systems according to the unique semantic label;

[0006] S2: Based on the adjusted usage quantity in the downstream system, a comprehensive weight formula is used to obtain a comprehensive weight of each data item, a weighted time calculation formula is used according to the comprehensive weight to obtain a weighted average waiting time, and a cache adjustment formula is obtained according to the weighted average waiting time;

[0007] S3: obtaining access data of each downstream system and access data of metadata in the data warehouse, using a cache adjustment formula according to the access frequency of each downstream system, setting an intermediate cache and adjusting the cache size of the intermediate cache;

[0008] S4: updating the metadata mapping rules in the data warehouse based on the unique semantic tag, and forgetting the auxiliary mapping rules using a forgetting formula.

[0009] Preferably, the access data of each downstream system and the access data of metadata in the data warehouse are obtained, and a cache adjustment formula is used according to the access frequency of each downstream system to set an intermediate cache and adjust the cache size of the intermediate cache, including: the cache adjustment formula is:

[0010] ;

[0011] In the formula, is the intermediate cache size; is the weight coefficient; The update frequency of the downstream system; is the cache hit rate; is the system resource utilization; is the offset; It is the weighted average waiting time of each data.

[0012] Preferably, the weighted average waiting time of each data includes: obtaining the weight of each data item according to the usage data of the same data item in each downstream system in the data warehouse, and processing the waiting time of each data in the intermediate buffer using a weighted time calculation formula to obtain the weighted average waiting time, wherein the weighted time calculation formula is:

[0013] ;

[0014] In the formula, For the The weight of each data item; For the The waiting time of a data item in the intermediate cache.

[0015] Preferably, obtaining the weight of each data item according to the usage data of the same data item in each downstream system in the data warehouse includes: collecting the usage times and update frequency data of each data item in the data warehouse, and using a comprehensive weight formula to obtain the comprehensive weight of each data item, wherein the comprehensive weight formula is:

[0016] ;

[0017] In the formula, is the weight coefficient; For the The number of times a data item appears in the data warehouse; For the The update frequency of each data item in the data warehouse; For the The number of uses of a data item in all downstream systems; is the total usage quantity of all data items in all downstream systems.

[0018] Preferably, before collecting the usage count and update frequency data of each data item in the data warehouse and using the comprehensive weight formula to obtain the comprehensive weight of each data item, the method includes: collecting the name, field annotation, data description, and contextual information of each data item field in the data warehouse, and analyzing the field annotation and semantic information using natural language processing technology; extracting the core semantic components of the field name through word segmentation and part-of-speech tagging technology, analyzing the field usage based on contextual relationships, and judging the semantic scope of the field in combination with the qualifiers in the data description; matching and comparing the semantic information in the field name and the field description, identifying ambiguous fields, and marking conflicting fields; automatically generating unique semantic tags for detected conflicting fields according to the semantic scope; and adjusting the usage quantity of each data item in all downstream systems according to the unique semantic tags.

[0019] Preferably, adjusting the usage quantity of each data item in all downstream systems according to the unique semantic tag comprises: obtaining a field relevance factor representing a field relevance degree, and obtaining the usage quantity of the data item in all downstream systems according to the field relevance degree,

[0020] When fields are highly correlated, they are treated as the same data item, and their usage counts in downstream systems are combined to be counted as the usage count of the data item in all downstream systems.

[0021] When the field correlation is weak, it is treated as an independent data item and the usage quantity of the data item in all downstream systems is calculated separately.

[0022] Preferably, obtaining the usage quantity of the data item in all downstream systems according to the field association degree includes: adjusting the usage quantity of the data item in all downstream systems using an association adjustment formula, wherein the association adjustment formula is:

[0023] ;

[0024] In the formula, The number of independent uses of each field; is the correlation factor.

[0025] Preferably, the metadata mapping rules in the data warehouse are updated based on the unique semantic tag, and the auxiliary mapping rules are forgotten using a forgetting formula, including: adding the unique semantic tag to the metadata mapping rule as an auxiliary mapping rule, and comparing the unique semantic tag with the remaining auxiliary mapping rules in the metadata mapping rule, when the semantic similarity is greater than a preset threshold, replacing the remaining auxiliary mapping rules; when the semantic similarity is less than or equal to the preset threshold, adding it to the auxiliary mapping rule; and forgetting the auxiliary mapping rule using a forgetting formula.

[0026] Preferably, the forgetting of the auxiliary mapping rule by using a forgetting formula includes: wherein the forgetting formula is:

[0027] ;

[0028] In the formula, is the adjustment factor, is the calling frequency of each auxiliary mapping rule; Interpolate the most recent call time of each auxiliary mapping rule, The time difference is called for the standard.

[0029] A platform data processing and management system based on big data, comprising:

[0030] The data collection module is used to collect the usage count, update frequency, field annotation, data description, and field context information of the fields in the data warehouse;

[0031] The weight calculation module calculates the comprehensive weight according to the weight calculation formula based on the number of times the data item is used, the update frequency, and the number of times it is used in the downstream system;

[0032] The cache adjustment module calculates the intermediate cache size based on the access frequency, cache hit rate and system resource utilization of the downstream system, monitors the cache performance indicators in real time, and dynamically adjusts the cache size;

[0033] The semantic analysis module uses natural language processing technology to extract the core semantic components in field names and descriptions, analyze field semantic conflicts, generate unique semantic labels, and mark the associations between fields.

[0034] The data adjustment module adjusts the usage quantity of data items in all downstream systems based on semantic tags and field relevance, merges highly relevant fields, and independently calculates the usage quantity of low-relevance fields;

[0035] The metadata management module updates metadata mapping rules based on unique semantic tags, uses the forgetting formula to eliminate infrequently used mapping rules, and optimizes storage resources.

[0036] The beneficial effects are as follows: the present invention solves the problem of inconsistent data real-time requirements of different applications by introducing intermediate cache;

[0037] 1. Solve the problem of data real-time and latency. The intermediate cache acts as a buffer between the data warehouse and the downstream system, so that the acquisition of real-time data and the request for regular updates are processed independently. When an application with high real-time requirements accesses data, the cache will be updated in advance, avoiding the reading of outdated information due to data latency, and ensuring that the downstream system reads the latest data;

[0038] 2. Optimize cache resource utilization and improve system performance. Based on the access frequency and resource requirements of different downstream systems, the present invention uses a cache adjustment formula to dynamically adjust the size and content of the cache, ensuring the reasonable allocation of the intermediate cache. Without affecting the real-time performance of the data, the system load is reduced through intelligent cache management, reducing the waste of resources caused by excessive caching;

[0039] 3. Reduce the reading of outdated data, improve data consistency, use the weighted time calculation formula to manage cache data, balance the cache hit rate and waiting time, so that applications with high real-time requirements can obtain the latest data in the first time, thus avoiding reading outdated information;

[0040] 4. Improve the efficiency of data synchronization and reduce storage requirements. Through dynamic management of cache contents of different systems, the intermediate cache can efficiently handle the synchronization issues between real-time data updates and regular updates, reducing the pressure on the data warehouse to directly provide real-time update data, thereby reducing the consumption of storage and computing resources;

[0041] 5. By adding the corresponding unique semantic label to generate auxiliary mapping rules, it is convenient for the system to match and distinguish next time. At the same time, by using the forgetting formula to forget the auxiliary mapping rules, the consumption of storage and computing resources is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a platform data processing and management method based on big data of the present invention;

[0043] Figure 2 The present invention is a flow chart of a platform data processing and management system based on big data. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Example 1: A platform data processing and management method based on big data, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0046] S1: Use NLP to obtain a unique semantic label for each data item field in the data warehouse, and adjust the usage quantity of each data item in all downstream systems according to the unique semantic label;

[0047] S2: Based on the adjusted usage quantity in the downstream system, a comprehensive weight formula is used to obtain a comprehensive weight of each data item, a weighted time calculation formula is used according to the comprehensive weight to obtain a weighted average waiting time, and a cache adjustment formula is obtained according to the weighted average waiting time;

[0048] S3: obtaining access data of each downstream system and access data of metadata in the data warehouse, using a cache adjustment formula according to the access frequency of each downstream system, setting an intermediate cache and adjusting the cache size of the intermediate cache;

[0049] The cache adjustment formula is:

[0050] ;

[0051] In the formula, is the intermediate cache size; is the weight coefficient; The update frequency of the downstream system; is the cache hit rate; is the system resource utilization; is the offset; It is the weighted average waiting time of each data.

[0052] It should be noted that the purpose of the cache adjustment formula is to dynamically calculate and adjust the size of the intermediate cache to ensure the real-time and efficiency of data access. The formula determines the size and content of the cache based on the access frequency of the downstream system, the cache hit rate, and the system resource utilization, thereby optimizing the overall performance of the system; To represent the frequency with which downstream systems access the data warehouse, frequently accessed systems require small cache spaces to reduce access latency; Indicates the ratio of data accessed in the cache. A high hit rate means that the cache is efficient and the system does not need to load data from the database frequently. Therefore, the cache size should be appropriately reduced. Indicates the resource usage of the entire system, usually including CPU utilization and memory usage. If the system resource utilization is high, it means that the system is under heavy load. In this case, the cache size needs to be adjusted appropriately to avoid further increasing the system load. It represents the weighted average waiting time of each data item in the intermediate cache. When the waiting time of data items in the cache is long, it indicates that the cache is not updated in time, resulting in data lag. Therefore, when the W value is high, the size of the intermediate cache is reduced and sent to the data warehouse or downstream system when the intermediate cache is full, reducing the data waiting time; To adjust for the effects of other variables, such as tuning based on the specific needs of the system or experimental data, ensure that the final calculation of the cache size meets the actual needs.

[0053] According to the usage data of the same data item in the data warehouse in each downstream system, the weight of each data item is obtained, and the waiting time of each data in the intermediate buffer is processed using the weighted time calculation formula to obtain the weighted average waiting time, wherein the weighted time calculation formula is:

[0054] ;

[0055] In the formula, For the The weight of each data item; For the The waiting time of a data item in the intermediate cache.

[0056] It should be noted that The weighted average waiting time of a data item in the intermediate cache is usually in seconds or milliseconds. This value reflects the length of time a data item is stored in the intermediate cache and its impact on system response. To indicate the importance or frequency of use of the data item in downstream systems; It indicates the waiting time for the data item to be stored in the cache. The longer the waiting time is, the longer the update cycle of the data item is and the data in the cache is delayed. If the waiting time is short, it means that the data item is updated in a timely manner.

[0057] Collect the usage and update frequency data of each data item in the data warehouse, and use the comprehensive weight formula to obtain the comprehensive weight of each data item, where the comprehensive weight formula is:

[0058] ;

[0059] In the formula, is the weight coefficient; For the The number of times a data item appears in the data warehouse; For the The update frequency of each data item in the data warehouse; For the The number of uses of a data item in all downstream systems; is the total usage quantity of all data items in all downstream systems.

[0060] It should be noted that, first of all, To collect the number of times each data item is used in the data warehouse, this can be done by tracking the call frequency of data items in the warehouse through logging, query statistics or other means. For example, the data item "user id" appears 100 times in the data warehouse, which indicates the number of times it is accessed in the data warehouse; Update frequency refers to the frequency with which the data item is updated in the data warehouse; The call status of the data item in each downstream system. For example, a data item "product type" is called 10 times, 20 times, 30 times, 40 times and 50 times in 5 downstream systems respectively. After combining all or part of these call times, the total usage of the data item in all downstream systems is obtained; is the total number of uses of all data items in all downstream systems. It serves as a normalization parameter so that the calculation of weights can remain consistent across datasets of different sizes.

[0061] Collect the name, field annotation, data description, and context information of each data item field in the data warehouse, and use natural language processing technology to analyze the field annotation and semantic information; extract the core semantic components of the field name through word segmentation and part-of-speech tagging technology, analyze the field usage based on the contextual relationship, and determine the semantic scope of the field in combination with the qualifiers in the data description; match and compare the semantic information in the field name and field description, identify ambiguous fields, and mark conflicting fields; for the detected conflicting fields, automatically generate unique semantic tags according to the semantic scope; and adjust the usage quantity of each data item in all downstream systems according to the unique semantic tags.

[0062] It should be noted that there are problems with ambiguity. For example, "customer ID" is ambiguous, with different situations such as "sales_customer ID" or "service_customer ID", but the word "customer ID" is used in all of them. First of all, each data item has its field name, field annotation, data description and context information in the data warehouse. Through this information, combined with natural language processing technology, the core semantics of the data item is extracted, and then NLP technology is used for semantic analysis, word segmentation and part-of-speech tagging: field names and field descriptions are segmented to extract core semantic components. For example, the word segmentation of "sales-amount" is "sales" and "amount", which represent "sales" and "amount" respectively; context analysis: by analyzing the qualifiers and context in the data description, the specific purpose of the data item is determined. For example, the field "amount" means "sales amount" or "refund amount" in different contexts, and NLP technology infers its actual meaning through context; polysemous word identification and conflicting field marking: when a field name has different meanings in different systems or scenarios, it is identified and marked as a polysemous field through semantic analysis. For example, "user-id" represents user identity in some scenarios and account ID in other scenarios. The system needs to mark these conflicting fields; for fields with ambiguity or semantic conflicts, the system automatically generates unique semantic labels to ensure the consistency and accuracy of the data items in all downstream systems. For example, Field 1: It is recommended to be named "sales-customer ID"; Field 2: It is recommended to be named "service-customer ID".

[0063] The relevance factor of the field is obtained to represent the degree of field relevance, and the number of uses of the data item in all downstream systems is obtained based on the degree of field relevance.

[0064] When fields are highly correlated, they are treated as the same data item, and their usage counts in downstream systems are combined to be counted as the usage count of the data item in all downstream systems.

[0065] When the field correlation is weak, it is treated as an independent data item and the usage quantity of the data item in all downstream systems is calculated separately.

[0066] Use the association adjustment formula to adjust the usage quantity of the data item in all downstream systems, where the association adjustment formula is:

[0067] ;

[0068] In the formula, The number of independent uses of each field; is the correlation factor.

[0069] It should be noted that the correlation between fields is determined by analyzing the semantic information of the fields (such as field name, field annotation, context). The correlation between fields is evaluated through natural language processing technology, combining context, semantics and application scenarios to determine whether two fields represent the same data item.

[0070] When the fields are strongly related, the two fields are different representations of the same data item or represent the same entity in different systems. For example, "user-id" and "user-account-id" are identifiers of the same user and they represent the same data item;

[0071] When the field association is weak, the two fields represent different data items and even have completely different meanings in different systems. For example, although "product-id" and "order-id" both contain "id", they represent product identifiers and order identifiers respectively, so they should be treated as independent data items;

[0072] This correlation factor reflects the degree of semantic or functional correlation between fields. If two fields are highly correlated, then is close to 1; if the correlation is weak, then Close to 0.

[0073] S4: updating the metadata mapping rules in the data warehouse based on the unique semantic tag, and forgetting the auxiliary mapping rules using a forgetting formula.

[0074] The unique semantic tag is added to the metadata mapping rule as an auxiliary mapping rule, and the unique semantic tag is compared with the remaining auxiliary mapping rules in the metadata mapping rule. When the semantic similarity is greater than a preset threshold, the remaining auxiliary mapping rules are replaced; when the semantic similarity is less than or equal to the preset threshold, it is added to the auxiliary mapping rule; and the auxiliary mapping rule is forgotten using a forgetting formula.

[0075] It should be noted that by using unique semantic tags to update metadata mapping rules, the management of metadata in the data warehouse is optimized. When a new data item or field is discovered, its semantic similarity is calculated and compared with the mapping rules of the existing metadata to determine whether it is necessary to replace or add new rules. This process helps to keep the mapping rules dynamically updated and ensure that the metadata in the data warehouse is always consistent with the data requirements of the downstream system. In the data warehouse, each data item or field is identified by a unique semantic label. These labels are generated based on the semantic information of the field name, field annotation, and data description to ensure that the meaning of the field can be accurately expressed when data is transferred between different systems. When a new data item or field is added to the data warehouse, a new unique semantic label is generated. By calculating the semantic similarity between the new label and the existing label, it is determined whether the existing mapping rules need to be updated. The semantic similarity is usually calculated using a natural language processing algorithm (such as a vector space model, a word embedding model). When the semantic similarity is greater than a preset threshold, the new unique semantic label has the ability to replace the original mapping rule; when the semantic similarity is less than or equal to the preset threshold, it is regarded as a new data item and added to the existing auxiliary mapping rules.

[0076] The forgetting formula is:

[0077] ;

[0078] In the formula, is the adjustment factor, is the calling frequency of each auxiliary mapping rule; Interpolate the most recent call time of each auxiliary mapping rule, The time difference is called for the standard.

[0079] It should be noted that the forgetting formula is used to dynamically manage and optimize the auxiliary mapping rules. The application goal of the forgetting formula is to ensure that those auxiliary mapping rules that are no longer frequently used can be deleted in a timely manner, thereby saving storage space and avoiding redundant rules from affecting the efficiency and accuracy of the system. The forgetting formula is calculated based on the calling frequency and time difference of the mapping rules to determine which rules should be forgotten.

[0080] Embodiment 2: Based on Embodiment 1, a platform data processing and management system based on big data includes:

[0081] The data collection module is used to collect the usage count, update frequency, field annotation, data description, and field context information of the fields in the data warehouse;

[0082] The weight calculation module calculates the comprehensive weight according to the weight calculation formula based on the number of times the data item is used, the update frequency, and the number of times it is used in the downstream system;

[0083] The cache adjustment module calculates the intermediate cache size based on the access frequency, cache hit rate and system resource utilization of the downstream system, monitors the cache performance indicators in real time, and dynamically adjusts the cache size;

[0084] The semantic analysis module uses natural language processing technology to extract the core semantic components in field names and descriptions, analyze field semantic conflicts, generate unique semantic labels, and mark the associations between fields.

[0085] The data adjustment module adjusts the usage quantity of data items in all downstream systems based on semantic tags and field relevance, merges highly relevant fields, and independently calculates the usage quantity of low-relevance fields;

[0086] The metadata management module updates metadata mapping rules based on unique semantic tags, uses the forgetting formula to eliminate infrequently used mapping rules, and optimizes storage resources.

[0087] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A platform data processing and management method based on big data, characterized in that: The following steps are involved: S1: Use NLP to obtain a unique semantic label for each data item field in the data warehouse, and adjust the usage quantity of each data item in all downstream systems according to the unique semantic label; S2: Based on the adjusted usage quantity in the downstream system, a comprehensive weight formula is used to obtain a comprehensive weight of each data item, a weighted time calculation formula is used according to the comprehensive weight to obtain a weighted average waiting time, and a cache adjustment formula is obtained according to the weighted average waiting time; S3: obtaining access data of each downstream system and access data of metadata in the data warehouse, using a cache adjustment formula according to the access frequency of each downstream system, setting an intermediate cache and adjusting the cache size of the intermediate cache; S4: updating the metadata mapping rule in the data warehouse based on the unique semantic tag, and forgetting the auxiliary mapping rule using a forgetting formula; The method further comprises: obtaining a field relevance factor to represent a field relevance degree, and obtaining a data item usage quantity in all downstream systems according to the field relevance degree, When the fields are highly correlated, they are treated as the same data item, and their usage counts in downstream systems are combined to uniformly count the usage counts of the data item in all downstream systems. When the fields are weakly correlated, they are treated as independent data items, and the usage counts of the data item in all downstream systems are calculated separately.

2. The platform data processing and management method based on big data according to claim 1 is characterized in that: The access data of each downstream system and the access data of metadata in the data warehouse are obtained, and a cache adjustment formula is used according to the access frequency of each downstream system to set an intermediate cache and adjust the cache size of the intermediate cache, including: the cache adjustment formula is: Where C is the size of the intermediate cache; K1, K2, K3, and K4 are weight coefficients; F is the update frequency of the downstream system; H is the cache hit rate; R is the system resource utilization; B is the offset; and W is the weighted average waiting time of each data.

3. The platform data processing and management method based on big data according to claim 2 is characterized in that: The weighted average waiting time of each data includes: obtaining the weight of each data item according to the usage data of the same data item in each downstream system in the data warehouse, and processing the waiting time of each data in the intermediate buffer using a weighted time calculation formula to obtain the weighted average waiting time, wherein the weighted time calculation formula is: In the formula, w i is the weight of the ith data item; t i is the waiting time of the i-th data item in the intermediate cache.

4. The platform data processing and management method based on big data according to claim 3 is characterized in that: The method of obtaining the weight of each data item according to the usage data of the same data item in each downstream system in the data warehouse includes: collecting the usage times and update frequency data of each data item in the data warehouse, and using a comprehensive weight formula to obtain the comprehensive weight of each data item, wherein the comprehensive weight formula is: Where k1, k2, k3, and k are weight coefficients; count is the number of times the i-th data item appears in the data warehouse; freq is the update frequency of the i-th data item in the data warehouse; n i is the number of times the i-th data item is used in all downstream systems; N is the total number of times all data items are used in all downstream systems.

5. The platform data processing and management method based on big data according to claim 4 is characterized in that: The method includes collecting the usage count and update frequency data of each data item in the data warehouse and obtaining the comprehensive weight of each data item by using the comprehensive weight formula, including: collecting the name, field annotation, data description, and context information of each data item field in the data warehouse, analyzing the field annotation and semantic information by using natural language processing technology; extracting the core semantic components of the field name by using word segmentation and part-of-speech tagging technology, analyzing the field usage based on contextual relationships, and judging the semantic scope of the field in combination with the qualifiers in the data description; matching and comparing the semantic information in the field name and the field description, identifying ambiguous fields, and marking conflicting fields; automatically generating unique semantic tags for detected conflicting fields according to the semantic scope; and adjusting the usage quantity of each data item in all downstream systems according to the unique semantic tags.

6. The platform data processing and management method based on big data according to claim 5 is characterized in that: The method of obtaining the usage quantity of the data item in all downstream systems according to the field association degree includes: adjusting the usage quantity of the data item in all downstream systems using an association adjustment formula, wherein the association adjustment formula is: n i =α(n i1 +n i2 )+(1-a)*n i1 ; Where n i1 、n i2 is the number of independent uses of each field; α is the correlation factor.

7. The platform data processing and management method based on big data according to claim 1 is characterized in that: The updating of the metadata mapping rule in the data warehouse based on the unique semantic tag and forgetting the auxiliary mapping rule using a forgetting formula include: The unique semantic tag is added to the metadata mapping rule as an auxiliary mapping rule, and the unique semantic tag is compared with the remaining auxiliary mapping rules in the metadata mapping rule. When the semantic similarity is greater than a preset threshold, the remaining auxiliary mapping rules are replaced; when the semantic similarity is less than or equal to the preset threshold, it is added to the auxiliary mapping rule; and the auxiliary mapping rule is forgotten using a forgetting formula.

8. The platform data processing and management method based on big data according to claim 7 is characterized in that: The method of forgetting the auxiliary mapping rule by using a forgetting formula includes: The forgetting formula is: Where β and γ are adjustment factors, F is the calling frequency of each auxiliary mapping rule, t1 is the time difference of the most recent calling of each auxiliary mapping rule, and t0 is the standard calling time difference.

9. A platform data processing and management system based on big data, according to a platform data processing and management method based on big data according to any one of claims 1 to 8, characterized in that: include: The data collection module is used to collect the usage count, update frequency, field annotation, data description, and field context information of the fields in the data warehouse; The weight calculation module calculates the comprehensive weight according to the weight calculation formula based on the number of times the data item is used, the update frequency, and the number of times it is used in the downstream system; The cache adjustment module calculates the intermediate cache size based on the access frequency, cache hit rate and system resource utilization of the downstream system, monitors the cache performance indicators in real time, and dynamically adjusts the cache size; The semantic analysis module uses natural language processing technology to extract the core semantic components in field names and descriptions, analyze field semantic conflicts, generate unique semantic labels, and mark the associations between fields. The data adjustment module adjusts the usage quantity of data items in all downstream systems based on semantic tags and field relevance, merges highly relevant fields, and independently calculates the usage quantity of low-relevance fields; The metadata management module updates metadata mapping rules based on unique semantic tags, uses the forgetting formula to eliminate infrequently used mapping rules, and optimizes storage resources.

Images