Data transmission line optimization system and method based on big data

Abstract

The invention discloses a data transmission line optimization system and method based on big data, and belongs to the technical field of data transmission line optimization. The method comprises the following steps: extracting metadata, establishing a metadata index, and mining frequent items among the metadata of different archive information by utilizing a data mining algorithm to obtain a frequent item set; based on the frequent item set, performing content similarity analysis to obtain a similarity set; according to the logic sequence of the business process, determining an association relationship between archives generated in different links in the business to obtain an association relationship set, and constructing an archive information association network; based on an archive information association network, obtaining association closeness degrees of different archives, and dividing priorities for different archive transmission tasks; on the basis of the priority, aggregation and flow distribution are carried out on archive data flow with the same priority and consistent transmission direction, and a flow distribution scheme is obtained; and setting a cache for the nodes in the network, and performing routing optimization to obtain a transmission line optimization scheme.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission line optimization, and in particular to a data transmission line optimization system and method based on big data. Background Art

[0002] With the rapid development of information technology, the amount of archival data generated and stored in various industries has exploded. These archival information covers various fields such as enterprise operations, scientific research projects, and government affairs processing, and plays a vital role in the organization's decision-making, business process execution, and knowledge inheritance.

[0003] Existing methods usually fail to fully explore the intrinsic connections between archival data. They often treat archival data as isolated individuals for transmission, without considering the relevance of different archival information in terms of business processes, content, etc. For archival transmission tasks of different importance and urgency, existing technologies lack a scientific and reasonable way to prioritize them. In terms of data traffic processing, existing technologies cannot perform traffic aggregation and diversion according to the characteristics of archival data. Summary of the invention

[0004] The purpose of the present invention is to provide a data transmission line optimization system and method based on big data to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions: In a first aspect, the present invention provides a method for optimizing a data transmission line based on big data, the method comprising the following steps: Obtain archival information, extract metadata, establish metadata index, and use data mining algorithms to mine frequent items between metadata of different archival information to obtain frequent item sets; Based on frequent item sets, content similarity analysis is performed to obtain similarity sets; Obtain the business process of different archival information, determine the association relationship between the archives generated in different links of the business according to the logical order of the business process, and obtain the association relationship set; based on the similarity set and the association relationship set, construct the archival information association network; Based on the archive information association network, the association degree of different archives is obtained, and the priorities of different archive transmission tasks are divided; based on the priority, the archive data traffic with the same priority and consistent transmission direction is aggregated and diverted to obtain the traffic allocation plan; Based on the archive information association network, cache is set for nodes in the network, and the cache update and elimination mechanism is optimized in combination with the traffic distribution plan to perform routing optimization and obtain a transmission line optimization plan; transmission line optimization is performed based on the transmission line optimization plan.

[0006] In combination with the first aspect, in a first implementation of the first aspect of the present application, the obtaining of archive information, extracting metadata, and establishing a metadata index includes: By connecting with the existing archive management system, we can obtain archive information, standardize archive information in different formats, and perform data quality checks; Metadata standards for different types of archives are formulated in advance. For different types of archives and predefined metadata standards, corresponding automatic extraction rules are determined. The methods are as follows: for text archives, metadata is extracted using regular expressions and natural language processing; for structured database files, relevant metadata information is obtained from the database table structure; for image files, text information is extracted using optical character recognition, and metadata is extracted according to predefined rules; Select the index database and use the inverted index to create a metadata index.

[0007] In combination with the first aspect, in a second implementation of the first aspect of the present application, the frequent items between metadata of different archive information are mined using a data mining algorithm to obtain frequent item sets, including: The data mining algorithm selects the Apriori algorithm, collects metadata based on metadata indexes, integrates them into a transaction data set, sets support and confidence thresholds; scans the entire transaction data set, counts the number of occurrences of each metadata item, calculates the support of each metadata item, takes metadata items that meet the support threshold as frequent 1-item sets, and generates candidate sets of k-item sets based on frequent (k-1)-item sets, where k represents the number of metadata items in the transaction data set; scans the transaction data set again, calculates the support of the candidate sets, and selects candidate sets that meet the support threshold as frequent 2-item sets; and so on, continuously generating higher-order frequent item sets; When it is impossible to generate a k-item set that meets the support threshold, the Apriori algorithm terminates and the obtained frequent item set is the final result.

[0008] In combination with the first aspect, in a third implementation of the first aspect of the present application, the content similarity analysis is performed based on frequent item sets to obtain a similarity set, including: Based on the archive identifiers in the frequent itemsets, the corresponding archive information is extracted from the archive library, and the feature vector is constructed by using the word frequency-inverse document frequency; based on the degree of association between the frequent itemsets and different archive information, the weight of the feature vector is adjusted; the feature vectors are calculated pair by pair using the word frequency-inverse document frequency to obtain the similarity value between each pair of archives; all the calculated similarity values ​​are sorted into a set to obtain a similarity set; the similarity set is stored in the form of a matrix, and the similarity set is evaluated to check the rationality of the similarity value.

[0009] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, determining the association relationship between the archives generated in different links of the business according to the logical sequence of the business process to obtain the association relationship set includes: Based on the business process, a business process diagram is drawn to determine the types of archives that may be generated in each link, and relevant archive information is collected; the collected archive information is classified and labeled according to the source link, content theme and business type of the archive information; according to the sequence of each link in the business process diagram, the direct association relationship generated by adjacent links and the indirect association relationship generated by non-adjacent links are determined; according to business rules, the nature of the direct association relationship and the indirect association relationship is determined, and the direct association relationship and the indirect association relationship are integrated to obtain an association relationship set.

[0010] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, the step of constructing an archive information association network based on a similarity set and an association relationship set includes: Each piece of archive information is used as a node in the archive information association network, each node has a unique identifier, and the archive number is used as the unique identifier; the edge is used to represent the association relationship between archives, the edge is bidirectional, and the edge weight reflects the closeness of the association, and the edge weight is determined by the similarity set and the association relationship set; Use graph data structure to represent the archive information association network, choose adjacency matrix to store, create an n×n matrix M, where n is the total number of archives, and the first element of the matrix is Represents the weight of the edge between the first file i and the second file j. Initially, all elements are set to 0. Traverse the similarity set and the association relationship set. For each pair of files i and j with an association relationship, update the corresponding element in the adjacency matrix according to the calculated edge weight. and , is the second weight of the matrix, which represents the weight of the edge between the second file j and the first file i; The archival information association network is optimized and adjusted, including eliminating weak connections and balancing the network structure.

[0011] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, the edge weight is determined by a similarity set and an association relationship set, including: For the similarity set, based on the first file i and the second file j, obtain the similarity value of i and j. ; For the association relationship set, an association strength value is assigned according to the nature and strength of the association relationship between the archives and ; Edge weight Taking into account the similarity value and the association strength value, it is calculated by the following formula: ;in, is the weight coefficient, which is used to balance the relative importance of similarity and association in determining edge weights.

[0012] In combination with the first aspect, in a seventh implementation of the first aspect of the present application, the priority-based, aggregated and traffic-divided archival data traffic with the same priority and consistent transmission direction to obtain a traffic allocation scheme includes: By monitoring the archival data traffic in the network, based on the metadata information of the archives, the archival data traffic with the same priority and the same transmission direction is identified; in the network topology, a core node is selected as a traffic aggregation point, and the core node is specifically a core switch; at the selected traffic aggregation point, a corresponding strategy is configured to aggregate the identified archival data traffic with the same priority and the same transmission direction, and through the traffic classification and marking function of the network device, the qualified traffic is marked as the same class and guided to the same transmission path; Monitor the load status of each transmission line in real time, set the conditions for traffic diversion, and determine the backup diversion lines in the network; when the diversion conditions are triggered, according to the pre-set rules, divert part of the archive data traffic with the same priority and consistent transmission direction from the overloaded line to the backup line, and realize dynamic routing adjustment; The operation process of traffic aggregation and traffic diversion, the lines involved, the nodes and the related configuration parameters are integrated to obtain the traffic distribution plan.

[0013] In combination with the first aspect, in an eighth implementation of the first aspect of the present application, the network is associated with the archive information, a cache is set for nodes in the network, and the cache update and elimination mechanism is optimized in combination with the traffic distribution plan to perform routing optimization to obtain a transmission line optimization plan, including: Based on the archive information association network, identify key network nodes, and set up caches at key network nodes based on business needs and network topology. Allocate cache capacity to each cache node based on the estimated size of archive data and the frequency of business access to different archives. Based on the file transmission priority determined in the traffic allocation plan, the data of high-priority files in the cache are updated first. When a high-priority file is updated, the new data is synchronized to the relevant cache nodes, and the update strategy is adjusted according to the frequency of traffic changes; according to the file transmission path in the traffic allocation plan, the cache update process is optimized; Set up an elimination mechanism, the method is as follows: when the cache space is insufficient, eliminate the cache data of low-priority files, and eliminate files that have not been accessed for more than the set threshold and whose access frequency is lower than the set threshold; predict future file traffic based on the traffic distribution plan and historical traffic data, and if it is predicted that some files will not be accessed in the future, eliminate their cache data in advance; According to the cache update process and elimination mechanism, the routing strategy is adjusted to dynamically adapt to traffic changes and cache updates; the cache is set based on the archive information-associated network, the cache update and elimination mechanism are optimized in combination with the traffic distribution plan, and various measures for routing optimization are integrated to obtain a transmission line optimization plan.

[0014] In a second aspect, the present invention provides a data transmission line optimization system based on big data, comprising: Information acquisition and mining module: including: metadata extraction unit, index establishment unit and frequent item set mining unit; wherein the metadata extraction unit acquires archive information and extracts metadata, the index establishment unit establishes metadata index, and the frequent item set mining unit uses data mining algorithm to mine frequent items between metadata of different archive information to obtain frequent item sets; Similarity analysis module: including: similarity analysis unit; wherein the similarity analysis unit performs content similarity analysis based on frequent item sets to obtain a similarity set; The association network construction module includes: an association relationship determination unit and a network construction unit; wherein the association relationship determination unit obtains the business process of different archive information, determines the association relationship between archives generated in different links of the business according to the logical sequence of the business process, and obtains an association relationship set; the network construction unit constructs an archive information association network based on the similarity set and the association relationship set; Priority division and traffic management module: including: priority division unit and traffic aggregation and diversion unit; wherein the priority division unit obtains the degree of association of different archives based on the archive information association network and prioritizes different archive transmission tasks; the traffic aggregation and diversion unit aggregates and diverts the archive data traffic with the same priority and the same transmission direction based on the priority to obtain a traffic allocation plan; Cache and route optimization module: including: cache setting unit, cache strategy optimization unit, route optimization unit and solution implementation unit; wherein, the cache setting unit sets cache for nodes in the network based on the archive information association network, the cache strategy optimization unit optimizes the cache update and elimination mechanism in combination with the traffic distribution plan, the route optimization unit performs route optimization and obtains the transmission line optimization plan; the solution implementation unit optimizes the transmission line based on the transmission line optimization plan.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention obtains archival information, extracts metadata and establishes indexes, uses data mining algorithms to mine frequent item sets, and combines business processes to determine the association relationship between archives, which can comprehensively and deeply reveal the internal connection between archival data.

[0016] 2. The present invention determines the degree of association between different archives based on the archive information association network, thereby scientifically prioritizing the archive transmission tasks; ensuring that key and important archives can be transmitted first, meeting the business needs for timeliness of important information, and improving business execution efficiency.

[0017] 3. The present invention aggregates and diverts the archive data traffic of the same priority and consistent transmission direction according to the priority, and obtains a reasonable traffic allocation plan. This method can effectively integrate network traffic, avoid network congestion, improve the utilization rate of network resources, and thus improve the overall data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the steps of a data transmission line optimization method based on big data of the present invention; Figure 2 It is a system structure diagram of a data transmission line optimization system based on big data of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution. like Figure 1 A step diagram of a data transmission line optimization method based on big data is shown in a schematic diagram of a data transmission line optimization method based on big data, the method comprises the following steps: Step S100: Acquire archive information, extract metadata, establish metadata index, and use data mining algorithm to mine frequent items between metadata of different archive information to obtain frequent item sets; Specifically, by connecting with the existing archive management system, archival information is obtained, archival information in different formats is standardized, and data quality checks are performed; Metadata standards for different types of archives are formulated in advance. For different types of archives and predefined metadata standards, corresponding automatic extraction rules are determined. The methods are as follows: for text archives, metadata is extracted using regular expressions and natural language processing; for structured database files, relevant metadata information is obtained from the database table structure; for image files, text information is extracted using optical character recognition, and metadata is extracted according to predefined rules; Select the index database and use the inverted index to create a metadata index.

[0021] Further, the data mining algorithm selects the Apriori algorithm, collects metadata based on metadata indexes, integrates them into a transaction data set, sets support and confidence thresholds; scans the entire transaction data set, counts the number of occurrences of each metadata item, calculates the support of each metadata item, takes metadata items that meet the support threshold as frequent 1-item sets, and generates candidate sets of k-item sets based on frequent (k-1)-item sets, where k represents the number of metadata items in the transaction data set; scans the transaction data set again, calculates the support of the candidate sets, and selects candidate sets that meet the support threshold as frequent 2-item sets; and so on, continuously generating higher-order frequent item sets; When it is impossible to generate a k-item set that meets the support threshold, the Apriori algorithm terminates and the obtained frequent item set is the final result.

[0022] In a specific embodiment, we connected with the company's existing archive management system through an interface and obtained 1,000 archives, including 500 text-based contract documents (.docx format), 300 structured financial statements (.xlsx format, stored in a database table) and 200 project-related image files (.jpg format, including project progress charts, etc.). These archives in different formats were standardized, text-based contract documents were uniformly encoded in UTF-8 format, the data format of financial statements was standardized, and the date format was unified. At the same time, data quality checks were performed, and 50 data errors or incompleteness were found and corrected, such as typos in text-based contracts and missing values ​​in financial statements.

[0023] The pre-established metadata standards include contract number, contract type, signing date, Party A, Party B, etc. Regular expressions are used to match the contract number format (such as "HT-\d{8}" means a contract number starting with "HT-" followed by an 8-digit number), and natural language processing technology (named entity recognition) is used to extract information such as the names of Party A and Party B. In this way, 480 complete metadata were successfully extracted from 500 contract documents, with an extraction accuracy rate of 96%.

[0024] Relevant metadata information is obtained from the database table structure, such as report date, report type, department, etc. The metadata acquisition success rate for 300 financial statements is 100%, because the acquisition of structured data is relatively straightforward.

[0025] Optical character recognition technology is used to extract text information from images, and then metadata is extracted according to predefined rules. For example, for project progress images, the predefined rules are to extract information such as project name and progress percentage that appear in the image. 180 valid metadata were successfully extracted from 200 image files, with an extraction accuracy rate of 90%.

[0026] Elasticsearch was selected as the index database, and the inverted index was used to build the metadata index. All the extracted metadata items (such as contract number, contract type, report date, etc.) were used as index entries, and each entry pointed to the archive information containing the metadata. In this way, the efficiency of subsequent metadata queries was greatly improved.

[0027] Scan the entire transaction data set, count the number of occurrences of each metadata item, and calculate the support of each metadata item. For example, the metadata item "Contract Type: Sales Contract" appears 250 times in 1,000 transaction records, and its support is 250 / 1,000 = 0.25, which meets the support threshold of 0.2 and is regarded as a frequent 1-item set. After the first round of scanning, 50 frequent 1-item sets are obtained.

[0028] Generate candidate sets of 2-item sets based on frequent 1-item sets. For example, combine "Contract Type: Sales Contract" with other frequent 1-item sets to generate candidate sets such as "Contract Type: Sales Contract, Signing Date: 2023-01".

[0029] Scan the transaction data set again and calculate the support of these candidate sets. For example, the candidate set "Contract Type: Sales Contract, Signing Date: 2023-01" appears 150 times in the transaction data set, with a support of 150 / 1000=0.15, which does not meet the support threshold of 0.2 and is not selected as a frequent 2-item set. After calculation and screening, 20 frequent 2-item sets are obtained.

[0030] In this way, we continue to generate and filter higher-order frequent item sets. When generating candidate sets for 4-item sets, it is found that the support of all candidate sets does not meet the threshold, and the Apriori algorithm terminates.

[0031] The final frequent itemsets include: Frequent 1-item sets: “Contract type: Purchase contract”, “Report type: Balance sheet”, “Project name: New product research and development project”, etc., totaling 50 items.

[0032] Frequent 2-item sets: "Contract type: Purchase contract, Party A: ABC Company", "Report type: Profit and loss statement, Department: Finance Department", etc., totaling 20 items.

[0033] Frequent 3-item sets: There are 5 in total, such as "Contract type: Sales contract, Signing date: 2023-06, Party B: XYZ Company", etc.

[0034] Step S200: Based on the frequent item sets, perform content similarity analysis to obtain a similarity set. Specifically, based on the file identifiers in the frequent item sets, extract the corresponding file information from the file library, and construct a feature vector using term frequency-inverse document frequency; adjust the weights of the feature vectors based on the degree of association between the frequent item sets and different file information; perform pairwise calculations on the feature vectors using term frequency-inverse document frequency to obtain the similarity values between each pair of files; organize all the calculated similarity values into a set to obtain the similarity set; the similarity set is stored in matrix form, and the similarity set is evaluated to check the reasonableness of the similarity values.

[0035] In a specific embodiment, the frequent item sets contain 5 file identifiers, namely doc1, doc2, doc3, doc4, and doc5. Extract the corresponding 5 file information from the file library. These files are all text files, and the content is as follows: doc1: "Our company focuses on the research and development of electronic products and recently launched a new mobile phone with a high-definition screen and a powerful processor."; doc2: "Our company continues to innovate in the field of electronic products. The new mobile phone has extremely high screen resolution and excellent processor performance."; doc3: "The company's main business is clothing manufacturing, and it launched several fashionable clothing series this year."; doc4: "The research and development of electronic products requires continuous investment to improve product performance, such as the optimization of the screen and the processor."; doc5: "The company's clothing business is developing well, and this year's new clothing is welcomed by the market."

[0036] Construct a feature vector using term frequency-inverse document frequency (TF-IDF). First, perform word segmentation on all documents and remove stop words (such as "this", "of", "了", etc.) to obtain the following vocabulary list: [Electronic products, Research and development, Launch, New type, Mobile phone, High-definition, Screen, Powerful, Processor, Field, Continuous, Innovation, Resolution, Extremely high, Performance, Excellent, Clothing, Manufacturing, This year, Several, Fashionable, Series, Investment, Improvement, Optimization, Business, Development, Good, New model, Received, Market, Welcome]; Take doc1 as an example, calculate the TF-IDF value to construct the feature vector. The total number of documents is 5, "electronic products" appears once in doc1, the term frequency (TF) is 1 / the total number of words in the document, and "electronic products" appears in 3 of the 5 documents, the inverse document frequency (IDF) is ln(5 / 3), then the TF-IDF value of "electronic products" is TF*IDF. Similarly, calculate the TF-IDF value of each word in doc1 and get the feature vector of doc1. Use the same method to calculate the feature vectors of other documents.

[0037] Frequent item sets indicate that doc1, doc2, and doc4 are closely related. Based on this, the weights of the feature vectors of these three documents are adjusted. For common words such as "electronic products", "research and development", and "mobile phones", their weights in the feature vectors of doc1, doc2, and doc4 are multiplied by 1.5 to enhance them, so as to highlight the similar features of these documents under the association of frequent item sets.

[0038] TF-IDF is used to calculate the weighted feature vectors pair by pair to obtain the similarity value between each pair of files. All the calculated similarity values ​​are organized into a similarity set in the form of a matrix: ; By observing the similarity matrix, we found that the similarity values ​​between doc1, doc2, and doc4 are relatively high, which is consistent with our previous expectation of adjusting weights based on frequent item sets, because they are closely related in frequent item sets and their contents are all related to the development of electronic products. However, doc3 and doc5, as documents related to the clothing business, have lower similarity with documents related to electronic products, but higher similarity with each other. The overall similarity value distribution conforms to the business logic, indicating that the similarity set is reasonable.

[0039] Step S300: Obtain the business process of different archive information, determine the association relationship between archives generated in different links of the business according to the logical sequence of the business process, and obtain an association relationship set; based on the similarity set and the association relationship set, construct an archive information association network; Specifically, based on the business process, a business process diagram is drawn to determine the types of archives that may be generated in each link, and collect relevant archive information; the collected archive information is classified and labeled according to the source link, content theme and business type of the archive information; according to the sequence of each link in the business process diagram, the direct association relationship generated by adjacent links and the indirect association relationship generated by non-adjacent links are determined; according to business rules, the nature of the direct association relationship and the indirect association relationship is determined, and the direct association relationship and the indirect association relationship are integrated to obtain an association relationship set.

[0040] Further, each piece of archive information is used as a node in the archive information association network, each node has a unique identifier, and the archive number is used as the unique identifier; the edge is used to represent the association relationship between archives, the edge is bidirectional, and the edge weight reflects the closeness of the association, and the edge weight is determined by the similarity set and the association relationship set; Use graph data structure to represent the archive information association network, choose adjacency matrix to store, create an n×n matrix M, where n is the total number of archives, and the first element of the matrix is Represents the weight of the edge between the first file i and the second file j. Initially, all elements are set to 0. Traverse the similarity set and the association relationship set. For each pair of files i and j with an association relationship, update the corresponding element in the adjacency matrix according to the calculated edge weight. and , is the second weight of the matrix, which represents the weight of the edge between the second file j and the first file i; The archival information association network is optimized and adjusted, including eliminating weak connections and balancing the network structure.

[0041] In a specific embodiment, the business process of an e-commerce enterprise mainly includes product listing, customer ordering, order processing, product delivery, customer receipt and evaluation. Classify and annotate the archive information, and classify by source link: divide the archive into product listing, customer ordering, order processing, product delivery, customer receipt and evaluation. Classify by content theme: such as product information theme, order information theme, logistics information theme, evaluation information theme, etc. Classify by business type: Since it is an e-commerce business, it is uniformly classified as an e-commerce business.

[0042] Determine direct association: The product information file of the product listing stage is directly associated with the order file of the customer order stage, because the list of purchased products in the order file comes from the product information of the product listing stage. The order file of the customer order stage is directly associated with the order processing record file of the order processing stage, and order processing is based on orders. The order processing record file is directly associated with the delivery order file of the product delivery stage, and the product will not be delivered until the order processing is completed. The product delivery file is directly associated with the customer evaluation file of the customer receipt and evaluation stage, and the customer can only evaluate the product after receiving it.

[0043] Determine the indirect correlation: There is an indirect correlation between the product information file at the product listing stage and the customer evaluation file at the customer receipt and evaluation stage. Product information affects customer purchasing decisions, which in turn affects customer evaluations.

[0044] Determine the nature of the association relationship: Product information file and order file: dependency relationship, order depends on product information. Order file and order processing record file: causal relationship, order leads to order processing. Order processing record file and delivery order file: causal relationship, order processing results determine delivery. Delivery order file and customer evaluation file: sequential relationship, delivery first and evaluation later. Product information file and customer evaluation file: influence relationship, product information affects customer evaluation.

[0045] The association relationship set is expressed as: [(product information file, order file, dependency relationship), (order file, order processing record file, causal relationship), (order processing record file, delivery order file, causal relationship), (delivery order file, customer evaluation file, sequential relationship), (product information file, customer evaluation file, influence relationship)]; There are 5 files in total, namely, the product information file (recorded as file 1), the order file (recorded as file 2), the order processing record file (recorded as file 3), the delivery order file (recorded as file 4), and the customer evaluation file (recorded as file 5). Create a 5×5 matrix M, and all elements are set to 0 initially. Through the previous similarity analysis and the closeness of the association relationship, the following edge weights are obtained: the edge weight between file 1 and file 2 is 0.8 (because the dependency relationship is close and there may be a certain degree of similarity); the edge weight between file 2 and file 3 is 0.9 (close causal relationship); the edge weight between file 3 and file 4 is 0.9; the edge weight between file 4 and file 5 is 0.8; the edge weight between file 1 and file 5 is 0.6 (indirect association, slightly less close). Traverse the association relationship set and update the adjacency matrix M.

[0046] The weight threshold is set to 0.65, and it is found that the edge weight between file 1 and file 5 is 0.6, which is lower than the threshold, so it is eliminated from the network, that is, .

[0047] It is observed that the degree of file 3 is relatively high. In order to balance the network structure, consider adding an edge between file 1 and file 3 with a weight of 0.5.

[0048] Step S400: Based on the archive information association network, the association degree of different archives is obtained, and priorities are assigned to different archive transmission tasks; based on the priorities, the archive data traffic with the same priority and the same transmission direction is aggregated and the traffic is split to obtain a traffic allocation plan; Specifically, for a similarity set, based on the first file i and the second file j, obtain the similarity value of i and j ; For the association relationship set, an association strength value is assigned according to the nature and strength of the association relationship between the archives and ; Edge weight Taking into account the similarity value and the association strength value, it is calculated by the following formula: ;in, is the weight coefficient, which is used to balance the relative importance of similarity and association in determining edge weights.

[0049] Furthermore, by monitoring the file data traffic in the network, the file data traffic with the same priority and the same transmission direction is identified according to the metadata information of the archives; in the network topology structure, a core node is selected as a traffic aggregation point, and the core node is specifically a core switch; at the selected traffic aggregation point, a corresponding strategy is configured to aggregate the identified file data traffic with the same priority and the same transmission direction, and through the traffic classification and marking function of the network device, the qualified traffic is marked as the same class and guided to the same transmission path; Monitor the load status of each transmission line in real time, set the conditions for traffic diversion, and determine the backup diversion lines in the network; when the diversion conditions are triggered, according to the pre-set rules, divert part of the archive data traffic with the same priority and consistent transmission direction from the overloaded line to the backup line, and realize dynamic routing adjustment; The operation process of traffic aggregation and traffic diversion, the lines involved, the nodes and the related configuration parameters are integrated to obtain the traffic distribution plan.

[0050] In a specific embodiment, there are 5 files in the enterprise, which are recorded as file A, file B, file C, file D, and file E. The similarity set and association relationship set between them have been obtained, and the edge weight is calculated.

[0051] The closeness of the association of a file is measured by the sum of the edge weights connected to the file, and the closeness of the association of the file is obtained accordingly.

[0052] Prioritize the tasks based on the closeness of the association, and assign higher priorities to the file transfer tasks with a high degree of association: High priority: File D, because it has the highest degree of association (3.21). Medium priority: File B (2.75) and File A (2.64). Low priority: File C (2.54) and File E (2.28).

[0053] There is traffic from different departments transmitting archive data to the data center in the network. By monitoring the archive data traffic in the network, based on the metadata information of the archive (such as the department to which the archive belongs, the purpose of transmission, etc.), the archive data traffic with the same priority and consistent transmission direction is identified. For example, it is found that the traffic of archive A and archive B transmitted by the sales department to the data center are both medium priority and have the same transmission direction. In the network topology, the core switch is selected as the traffic aggregation point. On the core switch, the corresponding policy is configured, and the traffic of medium priority archive A and archive B sent by the sales department to the data center is marked as the same class by using the traffic classification and marking function, and is guided to the same transmission path for aggregated transmission. The traffic of archive A is 10Mbps, and the traffic of archive B is 15Mbps. The traffic after aggregation is 25Mbps.

[0054] Monitor the load status of each transmission line in real time. There are three transmission lines L1, L2, and L3, with bandwidths of 100Mbps, 100Mbps, and 50Mbps respectively. The current load of L1 is 60Mbps, the load of L2 is 30Mbps, and the load of L3 is 20Mbps. Set the traffic diversion condition as line bandwidth utilization exceeding 70%. Determine L2 and L3 as backup diversion lines. As the business progresses, the load of L1 rises to 80Mbps, exceeding the 70% threshold, triggering the diversion condition. According to the pre-set rules, 10Mbps of the aggregated 25Mbps traffic is diverted from L1 to L2 through dynamic routing adjustment. After diversion, the load of L1 becomes 70Mbps, and the load of L2 becomes 40Mbps.

[0055] Step S500: Based on the archive information associating network, setting cache for nodes in the network, optimizing cache update and elimination mechanism in combination with the traffic distribution plan, performing route optimization, and obtaining a transmission line optimization plan; and performing transmission line optimization based on the transmission line optimization plan.

[0056] Specifically, the archival information-based network is associated with setting caches for nodes in the network, optimizing cache update and elimination mechanisms in combination with the traffic distribution scheme, performing routing optimization, and obtaining a transmission line optimization scheme, including: Based on the archive information association network, identify key network nodes, and set up caches at key network nodes based on business needs and network topology. Allocate cache capacity to each cache node based on the estimated size of archive data and the frequency of business access to different archives. Based on the file transmission priority determined in the traffic allocation plan, the data of high-priority files in the cache are updated first. When a high-priority file is updated, the new data is synchronized to the relevant cache nodes, and the update strategy is adjusted according to the frequency of traffic changes; according to the file transmission path in the traffic allocation plan, the cache update process is optimized; Set up an elimination mechanism, the method is as follows: when the cache space is insufficient, eliminate the cache data of low-priority files, and eliminate files that have not been accessed for more than the set threshold and whose access frequency is lower than the set threshold; predict future file traffic based on the traffic distribution plan and historical traffic data, and if it is predicted that some files will not be accessed in the future, eliminate their cache data in advance; According to the cache update process and elimination mechanism, the routing strategy is adjusted to dynamically adapt to traffic changes and cache updates; the cache is set based on the archive information-associated network, the cache update and elimination mechanism are optimized in combination with the traffic distribution plan, and various measures for routing optimization are integrated to obtain a transmission line optimization plan.

[0057] In a specific embodiment, by analyzing the archive information association network, the core switch nodes connecting multiple important departments and the data center and the data center entry nodes are identified as key network nodes. The enterprise has three departments (sales department, R&D department, and finance department) that interact with the data center for archive data, involving five types of archives: sales report (SR), R&D document (RD), financial report (FR), project contract (PC), and market research report (MR). According to the estimated size of the archive data, the average size of the sales report is 10MB, the average size of the R&D document is 20MB, the average size of the financial report is 15MB, the average size of the project contract is 5MB, and the average size of the market research report is 8MB. Combined with the frequency of business access to different archives, the sales report is accessed 50 times a day, the R&D document is accessed 30 times a day, the financial report is accessed 40 times a day, the project contract is accessed 20 times a day, and the market research report is accessed 10 times a day.

[0058] Allocate 200MB of cache capacity to core switch nodes and 300MB of cache capacity to data center entry nodes. Allocate cache capacity based on access frequency and data size: Core switch nodes: 60MB for sales reports, 40MB for R&D documents, 50MB for financial reports, 20MB for project contracts, and 30MB for market research reports. Data center entry nodes: 80MB for sales reports, 60MB for R&D documents, 70MB for financial reports, 40MB for project contracts, and 50MB for market research reports.

[0059] According to the traffic distribution plan, the file transmission priority is: sales report (high), financial report (high), R&D document (medium), project contract (medium), market research report (low). Update based on priority: When the sales report data is updated, the new data is immediately synchronized to the cache of the core switch node and the data center entry node. For example, the sales report was updated at 10:00 one day, and the new data was synchronized to the two cache nodes at 10:01. Adjust according to the frequency of traffic changes: The sales report traffic changes frequently, set it to check every 1 hour for updates, and synchronize it in time if there are any. Financial statements are relatively stable, set it to check for updates every 4 hours. Optimize the update process based on the transmission path: If the sales department's sales report is transmitted to the data center through the core switch node, the data is updated in the core switch node cache first, and then synchronized to the data center entry node cache to reduce transmission delays.

[0060] Set up elimination mechanisms: Elimination based on priority: The core switch node cache space is insufficient, so the cache data of the market research report is eliminated first because it has the lowest priority. Elimination based on access time and frequency: Set the elimination threshold as no access time for more than 24 hours and access frequency less than 5 times a day. If the project contract is only accessed 3 times in two days and the cache space is tight, eliminate the cache data of the project contract. Elimination based on traffic forecast: According to historical traffic data, it is found that the number of visits to the market research report at the beginning of each month is extremely low. At the beginning of each month, eliminate the data in the cache of the market research report in advance to make room for other files that may be frequently accessed.

[0061] Dynamically adjust routing strategies based on cache update and elimination mechanisms. For example, when the sales report data in the core switch node cache is updated and the cache hit rate is high, adjust the route so that more relevant traffic from the sales department passes through the node to increase data acquisition speed. If a file on a certain line loses cache data due to the elimination mechanism, and it is expected that the subsequent access volume of the file may increase, adjust the route to guide part of the traffic through other nodes that have the file cache.

[0062] The measures of setting up cache at key network nodes (including node selection and capacity allocation), optimizing cache update mechanism (priority update, frequency adjustment and process optimization), setting elimination mechanism (priority elimination, time-frequency elimination and prediction elimination) and route optimization will be recorded and integrated in detail to form a transmission line optimization plan.

[0063] Before implementing the transmission line optimization solution, the average network delay was 50ms and the packet loss rate was 5%. After implementation: Reduced latency: The average network latency is reduced to 30ms, because data can be retrieved and transmitted more quickly through caching and routing optimization. Reduced packet loss rate: The packet loss rate is reduced to 2%, and the reasonable cache update and elimination mechanism and optimized routing strategy ensure the stability of data transmission.

[0064] like Figure 2 As shown in the system structure diagram of a data transmission line optimization system based on big data, the present invention provides a data transmission line optimization system based on big data, including: Information acquisition and mining module: including: metadata extraction unit, index establishment unit and frequent item set mining unit; wherein the metadata extraction unit acquires archive information and extracts metadata, the index establishment unit establishes metadata index, and the frequent item set mining unit uses data mining algorithm to mine frequent items between metadata of different archive information to obtain frequent item sets; Similarity analysis module: including: similarity analysis unit; wherein the similarity analysis unit performs content similarity analysis based on frequent item sets to obtain a similarity set; The association network construction module includes: an association relationship determination unit and a network construction unit; wherein the association relationship determination unit obtains the business process of different archive information, determines the association relationship between archives generated in different links of the business according to the logical sequence of the business process, and obtains an association relationship set; the network construction unit constructs an archive information association network based on the similarity set and the association relationship set; Priority division and traffic management module: including: priority division unit and traffic aggregation and diversion unit; wherein the priority division unit obtains the degree of association of different archives based on the archive information association network and prioritizes different archive transmission tasks; the traffic aggregation and diversion unit aggregates and diverts the archive data traffic with the same priority and the same transmission direction based on the priority to obtain a traffic allocation plan; Cache and route optimization module: including: cache setting unit, cache strategy optimization unit, route optimization unit and solution implementation unit; wherein, the cache setting unit sets cache for nodes in the network based on the archive information association network, the cache strategy optimization unit optimizes the cache update and elimination mechanism in combination with the traffic distribution plan, the route optimization unit performs route optimization and obtains the transmission line optimization plan; the solution implementation unit optimizes the transmission line based on the transmission line optimization plan.

[0065] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data transmission line optimization method based on big data, characterized in that: The method comprises the following steps: Obtain archival information, extract metadata, establish metadata index, and use data mining algorithms to mine frequent items between metadata of different archival information to obtain frequent item sets; Based on frequent item sets, content similarity analysis is performed to obtain similarity sets; Obtain the business process of different archival information, determine the association relationship between the archives generated in different links of the business according to the logical order of the business process, and obtain the association relationship set; based on the similarity set and the association relationship set, construct the archival information association network; Based on the archive information association network, the association degree of different archives is obtained, and the priorities of different archive transmission tasks are divided; based on the priority, the archive data traffic with the same priority and consistent transmission direction is aggregated and diverted to obtain the traffic allocation plan; Based on the archive information association network, cache is set for nodes in the network, and the cache update and elimination mechanism is optimized in combination with the traffic distribution plan to perform routing optimization and obtain a transmission line optimization plan; transmission line optimization is performed based on the transmission line optimization plan.

2. The data transmission line optimization method based on big data according to claim 1 is characterized in that: The obtaining of archive information, extracting metadata, and establishing metadata indexes include: By connecting with the existing archive management system, we can obtain archive information, standardize archive information in different formats, and perform data quality checks; Metadata standards for different types of archives are formulated in advance. For different types of archives and predefined metadata standards, corresponding automatic extraction rules are determined. The methods are as follows: for text archives, metadata is extracted using regular expressions and natural language processing; for structured database files, relevant metadata information is obtained from the database table structure; for image files, text information is extracted using optical character recognition, and metadata is extracted according to predefined rules; Select the index database and use the inverted index to create the metadata index.

3. The data transmission line optimization method based on big data according to claim 1 is characterized in that: The data mining algorithm is used to mine frequent items between metadata of different archive information to obtain frequent item sets, including: The data mining algorithm selects the Apriori algorithm, collects metadata based on metadata indexes, integrates them into a transaction data set, sets support and confidence thresholds; scans the entire transaction data set, counts the number of occurrences of each metadata item, calculates the support of each metadata item, takes metadata items that meet the support threshold as frequent 1-item sets, and generates candidate sets of k-item sets based on frequent (k-1)-item sets, where k represents the number of metadata items in the transaction data set; scans the transaction data set again, calculates the support of the candidate sets, and selects candidate sets that meet the support threshold as frequent 2-item sets; and so on, continuously generating higher-order frequent item sets; When it is impossible to generate a k-item set that meets the support threshold, the Apriori algorithm terminates and the obtained frequent item set is the final result.

4. The data transmission line optimization method based on big data according to claim 1 is characterized in that: The content similarity analysis is performed based on the frequent item sets to obtain a similarity set, including: Based on the archive identifiers in the frequent itemsets, the corresponding archive information is extracted from the archive library, and the feature vector is constructed by using the word frequency-inverse document frequency; based on the degree of association between the frequent itemsets and different archive information, the weight of the feature vector is adjusted; the feature vectors are calculated pair by pair using the word frequency-inverse document frequency to obtain the similarity value between each pair of archives; all the calculated similarity values ​​are sorted into a set to obtain a similarity set; the similarity set is stored in the form of a matrix, and the similarity set is evaluated to check the rationality of the similarity value.

5. The data transmission line optimization method based on big data according to claim 1 is characterized in that: According to the logical sequence of the business process, the association relationship between the files generated in different links of the business is determined to obtain an association relationship set, including: Based on the business process, a business process diagram is drawn to determine the types of archives that may be generated in each link, and relevant archive information is collected; the collected archive information is classified and labeled according to the source link, content theme and business type of the archive information; according to the sequence of each link in the business process diagram, the direct association relationship generated by adjacent links and the indirect association relationship generated by non-adjacent links are determined; according to business rules, the nature of the direct association relationship and the indirect association relationship is determined, and the direct association relationship and the indirect association relationship are integrated to obtain an association relationship set.

6. The data transmission line optimization method based on big data according to claim 1 is characterized in that: The method of constructing an archive information association network based on the similarity set and the association relationship set includes: Each piece of archive information is used as a node in the archive information association network, each node has a unique identifier, and the archive number is used as the unique identifier; the edge is used to represent the association relationship between archives, the edge is bidirectional, and the edge weight reflects the closeness of the association, and the edge weight is determined by the similarity set and the association relationship set; Use graph data structure to represent the archive information association network, choose adjacency matrix to store, create an n×n matrix M, where n is the total number of archives, and the first element of the matrix is Represents the weight of the edge between the first file i and the second file j. Initially, all elements are set to 0. Traverse the similarity set and the association relationship set. For each pair of files i and j with an association relationship, update the corresponding element in the adjacency matrix according to the calculated edge weight. and , is the second weight of the matrix, which represents the weight of the edge between the second file j and the first file i; The archival information association network is optimized and adjusted, including eliminating weak connections and balancing the network structure.

7. The method for optimizing data transmission lines based on big data according to claim 6, characterized in that: The edge weight is determined by a similarity set and an association relationship set, including: For the similarity set, based on the first file i and the second file j, obtain the similarity value of i and j. ; For the association relationship set, an association strength value is assigned according to the nature and strength of the association relationship between the archives and ; Edge weight Taking into account the similarity value and the association strength value, it is calculated by the following formula: ;in, is the weight coefficient, which is used to balance the relative importance of similarity and association in determining edge weights.

8. The method for optimizing data transmission lines based on big data according to claim 1, characterized in that: Based on the priority, the archive data traffic with the same priority and the same transmission direction is aggregated and traffic is split to obtain a traffic distribution plan, including: By monitoring the archival data traffic in the network, based on the metadata information of the archives, the archival data traffic with the same priority and the same transmission direction is identified; in the network topology, a core node is selected as a traffic aggregation point, and the core node is specifically a core switch; at the selected traffic aggregation point, a corresponding strategy is configured to aggregate the identified archival data traffic with the same priority and the same transmission direction, and through the traffic classification and marking function of the network device, the qualified traffic is marked as the same class and guided to the same transmission path; Monitor the load status of each transmission line in real time, set the conditions for traffic diversion, and determine the backup diversion lines in the network; when the diversion conditions are triggered, according to the pre-set rules, divert part of the archive data traffic with the same priority and consistent transmission direction from the overloaded line to the backup line, and realize dynamic routing adjustment; The operation process of traffic aggregation and traffic diversion, the lines involved, the nodes and the related configuration parameters are integrated to obtain the traffic distribution plan.

9. The method for optimizing data transmission lines based on big data according to claim 1, characterized in that: The method of associating the network based on the archive information, setting cache for nodes in the network, optimizing the cache update and elimination mechanism in combination with the traffic distribution plan, performing route optimization, and obtaining a transmission line optimization plan includes: Based on the archive information association network, identify key network nodes, and set up caches at key network nodes based on business needs and network topology. Allocate cache capacity to each cache node based on the estimated size of archive data and the frequency of business access to different archives. Based on the file transmission priority determined in the traffic allocation plan, the data of high-priority files in the cache are updated first. When a high-priority file is updated, the new data is synchronized to the relevant cache nodes, and the update strategy is adjusted according to the frequency of traffic changes; according to the file transmission path in the traffic allocation plan, the cache update process is optimized; Set up an elimination mechanism, the method is as follows: when the cache space is insufficient, eliminate the cache data of low-priority files, and eliminate files that have not been accessed for more than the set threshold and whose access frequency is lower than the set threshold; predict future file traffic based on the traffic distribution plan and historical traffic data, and if it is predicted that some files will not be accessed in the future, eliminate their cache data in advance; According to the cache update process and elimination mechanism, the routing strategy is adjusted to dynamically adapt to traffic changes and cache updates; the cache is set based on the archive information-associated network, the cache update and elimination mechanism are optimized in combination with the traffic distribution plan, and various measures for routing optimization are integrated to obtain a transmission line optimization plan.

10. A data transmission line optimization system based on big data, using a data transmission line optimization method based on big data according to any one of claims 1 to 9, characterized in that: include: Information acquisition and mining module: including: metadata extraction unit, index establishment unit and frequent item set mining unit; wherein the metadata extraction unit acquires archive information and extracts metadata, the index establishment unit establishes metadata index, and the frequent item set mining unit uses data mining algorithm to mine frequent items between metadata of different archive information to obtain frequent item sets; Similarity analysis module: including: similarity analysis unit; wherein the similarity analysis unit performs content similarity analysis based on frequent item sets to obtain a similarity set; The association network construction module includes: an association relationship determination unit and a network construction unit; wherein the association relationship determination unit obtains the business process of different archive information, determines the association relationship between archives generated in different links of the business according to the logical sequence of the business process, and obtains an association relationship set; the network construction unit constructs an archive information association network based on the similarity set and the association relationship set; Priority division and traffic management module: including: priority division unit and traffic aggregation and diversion unit; wherein the priority division unit obtains the degree of association of different archives based on the archive information association network and prioritizes different archive transmission tasks; the traffic aggregation and diversion unit aggregates and diverts the archive data traffic with the same priority and the same transmission direction based on the priority to obtain a traffic allocation plan; Cache and route optimization module: including: cache setting unit, cache strategy optimization unit, route optimization unit and solution implementation unit; wherein, the cache setting unit sets cache for nodes in the network based on the archive information association network, the cache strategy optimization unit optimizes the cache update and elimination mechanism in combination with the traffic distribution plan, the route optimization unit performs route optimization and obtains the transmission line optimization plan; the solution implementation unit optimizes the transmission line based on the transmission line optimization plan.

Images