Complex network big data and big model combined intelligent governance and scheduling method, device and medium

Abstract

The invention discloses a complex network big data and big model combined intelligent governance and scheduling method and device and a medium. The device comprises a main thread, an asynchronous capture thread and a webpage analysis thread; the main thread is used for creating an asynchronous capture thread and a webpage analysis thread; the asynchronous capture thread obtains the website of the webpage from the URL task queue, then completes Socket capture of the webpage, and stores the obtained webpage into a webpage result queue; the webpage analysis threads are responsible for distributing the webpage analysis threads, extracting webpages from the webpage result queue for analysis, extracting effective URLs from webpage contents and storing the effective URLs into a URL task queue; a webpage analysis thread adopts a large model for secondary training and is uniformly allocated and scheduled according to a feedback result; and constructing an optimal solution model by adopting a Pareto dominating algorithm, wherein the optimal solution model is used for calculating data in the asynchronous grabbing threads, obtaining an optimal solution thread in the asynchronous grabbing threads and marking the optimal solution thread. According to the method, the treatment and scheduling efficiency of the complex network data can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of complex network big data storage, and in particular to a complex network big data combined with a large model intelligent management and scheduling method, device and medium. Background Art

[0002] Web crawlers are very critical and basic components in search systems, responsible for data collection. The working principle of a general crawler framework is to first select a portion of high-quality web page links from Internet web pages as seed URLs, put these seed URLs into the URL queue to be crawled, and the crawler reads them from the URL queue to be crawled in turn, and resolves the URL through DNS to convert the link address into the IP address corresponding to the website server. Then the IP address and the relative path name of the web page are handed over to the web page downloader, and the web page downloader downloads the web page content to the local. In this way, on the one hand, the page content can be stored in the web page library, waiting for subsequent processing such as index establishment; on the other hand, the URL of the downloaded page is placed in the URL queue that has been crawled, and the crawled pages are recorded to prevent repeated crawling. For the newly downloaded web page, take the URL link in it, check the crawled URL queue, put the URL that has not been crawled into the URL queue to be crawled, and repeat the above steps until the URL queue to be crawled is empty, that is, the crawler system completes a complete crawling process. Faced with massive amounts of captured data, the efficiency of preprocessing data is the main problem in data collection. Summary of the invention

[0003] The purpose of the present invention is to provide a complex network big data combined with large model intelligent management and scheduling method, equipment and medium to improve the management and scheduling efficiency of complex network data.

[0004] To achieve the above purpose, the technical solution provided by the present invention is:

[0005] The first aspect of the present application provides a complex network big data combined with a large model intelligent management and scheduling method, including a main thread, an asynchronous crawling thread and a web page parsing thread;

[0006] The main thread is used to create asynchronous crawling threads and web page parsing threads; the asynchronous crawling thread obtains the web page URL from the URL task queue, then completes the Socket crawling of the web page, and stores the obtained web page in the web page result queue; the web page parsing thread is responsible for allocating web page parsing threads, extracting web pages from the web page result queue for analysis, extracting valid URLs from the web page content and storing them in the URL task queue;

[0007] The web page parsing thread adopts a large model for secondary training and is uniformly allocated and scheduled according to the feedback results; the Pareto dominance algorithm is used to build an optimal solution model for calculating the data in the asynchronous crawling thread and obtaining the optimal solution thread in the asynchronous crawling thread and marking it.

[0008] To optimize the above technical solutions, the specific measures taken also include:

[0009] The web page parsing thread adopts a large model for secondary training and is uniformly allocated and scheduled according to the feedback results, including:

[0010] After extracting the features of the data set received by the web page parsing thread, weights are divided according to the crawler's crawling sources;

[0011] Deploy the large model in the web page parsing thread, match the crawler data with the data set features, sort the training parameters with the largest number of features in each dimension, and perform distributed large model training. Send the training instructions to multiple sub-web page parsing threads for training.

[0012] After receiving the training instruction, the sub-webpage parsing thread performs the first large model training;

[0013] The big model is calculated according to the training parameters and identification field flag passed by the training instruction, and the training results of the first big model training are uploaded to the main web page parsing thread, and the calculation results of each sub-web page parsing thread are centralized; after the main web page parsing thread receives the training results of the first big model training passed by the sub-web page parsing thread, it extracts the training parameters therein for the second calculation of the big model.

[0014] Furthermore, the training parameters of the first large model training include: multidimensional feature vectors, identification fields, business weights, and various business weight categories; the training parameters of the second large model training include: multidimensional feature vectors, identification fields, business weights, and the number of business data in each weight category fed back by the large model.

[0015] Furthermore, the communication between threads is handled by the web page result queue and the URL task queue. The two message queues are created by the lightweight message queue Nanomsg and adopt the Pipeline mode.

[0016] Furthermore, the optimal solution model is constructed as follows:

[0017] Objective function: Map the n-dimensional decision space Ω to the k-dimensional target space. The objective function contains all the optimization goals of the optimization problem, and the decision space covers the entire variable space.

[0018] min f(x)=(f1(x), f2(x),…, f k(x)), x∈Ω

[0019] Among them, minf(x) represents the objective function of the optimal solution model; f k (x) represents the optimization objective function corresponding to each optimization problem; x represents the decision vector in the decision space Ω;

[0020] The Pareto dominance algorithm is used to define the relationship between the objective functions obtained by two different decision vectors x and y, and compare the advantages and disadvantages of the two decision vectors: if x dominates y, it means that x is better than y in the evaluation of all objective functions; otherwise, it means that y is better than x in the evaluation of all objective functions;

[0021] If the decision vector x is not dominated by any vector in the decision space, then x is a Pareto optimal solution; the set of all Pareto optimal solutions constitutes the Pareto optimal solution set.

[0022] Furthermore, the Pareto dominance algorithm is used to define the relationship between the objective functions obtained by two different decision vectors x and y, and compare the advantages and disadvantages of the two decision vectors, specifically:

[0023] For two decision vectors x and y, if the objective function f(x) is not greater than the objective function f(y) in any objective and is less than the objective function f(y) in at least one objective, then x dominates y, or y is dominated by x;

[0024] If x and y have a mutual dominance relationship, then x and y can be compared; if the objective function f(x) and the objective function f(y) are equal in all objectives, then x and y are equivalent; if x and y neither dominate nor are equivalent, then x and y cannot be compared;

[0025] x dominates y and is recorded as objective function f(x)くobjective function f(y).

[0026] Furthermore, after the asynchronous crawling thread completes crawling data, it is stored in the database through the transmission line; a number of optimal transmission lines are selected between the asynchronous crawling thread and the database through the node contraction method;

[0027] According to the business weight of the transmitted data, the fairness strategy model is used to analyze the weighted business data preference data as the preference index. The fairness strategy model is combined with the transmitted business weight data to perform path frequency analysis on all optimal solution routes to obtain the global optimal solution route.

[0028] Furthermore, the node contraction method contracts a node and its neighboring nodes into a new node, and the degree of contraction It is expressed as:

[0029]

[0030] Among them, d ij Represents node v i With node v j The shortest path length between nodes; n is the adjustment parameter; node v i The importance index IMC(i) is expressed as:

[0031]

[0032] in, Indicates that the node v i The cohesion of the network obtained after shrinkage.

[0033] Furthermore, the fairness strategy model is expressed as follows:

[0034]

[0035] where π t+1 represents the sorting strategy, x t represents business data, σ t Indicates the sorting position, c t is preference information, indicating whether it is frequently transmitted together with weighted service data;

[0036] According to the sorting strategy π t Get score σ t , and then obtain the preference information feedback c of the business data t , and then feedback c based on the preference information t Update the sorting strategy and get the sorting strategy π t+1 .

[0037] The second aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the complex network big data combined with large model intelligent management and scheduling method as described above is implemented.

[0038] The third aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the complex network big data combined with large model intelligent governance and scheduling method as described above.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention is composed of three types of threads: a main thread, an asynchronous crawling thread, and a web page parsing thread. The web page parsing thread is uniformly allocated and scheduled by a web page parsing thread pool in combination with a large model. The web page parsing thread is uniformly allocated and scheduled in combination with a large model so that all processes in the web page parsing process pool can operate efficiently. The web page parsing thread is trained with a distributed large model, and the computing power of the thread pool cluster is fully utilized. The sub-thread is responsible for the first round of large model training, and the main thread pool is responsible for the second round of large model training to improve the timeliness of calculation. The secondary training also makes up for the inaccurate calculation results caused by the fuzzy query of the large model.

[0041] The present invention improves transmission efficiency by merging transmission path nodes through the node contraction method to enhance node anti-destruction performance and reduce transmission delays caused by path nodes; the fairness strategy model makes up for the deficiency of the optimal solution model constructed by the Pareto algorithm that the optimal solution may be the local final rather than the global optimal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 : Schematic diagram of the process of the complex network big data combined with large model intelligent management and scheduling method of the present invention.

[0043] Figure 2 : Flow chart of asynchronous crawling thread.

[0044] Figure 3 : Flowchart of web page parsing thread. DETAILED DESCRIPTION

[0045] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0046] Some of the terms involved in the solution of the present invention are explained as follows:

[0047] A web crawler (also known as a web spider, web robot, or more commonly a web chaser in the FOAF community) is a program or script that automatically crawls the World Wide Web for information according to certain rules. Other less commonly used names include ants, automatic indexers, emulators, or worms.

[0048] The thread pool is a form of multithreading that adds tasks to a queue and then automatically starts them after the thread is created. Thread pool threads are background threads. Each thread uses the default stack size, runs at the default priority, and is in a multithreaded apartment. If a thread is idle in managed code (such as waiting for an event), the thread pool will insert another auxiliary thread to keep all processors busy. If all thread pool threads are always busy, but the queue contains pending work, the thread pool will create another auxiliary thread after a period of time, but the number of threads will never exceed the maximum. Threads exceeding the maximum can be queued, but they will not start until other threads are completed.

[0049] Istio: An open platform for connecting, managing, and protecting microservices. Istio provides an easy way to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, without making any changes to the service code. You can add Istio support services by deploying a special sidecar proxy (sidecar proxy) throughout the environment, which intercepts all network communications between microservices and is configured and managed using Istio's control plane capabilities.

[0050] In one embodiment, the present invention proposes a complex network big data combined with a large model intelligent management and scheduling method, including a main thread, an asynchronous crawling thread and a web page parsing thread, as shown in the flowchart Figure 1 As shown, including;

[0051] The main thread is used to create asynchronous crawling threads and web page parsing threads; the asynchronous crawling thread obtains the web page URL from the URL task queue, then completes the Socket crawling of the web page, and stores the obtained web page in the web page result queue, such as Figure 2 As shown; the web page parsing thread is responsible for allocating web page parsing threads, extracting web pages from the web page result queue for analysis, extracting valid URLs from the web page content and storing them in the URL task queue;

[0052] The web page parsing thread adopts the large model for secondary training and uniformly allocates and schedules according to the feedback results; the Pareto dominance algorithm is used to build the optimal solution model, which is used to calculate the data in the asynchronous crawling thread and obtain the optimal solution thread in the asynchronous crawling thread and mark it, such as Figure 3 shown.

[0053] In some implementations, the web page parsing thread uses a large model for secondary training and is uniformly allocated and scheduled according to the feedback results, including:

[0054] After extracting the features of the data set received by the web page parsing thread, weights are divided according to the crawler's crawling sources;

[0055] Deploy the large model in the web page parsing thread, match the crawler data with the data set features, sort the training parameters with the largest number of features in each dimension, and perform distributed large model training. Send the training instructions to multiple sub-web page parsing threads for training.

[0056] After receiving the training instruction, the sub-webpage parsing thread performs the first large model training;

[0057] The big model is calculated according to the training parameters and identification field flag passed by the training instruction, and the training results of the first big model training are uploaded to the main web page parsing thread, and the calculation results of each sub-web page parsing thread are centralized; after the main web page parsing thread receives the training results of the first big model training passed by the sub-web page parsing thread, it extracts the training parameters therein for the second calculation of the big model.

[0058] The training parameters of the first large model training include: multidimensional feature vector, identification field, business weight, and each business weight category; the training parameters of the second large model training include: multidimensional feature vector, identification field, business weight, and the number of business data of each weight category fed back by the large model.

[0059] In some implementations, the communication between threads is handled by the web page result queue and the URL task queue. The two message queues, the web page result queue and the URL task queue, are created by the lightweight message queue Nanomsg, using the Pipeline mode.

[0060] In some implementations, extracting features of a data set received by a web page parsing thread specifically includes: extracting distribution features of the data, including single column and paired combination features, wherein single column features include four categories of features, including row number features, type features, important features such as value statistics, distribution, sequence, anomaly, unique value, missing, and column name features; paired features include value correlation, chi-square, variance, and sharing features, column name similarity features, and finally, integrating the above features together through the Aggregate method.

[0061] In some implementations, the matters needing attention in the web page parsing thread are:

[0062] The URL in the HTML link is mainly placed in the href attribute of the 〈a〉 tag, and the URL in the tag is extracted using a state machine. When analyzing, invalid links such as href-"#", href="javascript::void(0)" and so on need to be removed. For anchor links such as href="a.html#a1", the anchor point behind needs to be removed, leaving only a.html. In this way, when deduplicating, it can be ensured that a.html#al and a.html#a2 are not regarded as two links. At the same time, the relative URL address needs to be converted into an absolute address, and the invalid part of the URL needs to be removed.

[0063] During the extraction process, URL deduplication must be completed to avoid repeatedly capturing the same URL address; during the deduplication process, the TRIE tree used in large-scale string search will be used for duplicate checking. When using the TRIE tree, thread synchronization is required; since the extraction operation is single-threaded, the insertion and search operations of the TRIE tree are also single-threaded; therefore, pthresd_mutex_t is required for synchronization and mutual exclusion operations between threads.

[0064] In some embodiments, the optimal solution model is constructed as follows:

[0065] Objective function: Map the n-dimensional decision space Ω to the k-dimensional target space. The objective function contains all the optimization goals of the optimization problem, and the decision space covers the entire variable space.

[0066] minf(x)=(f1(x), f2(x),…,f k (x)), x∈Ω

[0067] Among them, minf(x) represents the objective function of the optimal solution model; f k (x) represents the optimization objective function corresponding to each optimization problem; x represents the decision vector in the decision space Ω;

[0068] The Pareto dominance algorithm is used to define the relationship between the objective functions obtained by two different decision vectors x and y, and compare the advantages and disadvantages of the two decision vectors: if x dominates y, it means that x is better than y in the evaluation of all objective functions; otherwise, it means that y is better than x in the evaluation of all objective functions;

[0069] If the decision vector x is not dominated by any vector in the decision space, then x is a Pareto optimal solution; the set of all Pareto optimal solutions constitutes the Pareto optimal solution set;

[0070] The Pareto dominance algorithm is used to define the relationship between the objective functions obtained by two different decision vectors x and y, and compare the advantages and disadvantages between the two decision vectors, specifically:

[0071] For two decision vectors x and y, if the objective function f(x) is not greater than the objective function f(y) in any objective and is less than the objective function f(y) in at least one objective, then x dominates y, or y is dominated by x;

[0072] If x and y have a mutual dominance relationship, then x and y can be compared; if the objective function f(x) and the objective function f(y) are equal in all objectives, then x and y are equivalent; if x and y neither dominate nor are equivalent, then x and y cannot be compared;

[0073] x dominates y and is recorded as objective function f(x)くobjective function f(y).

[0074] After the asynchronous crawling thread completes crawling data, it is stored in the database through the transmission line; a number of optimal transmission lines are selected between the asynchronous crawling thread and the database through the node contraction method;

[0075] According to the business weight of the transmitted data, the fairness strategy model is used to analyze the weighted business data preference data as the preference index. The fairness strategy model is combined with the transmitted business weight data to perform path frequency analysis on all optimal solution routes to obtain the global optimal solution route.

[0076] In some embodiments, the node shrinkage method shrinks a node and its neighboring nodes into a new node, and the shrinkage degree It is expressed as:

[0077]

[0078] Among them, d ij Represents node v i With node v j The shortest path length between nodes; n is the adjustment parameter; node v i The importance index IMC(i) is expressed as:

[0079]

[0080] in, Indicates that the node v i The cohesion of the network obtained after shrinkage.

[0081] The fairness strategy model is expressed as follows:

[0082]

[0083] where π t+1represents the sorting strategy, x t represents business data, σ t Indicates the sorting position, c t is preference information, indicating whether it is frequently transmitted together with weighted service data;

[0084] According to the sorting strategy π t Get score σ t , and then obtain the preference information feedback c of the business data t , and then feedback c based on the preference information t Update the sorting strategy and get the sorting strategy π t+1 .

[0085] The second aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned complex network big data combined with large model intelligent governance and scheduling method.

[0086] The third aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above-mentioned complex network big data combined with large model intelligent governance and scheduling method.

[0087] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A complex network big data combined with big model intelligent management and scheduling method, characterized by: Including main thread, asynchronous crawling thread and web page parsing thread; The main thread is used to create asynchronous crawling threads and web page parsing threads; the asynchronous crawling thread obtains the web page URL from the URL task queue, then completes the Socket crawling of the web page, and stores the obtained web page in the web page result queue; the web page parsing thread is responsible for allocating web page parsing threads, extracting web pages from the web page result queue for analysis, extracting valid URLs from the web page content and storing them in the URL task queue; The web page parsing thread adopts a large model for secondary training and is uniformly allocated and scheduled according to the feedback results; the Pareto dominance algorithm is used to build an optimal solution model for calculating the data in the asynchronous crawling thread and obtaining the optimal solution thread in the asynchronous crawling thread and marking it.

2. The complex network big data combined with large model intelligent management and scheduling method according to claim 1 is characterized by: The web page parsing thread adopts a large model for secondary training and is uniformly allocated and scheduled according to the feedback results, including: After extracting the features of the data set received by the web page parsing thread, weights are divided according to the crawler's crawling sources; Deploy the large model in the web page parsing thread, match the crawler data with the data set features, sort the training parameters with the largest number of features in each dimension, and perform distributed large model training. Send the training instructions to multiple sub-web page parsing threads for training. After receiving the training instruction, the sub-webpage parsing thread performs the first large model training; The big model is calculated according to the training parameters and identification field flag passed by the training instruction, and the training results of the first big model training are uploaded to the main web page parsing thread, and the calculation results of each sub-web page parsing thread are centralized; after the main web page parsing thread receives the training results of the first big model training passed by the sub-web page parsing thread, it extracts the training parameters therein for the second calculation of the big model.

3. The complex network big data combined with large model intelligent management and scheduling method according to claim 2 is characterized by: The training parameters of the first large model training include: multidimensional feature vector, identification field, business weight, and each business weight category; the training parameters of the second large model training include: multidimensional feature vector, identification field, business weight, and the number of business data of each weight category fed back by the large model.

4. The complex network big data combined with large model intelligent management and scheduling method according to claim 1 is characterized by: The communication between threads is handled by the web page result queue and the URL task queue. The two message queues are created by the lightweight message queue Nanomsg and use the Pipeline mode.

5. The complex network big data combined with large model intelligent management and scheduling method according to claim 1 is characterized by: The optimal solution model is constructed as follows: Objective function: Map the n-dimensional decision space Ω to the k-dimensional target space. The objective function contains all the optimization goals of the optimization problem, and the decision space covers the entire variable space. min f(x)=(f1(x),f2(x),···,f k (x)),x∈Ω Among them, min f(x) represents the objective function of the optimal solution model; f k (x) represents the optimization objective function corresponding to each optimization problem; x represents the decision vector in the decision space Ω; The Pareto dominance algorithm is used to define the relationship between the objective functions obtained by two different decision vectors x and y, and compare the advantages and disadvantages of the two decision vectors: if x dominates y, it means that x is better than y in the evaluation of all objective functions; otherwise, it means that y is better than x in the evaluation of all objective functions; If the decision vector x is not dominated by any vector in the decision space, then x is a Pareto optimal solution; the set of all Pareto optimal solutions constitutes the Pareto optimal solution set; The Pareto dominance algorithm is used to define the relationship between the objective functions obtained by two different decision vectors x and y, and compare the advantages and disadvantages of the two decision vectors, specifically: For two decision vectors x and y, if the objective function f(x) is not greater than the objective function f(y) in any objective and is less than the objective function f(y) in at least one objective, then x dominates y, or y is dominated by x; If x and y have a mutual dominance relationship, then x and y can be compared; if the objective function f(x) and the objective function f(y) are equal in all objectives, then x and y are equivalent; if x and y neither dominate nor are equivalent, then x and y cannot be compared; x dominates y and is recorded as objective function f(x) < objective function f(y).

6. The complex network big data combined with large model intelligent management and scheduling method according to claim 1 is characterized by: After the asynchronous crawling thread completes crawling data, it is stored in the database through the transmission line; a number of optimal transmission lines are selected between the asynchronous crawling thread and the database through the node contraction method; According to the business weight of the transmitted data, the fairness strategy model is used to analyze the weighted business data preference data as the preference index. The fairness strategy model is combined with the transmitted business weight data to perform path frequency analysis on all optimal solution routes to obtain the global optimal solution route.

7. The complex network big data combined with large model intelligent management and scheduling method according to claim 6 is characterized by: The node contraction method described above contracts a node and its neighboring nodes into a new node, contracting the degree of cohesion. It is expressed as: Among them, d ij Represents node v i With node v j The shortest path length between nodes; n is the adjustment parameter; node v i The importance index IMC(i) is expressed as: in, Indicates that the node v i The cohesion of the network obtained after shrinkage.

8. The complex network big data combined with large model intelligent management and scheduling method according to claim 6 is characterized by: The fairness strategy model is expressed as follows: where π t+1 represents the sorting strategy, x t represents business data, σ t Indicates the sorting position, c t is preference information, indicating whether it is frequently transmitted together with weighted service data; According to the sorting strategy π t Get score σ t , and then obtain the preference information feedback c of the business data t , and then feedback c based on the preference information t Update the sorting strategy and get the sorting strategy π t+1 .

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for intelligent management and scheduling of complex network big data combined with large models as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the complex network big data combined with large model intelligent governance and scheduling method as described in any one of claims 1 to 8.

Images