LLM data acquisition processing method and system based on automatic capture technology

Abstract

The invention discloses an LLM data acquisition processing method and system based on an automatic capture technology, and relates to the technical field of artificial intelligence, and the method comprises the steps: distributing a data acquisition task for each crawling node in a distributed node architecture; the crawling node accesses different data sources based on the data acquisition task to obtain acquired data; after the crawling node completes a round of data acquisition, task allocation is adjusted based on the acquisition progress and the acquisition efficiency; performing first data processing on the acquired data, and performing data quality evaluation, invalid information removal and text correction on the acquired data after the first data processing; according to the method and the device, the collected data which is from different data sources and is subjected to the first data processing are integrated to obtain the integrated data, and the integrated data is subjected to the second data processing to obtain the LLM data, so that the problem that the data of various data sources for LLM cannot be efficiently collected and processed with high quality in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an LLM data acquisition and processing method and system based on automatic crawling technology. Background Art

[0002] With the rapid development of artificial intelligence (AI), large language models (LLMs) have demonstrated strong application potential in numerous fields, from intelligent customer service and text generation to intelligent writing assistance. However, the superior performance of LLMs relies on massive amounts of high-quality data, making efficient and accurate data collection and processing methods a key research topic.

[0003] In terms of data collection, traditional crawling technology is prone to network congestion, anti-crawling mechanisms and other problems when collecting large-scale data, resulting in low data collection efficiency; existing data collection methods are often limited to specific data sources or data types, resulting in a lack of diversity in the collected data and difficulty in meeting the needs of complex applications; data crawling strategies often rely on manually written rules and scripts, lack intelligent analysis and decision-making capabilities, and are difficult to adapt to the ever-changing web page structure and data format.

[0004] In terms of data cleaning, existing data cleaning methods often rely on simple rule filtering and regular expression matching, which makes it difficult to accurately identify and remove all invalid information, especially those embedded in the text, such as advertisements and navigation links; for spelling errors, grammatical errors and other problems in text data, existing correction methods often lack intelligent processing mechanisms, resulting in unsatisfactory correction effects; existing technologies often lack systematic data quality assessment methods and indicators, making it difficult to ensure data quality, which in turn affects the subsequent analysis and modeling effects.

[0005] Therefore, there is an urgent need for a method that can efficiently and high-quality collect and process LLM data. Summary of the Invention

[0006] In view of this, the present invention proposes a LLM data acquisition and processing method and system based on automated crawling technology, which can construct data acquisition tasks based on automated crawling technology, adjust task allocation using intelligent scheduling algorithms, and perform multiple rounds of processing on the collected data, including deduplication, format unification, and quality assessment, to achieve efficient and high-quality acquisition and processing of data from various data sources used for LLM.

[0007] To achieve the above object, the present invention provides the following technical solutions: A method for collecting and processing LLM data based on automated crawling technology, comprising: In response to the data collection request, construct a data collection task and assign the data collection task to each crawling node in the distributed node architecture; The crawling node accesses different data sources based on the data collection task to obtain collected data; After a crawling node completes a round of data collection, the collection efficiency and progress of each crawling node are calculated through an intelligent scheduling algorithm, and task allocation is adjusted based on the collection progress and efficiency. Performing a first data processing on the collected data, wherein the first data processing includes data deduplication, data format unification, and data missing value processing, and performing data quality assessment, invalid information removal, and text correction on the collected data after the first data processing; The collected data from different data sources that have completed the first data processing are integrated to obtain integrated data, and the integrated data are subjected to a second data processing to obtain LLM data. The second data processing includes data deduplication and data format unification.

[0008] On the basis of the above technical solution, the present invention can also be improved as follows: Optionally, in response to the data collection request, constructing a data collection task and assigning the data collection task to each crawling node includes: Task information is configured for each data collection task, and the task information includes a data source address and a collection frequency.

[0009] Optionally, the LLM data collection and processing method based on the automated crawling technology further includes: While collecting data through crawling nodes, cache prefetching technology is used to cache the collected data whose access frequency exceeds the preset threshold to reduce repeated data collection requests.

[0010] Optionally, after the crawling node completes a round of data collection, the collection progress and collection efficiency corresponding to each crawling node are calculated using an intelligent scheduling algorithm, including: The acquisition progress is calculated by formula (1); Formula (1); Where, To collect progress, is the actual average download speed, is the time elapsed after the start of collection, is the crawling node load factor, is the acquisition rate of a single node under normal no-load conditions, is the total amount of data to be collected, Update frequency for data sources; The collection efficiency is calculated by formula (2); Formula (2); Where, For collection efficiency, is the actual network bandwidth utilization, is the average data transmission stability, is the current number of active connections of the node, is the maximum number of concurrent connections for the node, is the node memory usage, is the node CPU usage, Updates the frequency factor for a data source.

[0011] Optionally, the data deduplication includes: removing duplicate data records in the knowledge data by using a deduplication algorithm to ensure the uniqueness of the data; The data format unification includes: converting knowledge data in different formats into a preset standard format; The data missing value processing includes: identifying missing values ​​in the knowledge data, and filling the missing values ​​to ensure the integrity and availability of the data.

[0012] Optionally, the data quality evaluation includes: calculating a quality evaluation value corresponding to each collected data, and filtering or marking the collected data having a quality evaluation value less than a preset threshold; The invalid information removal and text correction include: extracting text data from the filtered or marked collected data, performing semantic analysis on the text data to identify invalid information in the text data, removing the invalid information and performing correction processing.

[0013] Optionally, calculating the quality assessment value corresponding to each collected data includes: The quality assessment value is calculated by formula (3); Formula (3); Where, is the data quality assessment value, is the accuracy weight, Score the data accuracy, is the integrity weight, Score for data completeness, is the timeliness weight, For timeliness score, is the consistency weight, Score for consistency.

[0014] An LLM data acquisition and processing system based on automated crawling technology, comprising: A task assignment module, configured to construct a data collection task in response to a data collection request and assign the data collection task to each crawling node in the distributed node architecture; A data collection module, configured for the crawling node to access different data sources based on the data collection task to obtain collected data; An intelligent scheduling module is used to calculate the collection efficiency and progress of each crawling node through an intelligent scheduling algorithm after the crawling node completes a round of data collection, and adjust task allocation based on the collection progress and efficiency; A data quality assessment module is used to perform a first data processing on the collected data, wherein the first data processing includes data deduplication, data format unification, and data missing value processing, and to perform a data quality assessment on the collected data after the first data processing; Invalid information removal and text correction module, used to remove invalid information and correct text in collected data after data quality assessment; The multi-data source integration module is used to integrate the collected data from different data sources that have completed the first data processing to obtain integrated data, and is used to perform a second data processing on the integrated data to obtain LLM data. The second data processing includes data deduplication and data format unification.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0016] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0017] The present invention has the following advantages: The LLM data collection and processing method based on automated crawling technology in this invention allows each crawling node to perform its duties through precise task allocation, avoiding idle resources or competition for resources. For example, in the LLM construction that integrates multi-domain knowledge, different crawling nodes specialize in academic, news, social and other data sources, greatly improving the overall startup efficiency, shortening the initial time spent on data acquisition, and accelerating the model iteration cycle.

[0018] The LLM data collection and processing method, based on automated crawling technology, leverages the advantages of a distributed node architecture to concurrently access a vast array of heterogeneous data sources, transcending the limitations of a single data source. From authoritative professional databases to real-time social dynamics, it comprehensively covers a wide range of information types, infusing LLM with diverse knowledge nutrients and enhancing the model's breadth and depth of semantic understanding of complex real-world scenarios. This enables it to excel in tasks such as cross-domain dialogue and multi-topic creation, significantly improving the accuracy and comprehensiveness of output content.

[0019] The LLM data collection and processing method based on automated crawling technology in the present invention uses intelligent scheduling and real-time tracking to ensure node load balancing and avoid slowing down the overall situation due to lagging nodes. Efficient nodes are assigned more tasks to fully tap the potential of hardware. In the event of network fluctuations or slow response of data sources, resources can be flexibly allocated. When sudden hot events cause a surge in news source traffic, tasks can be evacuated in a timely manner to maintain a stable collection rate, reduce data lag, and ensure that fresh information is delivered to LLM, allowing the model to keep up with current hot topics and improve the timeliness of knowledge.

[0020] The LLM data collection and processing method based on automated crawling technology in the present invention deduplicates and eliminates redundancy in data, reduces storage burden, avoids duplicate data from misleading model learning rules, focuses training on key information, accelerates the training process and improves model accuracy; unified format eliminates parsing difficulties caused by differences in data sources, such as unified time format for easy time series analysis, and ensures smooth data access to the model preprocessing process; properly handles missing values ​​to complete the knowledge puzzle, prevents the model from misjudging due to information gaps, ensures data integrity, and enhances model reasoning consistency.

[0021] The LLM data collection and processing method based on automated crawling technology in the present invention conducts rigorous quality assessment to screen out high-quality data, eliminate low-quality data in advance, consolidate the foundation of model learning, and reduce the risk of accumulated errors and deviations; through semantic analysis, it accurately eliminates invalid information such as advertisements and garbled codes interspersed in the text, purifies the learning materials, and allows the model to focus on the semantic essence and avoid being misled by interference and noise; intelligent text correction repairs spelling and grammatical flaws, helps the model capture accurate semantic associations, improves the understanding of language norms, and outputs content more fluently and naturally.

[0022] The LLM data collection and processing method based on automated crawling technology in the present invention integrates the content of multiple data sources, thereby realizing the mutual complementation and verification of knowledge from different perspectives, weaving a three-dimensional knowledge network for LLM. For example, by combining cutting-edge scientific research and industrial practice data, the model has both theoretical depth and application flexibility, broadens the adaptability of application scenarios, and can integrate knowledge from various fields to provide comprehensive and sophisticated answers to complex and comprehensive queries.

[0023] The LLM data collection and processing method based on automated crawling technology in the present invention performs a secondary check to ensure that no new and duplicate content is mixed in during the integration phase, thereby continuously reducing burdens and increasing efficiency. The format is then regularized to adapt to the subsequent deep training framework, ensuring efficient data flow between various levels and modules of the complex model architecture, eliminating training interruptions and errors caused by format risks, and ensuring the model's stable and efficient absorption of knowledge, thereby improving training results and final performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] For purposes of illustration and not limitation, the present invention will now be described with reference to embodiments thereof and the accompanying drawings, in which: Figure 1 Schematic diagram of the process of LLM data acquisition and processing method based on automatic crawling technology in an embodiment of the present invention; Figure 2 Schematic diagram of the main components of the LLM data acquisition and processing system based on the automated crawling technology in an embodiment of the present invention; Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," and the like in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0027] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features thereof can be combined with each other. The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 FIG. 1 is a flow chart of the LLM data collection and processing method based on the automated crawling technology in an embodiment of the present invention. Figure 1 As shown, the LLM data collection and processing method based on the automatic crawling technology provided by the embodiment of the present invention includes the following steps S101 to S105.

[0029] S101, in response to a data collection request, construct a data collection task and assign the data collection task to each crawling node in the distributed node architecture.

[0030] Specifically, when executing this step, you first need to configure task information for each data collection task. The task information includes the data source address and collection frequency, and send the task information to the intelligent scheduling module to guide subsequent crawling task allocation.

[0031] S102: The crawling node accesses different data sources based on the data collection task to obtain collected data.

[0032] Specifically, while collecting data through crawling nodes, cache prefetching technology is used to cache the collected data whose access frequency exceeds a preset threshold to reduce repeated data collection requests.

[0033] After receiving the task instruction, the crawler node starts accessing the specified data source to collect data. The collected data will be temporarily stored in the local cache of the crawler node to wait for subsequent processing and transmission.

[0034] S103, after the crawling node completes a round of data collection, the collection efficiency and collection progress of each crawling node are calculated through the intelligent scheduling algorithm, and the task allocation is adjusted based on the collection progress and collection efficiency.

[0035] Specifically, when a crawling node completes a round of data collection, it transmits the collected data back to a central server or designated storage location via the network. Simultaneously, the intelligent scheduling module monitors the node's collection progress and efficiency, and adjusts task allocation based on this progress and efficiency.

[0036] The intelligent scheduling module dynamically calculates and assigns collection tasks to each crawling node in the distributed crawling architecture based on factors such as network conditions, crawling node load, and data source update frequency. During this process, the intelligent scheduling module will send task instructions to the crawling node, including the data source address to be collected, collection frequency, etc.

[0037] Traditional data collection often proceeds continuously according to predetermined task assignments, making it difficult to monitor the specific collection progress and efficiency of each node in real time. Even if a node experiences slow collection or inefficiency, timely adjustments and optimizations are impossible. By monitoring the collection progress and efficiency of nodes through an intelligent scheduling module, this "blind collection" state can be overcome, allowing for timely identification of the actual performance of different nodes in their collection work.

[0038] In the prior art, task allocation is often static and remains basically unchanged after being initially set, without taking into account the actual working capacity changes of each node during the collection process. The present invention dynamically adjusts task allocation based on the collection progress and collection efficiency. For example, if a node is found to have high collection efficiency and fast progress, more tasks can be appropriately allocated to it; on the contrary, for nodes with slow collection progress and low efficiency, the amount of tasks can be reduced or they can be arranged for maintenance and optimization, etc., so that task allocation is more scientific and reasonable, resource waste is avoided, and the overall data collection efficiency is improved, overcoming the shortcomings of rigid task allocation and inability to adapt to actual collection conditions.

[0039] Previous technologies lacked a mechanism for timely node status feedback, making it difficult to determine whether data collection was proceeding as expected or if intervention was needed. This approach leverages an intelligent scheduling module to monitor data in real time and adjust task allocation based on the results. This effectively establishes an effective feedback loop that responds quickly to data collection deviations, ensuring smoother and more efficient data collection.

[0040] The acquisition progress is calculated by formula (1); Formula (1); Where, To collect progress, is the actual average download speed, is the time elapsed after the start of collection, is the crawling node load factor, is the acquisition rate of a single node under normal no-load conditions, is the total amount of data to be collected, Update frequency for data sources; It converts the network download speed from Mbps to MB / s so that it can be calculated in the same way as the data volume unit. The time it takes to collect data is multiplied by the converted speed to get the theoretical amount of data downloaded during this period, but the node load must be considered. Indicates the influence coefficient of node load on the acquisition rate. The greater the load, the lower the acquisition rate. Multiply it by the acquisition rate when no load is applied. , the overall calculation is based on the network and node load. The actual amount of data collected at that time; The total amount of data to be collected is the total amount benchmark for progress calculation. Since the data source may be updated during the collection process, the update frequency Multiply by time Indicates the amount of data added by the data source during the collection period, plus the original total amount This is the total amount of data that actually needs to be collected after the update (unit: MB).

[0041] Finally, the proportion of the collection progress is obtained by dividing the numerator by the denominator , which is used to visually reflect the completion degree of the collection task.

[0042] The collection efficiency is calculated by formula (2); Formula (2); Where, For collection efficiency, is the actual network bandwidth utilization, is the average data transmission stability, is the current number of active connections of the node, is the maximum number of concurrent connections for the node, is the node memory usage, is the node CPU usage, Updates the frequency factor for a data source.

[0043] Combining the network bandwidth utilization and transmission stability, both are indispensable. An efficient network must not only have sufficient bandwidth, but also stable transmission. An unstable high-bandwidth network may frequently interrupt and retransmit during collection, affecting efficiency. The product reflects the basic support capabilities of the network.

[0044] This reflects the rationality of node connection resource utilization. Full-load operation may lead to problems such as slow response and packet loss. The appropriate number of connections can ensure smooth collection and avoid excessive concurrency or idleness.

[0045] From the perspective of node hardware resources, the higher the CPU and memory idleness, the faster the collection tasks can be processed, avoiding queuing and slow execution of tasks due to resource constraints.

[0046] It indicates the discount of data source updates on collection efficiency. Frequent updates mean that the collected data may become outdated quickly, requiring repeated collection or facing data inconsistency issues, which reduces overall efficiency.

[0047] The collection efficiency obtained by formula (2) The value ranges from 0 to 1. A higher value indicates better collection efficiency. This allows for intuitive comparison of efficiency performance at different time periods or under different collection task configurations, allowing for targeted optimization and adjustment.

[0048] S104, performing the first data processing on the collected data, which includes data deduplication, data format unification and data missing value processing, and performing data quality assessment, invalid information removal and text correction on the collected data after the first data processing.

[0049] The above-mentioned first data processing may specifically include: Deduplication algorithms are used to remove duplicate data records in knowledge data to ensure the uniqueness of the data. Duplicate data not only takes up valuable storage space, but also interferes with the accuracy of subsequent data analysis and processing. After removing duplicate records, data storage costs can be effectively reduced.

[0050] Convert knowledge data in various formats into a pre-set standard format. Data security data comes from a wide range of sources and formats, including structured database tables, semi-structured JSON files, and plain text logs. This unified format lays the foundation for standardized data processing. This allows for seamless integration of subsequent data storage, algorithm-based analysis and mining, and data sharing between modules, eliminating the need for frequent adaptation to different formats and compatibility challenges. This significantly improves overall data processing efficiency, accelerates the data cycle from acquisition to application, and ensures timely responses to data security efforts.

[0051] Identify missing values ​​in knowledge data and fill them in to ensure data integrity and availability. Missing data can be caused by a variety of reasons, such as interruptions in the collection process or restricted access to certain fields by some data sources. If missing values ​​are not addressed, many data analysis algorithms will not function properly, leading to numerous vulnerabilities in data security assessments. After filling missing values, the integrity of the dataset is restored, allowing complex analyses such as time-series-based security trend analysis and multi-dimensional association rule mining to be carried out smoothly. This ensures that security decisions comprehensively consider all factors from a data integrity perspective, maximizes data availability, and uncovers potential security risks and opportunities hidden behind complete data.

[0052] The data quality assessment module receives data returned from the distributed crawling architecture. This data may be raw data or formatted data that has undergone preliminary processing.

[0053] The pre-processed data is evaluated in real time using metrics such as classifier accuracy and perplexity. The evaluation results serve as metadata about the data quality and are passed along with the data itself to subsequent processing modules.

[0054] Calculate the quality assessment value corresponding to each collected data, and filter or mark the collected data with a quality assessment value less than a preset threshold; The quality assessment value is calculated by formula (3); Formula (3); Where, is the data quality assessment value, is the accuracy weight, Score the data accuracy, is the integrity weight, Score for data completeness, is the timeliness weight, For timeliness score, is the consistency weight, Score for consistency.

[0055] Compare the calculated quality assessment value Q with the preset threshold Z: If Q < Z, the collected data is filtered (e.g., removed from the collection result set) or marked (e.g., added with a specific low-quality mark to facilitate subsequent review and processing) accordingly.

[0056] If Q≥Z, the collected data is considered to be of good quality and can be normally included in subsequent data analysis, use and other processes.

[0057] It should be noted that the above weights 、 、 and The sum should be equal to 1 to ensure that each factor affects the quality assessment value in a reasonable proportion. At the same time, in actual applications, the values ​​of each weight and threshold should be continuously adjusted according to specific data collection scenarios, business needs, etc., so that the quality assessment is more in line with the actual situation.

[0058] The invalid information removal and text correction module receives data that has undergone data quality assessment, including the data itself and metadata about the data quality.

[0059] Natural language processing techniques are used to perform semantic analysis on text data to identify and remove invalid information. During this process, metadata provided by the data quality assessment module (such as error type and confidence level) may be used as auxiliary information to improve the accuracy of invalid information removal.

[0060] The remaining text is processed by spelling checking, grammar correction, etc.; similarly, the metadata provided by the data quality assessment module may be used as reference information to guide text correction; the corrected data is output to the subsequent processing module or stored in a specified location.

[0061] S105 , integrating the collected data from different data sources that have completed the first data processing to obtain integrated data, and performing a second data processing on the integrated data to obtain LLM data. The second data processing includes data deduplication and data format unification.

[0062] Specifically, the multi-data source integration module receives data from the invalid information removal and text correction module, which may come from different data sources and crawling nodes.

[0063] Integrate the received data, including removing duplicate and redundant data, unifying the data format, etc. During this process, data cleaning and pre-processing techniques may be required to ensure data availability and consistency.

[0064] Enhance data diversity and comprehensiveness by introducing external data sources or utilizing data fusion techniques. This helps improve the accuracy and robustness of subsequent analysis and modeling. The integrated and enhanced data is then output to subsequent analysis or modeling modules or stored in a designated database or data warehouse for LLM generation.

[0065] Figure 2 Schematic diagram of the main components of the LLM data acquisition and processing system based on the automatic crawling technology in the embodiment of the present invention. Figure 2 As shown, the LLM data acquisition and processing system 1 based on the automated crawling technology provided by an embodiment of the present invention includes a task allocation module 10, a data acquisition module 20, an intelligent scheduling module 30, a data quality assessment module 40, an invalid information removal and text correction module 50 and a multi-data source integration module 60.

[0066] The task assignment module 10 is used to construct a data collection task in response to a data collection request and assign the data collection task to each crawling node in the distributed node architecture; The data collection module 20 is used for the crawling node to access different data sources based on the data collection task to obtain the collected data; The intelligent scheduling module 30 is used to calculate the collection efficiency and collection progress of each crawling node through the intelligent scheduling algorithm after the crawling node completes a round of data collection, and adjust the task allocation based on the collection progress and collection efficiency; The data quality assessment module 40 is used to perform a first data processing on the collected data, which includes data deduplication, data format unification and data missing value processing, and perform data quality assessment on the collected data after the first data processing; The invalid information removal and text correction module 50 is used to remove invalid information and correct text on the collected data after data quality assessment; The multi-data source integration module 60 is used to integrate the collected data from different data sources that have completed the first data processing to obtain integrated data, and perform a second data processing on the integrated data to obtain LLM data. The second data processing includes data deduplication and data format unification.

[0067] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3 As shown, the electronic device 70 includes: a processor 701 (processor), a memory 702 (memory) and a bus 703; The processor 701 and the memory 702 communicate with each other via the bus 703. The processor 701 is used to call the program instructions in the memory 702 to execute the methods provided by the above-mentioned method embodiments, so as to execute the methods provided by the implementation methods of the present invention.

[0068] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the method provided by the embodiment of the present invention.

[0069] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various storage media that can store program codes.

[0070] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A LLM data collection and processing method based on automated crawling technology, characterized in that: include: In response to the data collection request, construct a data collection task and assign the data collection task to each crawling node in the distributed node architecture; The crawling node accesses different data sources based on the data collection task to obtain collected data; When a crawling node completes a round of data collection, the collection efficiency and progress of each crawling node are calculated through an intelligent scheduling algorithm, and task allocation is adjusted based on the collection progress and efficiency; Performing a first data processing on the collected data, wherein the first data processing includes data deduplication, data format unification and data missing value processing, and performing data quality assessment, invalid information removal and text correction on the collected data after the first data processing; The collected data from different data sources that have completed the first data processing are integrated to obtain integrated data, and the integrated data are subjected to a second data processing to obtain LLM data. The second data processing includes data deduplication and data format unification.

2. The LLM data acquisition and processing method based on the automated crawling technology according to claim 1 is characterized in that: In response to the data collection request, constructing a data collection task and assigning a data collection task to each crawling node includes: Task information is configured for each data collection task, wherein the task information includes a data source address and a collection frequency.

3. The LLM data acquisition and processing method based on the automated crawling technology according to claim 1 is characterized in that: The LLM data collection and processing method based on the automated crawling technology also includes: While collecting data through crawling nodes, cache pre-fetching technology is used to cache the collected data whose access frequency exceeds the preset threshold to reduce repeated data collection requests.

4. The LLM data acquisition and processing method based on the automated crawling technology according to claim 1 is characterized in that: After the crawling node completes a round of data collection, the collection progress and collection efficiency corresponding to each crawling node are calculated through the intelligent scheduling algorithm, including: The collection progress is calculated by formula (1); Formula (1): In the formula, To collect progress, is the actual average download speed, is the time elapsed after the start of collection, is the crawling node load factor, is the acquisition rate of a single node under normal no-load conditions, is the total amount of data to be collected, Update frequency for data sources; The collection efficiency is calculated by formula (2); Formula (2): In the formula, For collection efficiency, is the actual network bandwidth utilization, is the average data transmission stability, is the number of active connections of the node. is the maximum number of concurrent connections for the node, is the node memory usage, is the node CPU usage, Updates the frequency factor for a data source.

5. The LLM data acquisition and processing method based on the automated crawling technology according to claim 1 is characterized in that: The data deduplication includes: removing duplicate data records in the knowledge data by using a deduplication algorithm to ensure the uniqueness of the data; The data format unification includes: converting knowledge data in different formats into a preset standard format; The data missing value processing includes: identifying missing values ​​in the knowledge data, and filling the missing values ​​to ensure the integrity and availability of the data.

6. The LLM data acquisition and processing method based on the automated crawling technology according to claim 1 is characterized in that: The data quality evaluation includes: calculating the quality evaluation value corresponding to each collected data, filtering or marking the collected data whose quality evaluation value is less than a preset threshold; The invalid information removal and text correction include: extracting text data from the filtered or marked collected data, performing semantic analysis on the text data to identify invalid information in the text data, removing the invalid information and performing correction processing.

7. The LLM data acquisition and processing method based on the automated crawling technology according to claim 1 is characterized in that: The calculating of the quality assessment value corresponding to each collected data includes: The quality assessment value is calculated by formula (3); Formula (3): In the formula, is the data quality assessment value, is the accuracy weight, Score the data for accuracy, is the integrity weight, is the data completeness score, is the timeliness weight, For timeliness score, is the consistency weight, Score for consistency.

8. A system for LLM data collection and processing based on automated crawling technology, characterized in that: include: A task allocation module, for constructing a data collection task in response to a data collection request, and allocating the data collection task to each crawling node in the distributed node architecture; A data collection module, used for the crawling node to access different data sources based on the data collection task to obtain collected data; An intelligent scheduling module is used to calculate the collection efficiency and progress of each crawling node through an intelligent scheduling algorithm after the crawling node completes a round of data collection, and adjust task allocation based on the collection progress and collection efficiency; A data quality assessment module is used to perform a first data processing on the collected data, wherein the first data processing includes data deduplication, data format unification and data missing value processing, and to perform data quality assessment on the collected data after the first data processing; The invalid information removal and text correction module is used to remove invalid information and correct text for the collected data after data quality assessment; The multi-data source integration module is used to integrate the collected data from different data sources that have completed the first data processing to obtain integrated data, and perform a second data processing on the integrated data to obtain LLM data. The second data processing includes data deduplication and data format unification.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images