Automatic construction system of local knowledge base based on multi-source collection and distributed calculation

Abstract

This invention provides an automated local knowledge base construction system based on multi-source data acquisition and distributed computing, belonging to the field of computer system technology. The system includes a multi-source data acquisition unit and a distributed data processing and parsing module. The multi-source data acquisition unit handles the access, acquisition, transmission, and incremental updates of multi-source data, generating incremental change records. The distributed data processing and parsing module includes multiple distributed nodes used to process and parse the data files collected by the multi-source data acquisition unit based on the incremental change records, and establish a local knowledge base. This system can automate the construction of a local knowledge base, improve construction efficiency, shorten the construction cycle, ensure the freshness of knowledge data, and improve data quality.

Description

Technical Field

[0001] This invention relates to the field of computer system technology, and in particular to an automated local knowledge base construction system based on multi-source acquisition and distributed computing. Background Technology

[0002] With the rapid development of the artificial intelligence industry, next-generation general-purpose AI models, represented by DeepSeek, are increasingly being applied across various industries due to their powerful natural language processing and logical reasoning capabilities. Sectors such as finance, healthcare, manufacturing, and government have already integrated DeepSeek, building and integrating their own localized knowledge bases to implement core application scenarios such as intelligent customer service, decision support, and process automation.

[0003] The market size of smart application projects based on large language models is growing. However, current industry practices have revealed significant technical bottlenecks. The local knowledge base supporting the intelligentization of applications suffers from core pain points such as "long construction cycle, low parsing efficiency, lagging data updates, and low data quality".

[0004] Therefore, it is necessary to establish a more general and efficient technical methodology based on distributed computing to automate the construction of local knowledge bases, improve the efficiency of knowledge base construction, shorten the knowledge base construction cycle, ensure the freshness of knowledge data, and improve data quality. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing an automated local knowledge base construction system based on multi-source acquisition and distributed computing, thereby realizing the automated construction of local knowledge bases.

[0006] To address the aforementioned technical problems, the technical solution adopted by this invention is: an automated local knowledge base construction system based on multi-source data acquisition and distributed computing, comprising a multi-source data collector and a distributed data processing and parsing module; the multi-source data collector realizes the access, data acquisition, data transmission, and incremental updates of multi-source data, generating incremental change records; the distributed data processing and parsing module includes multiple distributed nodes, used to perform distributed processing and parsing of the data collected by the multi-source data collector based on the incremental change records, and establish a local knowledge base;

[0007] The multi-source data collector includes a data acquisition probe module and a data transmission module; the data acquisition probe module is used to realize multi-source data access, data snapshot generation, data snapshot comparison, incremental change record generation, and data transmission submission; the data transmission module is used to realize incremental change record transmission, data file transmission, and expired file cleanup;

[0008] The specific method by which the distributed data processing and parsing module performs distributed processing and parsing of data files based on incremental change records is as follows:

[0009] When the incremental change record records a file change type of "new", the new file is sequentially subjected to distributed block operation, distributed classification operation, distributed vectorization operation, and knowledge base writing operation.

[0010] When the incremental change record records a file change type of deletion, the related files, block content, and block result vector data in the knowledge base, persistent media, etc. are cleaned up in sequence.

[0011] Furthermore, the data acquisition probe module enables multi-source data access by defining the data source structure and acquisition strategy, and supports data source categories including: file-based, database-based, and API-based.

[0012] Furthermore, the data source structure defined by the data acquisition probe module is as follows:<id,name,type,sconfig,dconfig> Where: id is the identifier of the data source; name is the name of the data source, which can be configured to be any string; type is the type of data source, including file type, database type, and API type.

[0013] sconfig configures the data collection strategy for the data source side, and is used to configure the collection strategy of the multi-source data collector. The format is: <srcs[src<datakey,interval,option> The data structure is defined as follows: `srcs` is a list of data source collection strategies; `src` is a specific element that identifies the collection strategy for a particular data source; `datakey` is the key parameter for data retrieval from the data source. When the data source is a file, `datakey` is the specific file storage path; when the data source is a database, `datakey` is the specific SQL statement for retrieving data; when the data source is an API, `datakey` is the corresponding API address; `interval` is the data collection time interval; and `option` is an optional configuration used to configure different parameters required when retrieving data using `datakey`.

[0014] dconfig specifies the data transmission module's configuration for receiving collected data. The format is:<dtype, path,option> dtype is the method by which the data transmission module receives files, with options for "HTTP service" and "object storage"; path is the path address for the data transmission module to receive files, including the address for receiving incremental change records and the address for receiving data files; option is an optional configuration.

[0015] Furthermore, the data acquisition probe module periodically generates data snapshots according to the strategy defined by the data source structure. The data snapshot structure is as follows: <snapshots[snapshot<snapid,createtime,rootpath,files<path,size,updatetime, hash> The data snapshots list is defined as follows: `snapshots` represents the list of data snapshots; `snapshot` represents the data snapshot element, specifically including the snapshot's `snapid`, creation time (`createtime`), rootpath (the directory corresponding to the snapshot), and `files` (the list of snapshot files). The `files` list includes information about multiple database snapshot files, specifically the snapshot file path, size, updatetime, and hash. When the data source is a file, the snapshot is generated directly based on the file structure. When the data source is a database, the acquired data is first saved to a file, and then a snapshot is generated based on the file structure. When the data source is an API, the data obtained through the API interface is first saved to a file, and then a snapshot is generated based on the file structure.

[0016] Furthermore, the data acquisition probe module, after periodically generating data snapshots according to the strategy defined by the data source structure, compares the latest generated data snapshot with the most recent saved data snapshot, and generates incremental change records in units of files; the specific format of the incremental change record is as follows:<optype,optime,filepath,filetype,filesize,filehash,fileupdatetime> Where optype is the incremental change type, including addition and deletion; optimum is the incremental change time; filepath is the original file path, filetype is the file type, filesize is the file size, filehash is the hash of the data snapshot file, and fileupdatetime is the latest update time of the file; when the data snapshot comparison finds that the file has been added or deleted, a corresponding incremental change record is generated; when the snapshot comparison finds that the file has been updated, an incremental change record of type addition is first generated for the updated file, and then an incremental change record of type deletion is generated for the original file before the update.

[0017] Furthermore, after generating incremental change records, the data acquisition probe module, for each file in the incremental change records, when the file is a new file, submits the new file to the distributed data processing and parsing module through the data transmission module. After the file upload is successful, the incremental change record is then submitted to the distributed data processing and parsing module through the data transmission module. When the file is a deleted file, only the incremental change record is submitted to the distributed data processing and parsing module through the data transmission module. If the file upload fails, the file upload is skipped, and the data snapshot comparison, file generation, and file upload are re-performed in the next round. After the data transmission module completes the file and incremental change record submission operations, it regenerates and saves the data snapshot for the successfully uploaded files and incremental change records.

[0018] Furthermore, the data transmission module is configured with a persistent medium to receive incremental change records uploaded by the multi-source data collector, and the incremental change records are stored in the persistent medium in the form of files;

[0019] It receives data files uploaded by multi-source data collectors and stores them in persistent media;

[0020] Based on the incremental change log, files involved in deletion are periodically cleaned up from the persistent media.

[0021] Furthermore, the distributed chunking operation performs chunking tasks in a distributed manner for newly added files. Based on the Spark framework, it is configured with dynamic resource allocation and fault tolerance mechanisms, specifically including:

[0022] Task division involves reading incremental change records uploaded by multi-source data collectors through the data transmission module, and dividing all files to be processed into sub-tasks using a custom file division strategy to ensure that the total file size processed by each sub-task is the same.

[0023] Distributed processing: each subtask parses the file one by one to generate text parsing results, divides the parsing results into blocks, and generates the block content of the file.

[0024] Results aggregation involves collecting the generated content blocks and writing them to persistent media.

[0025] Furthermore, the distributed classification operation uses a deep learning algorithm to classify the block content in the persistent medium and persistently stores the classification results.

[0026] Furthermore, the distributed vectorization operation determines the corresponding embedding model for the classification results obtained from the distributed classification; all block content with the same classification result is processed using a distributed task, and there are multiple distributed tasks. Each distributed task is processed as follows:

[0027] Divide the data blocks to be processed into subtasks according to the number of blocks; each subtask processes the same number of blocks.

[0028] For each distributed node, initialize the embedding model corresponding to the classification result, and set the embedding model to use the CPU for block content vectorization;

[0029] The subtasks are assigned to distributed nodes to run, and the embedded model is used to parse the block content and generate block result vectors in turn.

[0030] Write the block result vector parsed from the embedded model into the persistent medium.

[0031] Furthermore, the knowledge base writing operation is as follows: the new file, block content, and block result vector are written into the knowledge base.

[0032] Furthermore, when the incremental change record's change type is deletion, the specific method for sequentially cleaning up the knowledge base, related files in the persistent media, block content, and block result vector data is as follows:

[0033] Based on the acquired incremental change records, the file to be deleted, the content of the block to be deleted, and the result vector of the block to be deleted are determined in sequence, and the relevant data in the knowledge base are deleted; after the knowledge base deletion operation is successful, the block result vector, block, and content in the persistent medium are deleted in sequence.

[0034] The beneficial effects of adopting the above technical solution are as follows: The local knowledge base automated construction system based on multi-source acquisition and distributed computing provided by the present invention can realize the automated construction of local knowledge bases, improve the efficiency of knowledge base construction, shorten the knowledge base construction cycle, ensure the freshness of knowledge data, and improve data quality. Attached Figure Description

[0035] Figure 1 A structural block diagram of a local knowledge base automated construction system based on multi-source acquisition and distributed computing provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the working principle of a multi-source data acquisition device provided in an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram illustrating the working principle of the distributed data processing and parsing module provided in an embodiment of the present invention. Detailed Implementation

[0038] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0039] In this embodiment, a local knowledge base automated construction system based on multi-source acquisition and distributed computing, such as... Figure 1 As shown, it includes a multi-source data collector and a distributed data processing and parsing module; the multi-source data collector realizes the access, data collection, data transmission and incremental update of multi-source data, and generates incremental change records; the distributed data processing and parsing module includes multiple distributed nodes, which are used to perform distributed processing and parsing of data files based on incremental change records on the data collected by the multi-source data collector, and establish a local knowledge base.

[0040] In this embodiment, the multi-source data acquisition device is as follows: Figure 2 As shown, it includes a data acquisition probe module and a data transmission module; the data acquisition probe module is used to realize multi-source data access, data snapshot generation, data snapshot comparison, incremental change record generation, and data transmission submission; the data transmission module is used to realize incremental change record transmission, data file transmission, and expired file cleanup.

[0041] The data acquisition probe module enables multi-source data access by defining the data source structure and acquisition strategy. The data source categories supported include: file, database, and API.

[0042] The data source structure defined by the data acquisition probe module is as follows:<id,name,type,sconfig,dconfig> Where: id is the identifier of the data source; name is the name of the data source, which can be configured to be any string; type is the data source type, including three types: file, database, and API.

[0043] sconfig configures the data collection strategy for the data source side, and is used to configure the collection strategy of the multi-source data collector. The format is: <srcs[src<datakey,interval,option> The `<database>` element is defined as follows: `srcs` is a list of data source collection strategies; `src` is a specific element that identifies the collection strategy of a particular data source; `datakey` is the key parameter for data retrieval from the data source. When the data source is a file, `datakey` is the specific file storage path; when the data source is a database, `datakey` is the specific SQL statement for retrieving data; when the data source is an API, `datakey` is the corresponding API interface address; `interval` is the data collection time interval; and `option` is an optional configuration used to configure different parameters required when retrieving data using `datakey`, including authentication methods and request / response format definitions.

[0044] dconfig specifies the data transmission module's configuration for receiving collected data. The format is:<dtype, path,option> dtype is the method by which the data transmission module receives files, with two possible values: "HTTP Service" and "Object Storage (MinIO)"; path is the path address for the data transmission module to receive files, including the address for receiving incremental change records and the address for receiving data files; option is an optional configuration, including information such as the authentication method of the data transmission module.

[0045] In this embodiment, the data acquisition probe module periodically generates data snapshots according to the strategy defined by the data source structure. The data snapshot structure is as follows: <snapshots[snapshot<snapid,createtime,rootpath,files<path,size,updatetime, hash> The data snapshots list is defined as follows: `snapshots` represents the list of data snapshots; `snapshot` represents the data snapshot element, specifically including the snapshot's `snapid`, creation time (`createtime`), rootpath (the directory corresponding to the snapshot), and `files` (the list of snapshot files). The `files` list includes information about multiple database snapshot files, specifically the snapshot file path, size, updatetime, and hash. When the data source is a file, the snapshot is generated directly based on the file structure. When the data source is a database, the acquired data is first saved to a file, and then a snapshot is generated based on the file structure. When the data source is an API, the data obtained through the API interface is first saved to a file, and then a snapshot is generated based on the file structure.

[0046] The data acquisition probe module periodically generates data snapshots according to the strategy defined by the data source structure. It then compares the newly generated data snapshot with the most recent saved data snapshot and generates incremental change records on a file-by-file basis. The specific format of the incremental change record is as follows:<optype,optime,filepath,filetype, filesize,filehash,fileupdatetime> Where optype is the incremental change type, including addition and deletion; optimum is the incremental change time; filepath is the original file path, filetype is the file type, filesize is the file size, filehash is the hash of the data snapshot file, and fileupdatetime is the latest update time of the file; when the data snapshot comparison finds that the file has been added or deleted, a corresponding incremental change record is generated; when the snapshot comparison finds that the file has been updated, an incremental change record of type addition is first generated for the updated file, and then an incremental change record of type deletion is generated for the original file before the update.

[0047] After generating incremental change records, the data acquisition probe module, for each file in the incremental change records, submits the newly added file to the distributed data processing and parsing module via the data transmission module when the file is a new file. After the file upload is successful, the incremental change record is then submitted to the distributed data processing and parsing module via the data transmission module. When the file is a deleted file, only the incremental change record is submitted to the distributed data processing and parsing module via the data transmission module. If the file upload fails, the current file upload is skipped, and the data snapshot comparison, file generation, and file upload are re-performed in the next round. After the data transmission module completes the file and incremental change record submission operations, it regenerates and saves the data snapshot for the successfully uploaded files and incremental change records.

[0048] The data transmission module is equipped with a persistent medium, receives incremental change records uploaded by the multi-source data collector, and stores the incremental change records in the persistent medium in the form of files; receives data files uploaded by the multi-source data collector and stores them in the persistent medium; and periodically cleans up files involved in deletion from the persistent medium based on the incremental change records.

[0049] The distributed data processing and parsing module performs distributed processing and parsing of the data file based on the incremental change records, such as... Figure 3 As shown, the specific method is as follows:

[0050] When the incremental change record records a file change type of "new", the new file is sequentially subjected to distributed block operation, distributed classification operation, distributed vectorization operation, and knowledge base writing operation.

[0051] When the incremental change record records a file change type of deletion, the related files, block content, and block result vector data in the knowledge base, persistent media, etc. are cleaned up in sequence.

[0052] The distributed chunking operation executes chunking tasks in a distributed manner for newly added files. Based on the Spark framework, it is configured with dynamic resource allocation and fault tolerance mechanisms, specifically including:

[0053] Task division involves reading incremental change records uploaded by multi-source data collectors through the data transmission module, and dividing all files to be processed into sub-tasks using a custom file division strategy to ensure that the total file size processed by each sub-task is basically the same.

[0054] Distributed processing: each subtask parses the file one by one to generate text parsing results, divides the parsing results into blocks, and generates the block content of the file.

[0055] Results aggregation involves collecting the generated content blocks and writing them to persistent media.

[0056] The distributed classification operation uses a deep learning algorithm to classify the block content in the persistent medium and persistently stores the classification results.

[0057] Distributed vectorization operations determine the corresponding embedding model for the classification results obtained from distributed classification. All block content with the same classification result is processed using a single distributed task. Multiple distributed tasks are used, and each task is processed as follows:

[0058] Divide the data blocks to be processed into subtasks according to the number of blocks; each subtask processes the same number of blocks.

[0059] For each distributed node, initialize the embedding model corresponding to the classification result, and set the embedding model to use the CPU for block content vectorization;

[0060] The subtasks are assigned to distributed nodes to run, and the embedded model is used to parse the block content and generate block result vectors in turn.

[0061] Write the block result vector parsed from the embedded model into the persistent medium.

[0062] The knowledge base writing operation is as follows: adding new files, block content, and block result vectors to the knowledge base.

[0063] When the incremental change record's change type is deletion, the specific method for sequentially cleaning up the knowledge base, related files, block content, and block result vector data in the persistent media is as follows:

[0064] Based on the acquired incremental change records, the file to be deleted, the content of the block to be deleted, and the result vector of the block to be deleted are determined in sequence, and the relevant data in the knowledge base are deleted; after the knowledge base deletion operation is successful, the block result vector, block, and content in the persistent medium are deleted in sequence.

[0065] In this embodiment, the multi-source data collector supports the definition and collection of data source structures of file type, database type, and API type. The following is an example of the definition of a file type data source structure:

[0066] {

[0067] "id": 1,

[0068] "name": "Formula, Policy and System Data Collection",

[0069] "type": "file",

[0070] "sconfig": [

[0071] {

[0072] "datakey": " / data1 / rule1",

[0073] "interval": "30min"

[0074] },

[0075] {

[0076] "datakey": " / data1 / rule2",

[0077] "interval": "45min"

[0078] }

[0079] ],

[0080] "dconfig": {

[0081] "type": "httpservice",

[0082] "path": "http: / / host1:8080 / rule",

[0083] "option": {

[0084] "user": "user1",

[0085] "password": "password

[0086] }

[0087] }

[0088] };

[0089] In the above data source structure definition, the multi-source data collector reads files in / data1 / rule1 and / data1 / rule2, and periodically reads the directory to generate data snapshots according to the configured policy. The specific data snapshot files are shown below:

[0090] {

[0091] "id": "id1",

[0092] "createtime": "2025-05-14T10:30:45Z",

[0093] "rootpath": " / data1 / rule1",

[0094] "files": [

[0095] {

[0096] "path": "rule1.pdf",

[0097] "size": 1024567,

[0098] "lastModified": "2023-05-10T14:25:30Z",

[0099] "hash":"sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"

[0100] },

[0101] {

[0102] "path": "rule2.pdf",

[0103] "size": 2048152,

[0104] "lastModified": "2023-05-12T09:15:22Z",

[0105] "hash": "md5:098f6bcd4621d373cade4e832627b4f6"

[0106] } ]

[0108] };

[0109] After a data snapshot is generated, it is compared with the most recent saved data snapshot, and incremental change records are generated on a file-by-file basis, as shown below: [

[0111] {

[0112] "optype":"add",

[0113] "optime":"2025-05-14T10:30:45Z",

[0114] "filepath":" / data1 / rule1 / rule1.pdf",

[0115] "filetype":"pdf",

[0116] "filesize":1024567,

[0117] "filehash":"sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",

[0118] "fileupdatetime":"2023-05-10T14:25:30Z"

[0119] },

[0120] {

[0121] "optype":"add",

[0122] "optime":"2025-05-14T10:30:45Z",

[0123] "filepath":" / data1 / rule1 / rule2.pdf",

[0124] "filetype":"pdf",

[0125] "filesize":2048152,

[0126] "filehash":"md5:098f6bcd4621d373cade4e832627b4f6",

[0127] "fileupdatetime":"2023-05-12T09:15:22Z"

[0128] } ];

[0130] After generating incremental change records, the incremental change records and change files are uploaded through the data transmission module. At the same time, the upload status of the file parsing results generated by the distributed data processing and parsing module is monitored periodically. When the file is in the uploaded status, all parsing results corresponding to that file are deleted.

[0131] The distributed data processing and parsing module is used to capture changes periodically, perform distributed computing and parsing on incremental change files based on incremental change records, and feed the results back to the local knowledge base.

[0132] In this embodiment, developing a distributed data processing and parsing module is an optional implementation for when the change type is "new": it uses the Spark framework for distributed processing, configuring dynamic resource allocation and fault tolerance mechanisms. The entire processing flow includes: distributed partitioning, distributed classification, distributed vectorization, and knowledge base updating. When the number of incremental changes reaches a certain threshold (either time or number of files reaches the threshold), the Spark distributed processing program starts:

[0133] 1. Distributed chunking: When the distributed nodes start, they initialize the deep learning models and various parsers used. Supported formats include PDF, DOCX, JSON, Markdown, and HTML. Task partitioning strategy: Subtasks are divided according to file size; in this embodiment, the threshold is set to 512MB. During subtask processing, the file needs to be chunked, with a chunk size threshold of 512B. When processing adjacent chunks, if the last sentence of the previous chunk is truncated, that sentence will be added to the next chunk. Simultaneously, chunk position information is recorded, generating the following persistent storage medium directory structure:

[0134] file_id 1 /

[0135] / bulk_id 1 / block content

[0136] Location information recording

[0137] / bulk_id 2 / block content

[0138] Location information recording

[0139] / bulk_id 3 / block content

[0140] Location information recording;

[0141] In this embodiment, the position information of the generated block is recorded as follows, where start represents the starting position in the document, and end represents the ending position in the document. file_id represents the file ID, and bulk_id represents the block ID;

[0142] {

[0143] "start": {

[0144] "x":0,

[0145] "y":0

[0146] },

[0147] "end": {

[0148] "x":100,

[0149] "y":80

[0150] },

[0151] "file_id": "a1",

[0152] "bulk_id":"1"

[0153] };

[0154] 2. Distributed classification: A deep learning model is used to analyze the block content and classify it by file based on the analysis results, outputting the classification results. The directory structure stored on persistent storage media is as follows:

[0155] file_id 1 / classification result 1;

[0156] file_id 2 / classification result 2;

[0157] 3. Distributed vectorization: Blocks with the same classification result will use the same embedding model and be processed by the same distributed vectorization task. Therefore, when a distributed node starts, it initializes the corresponding embedding model according to the classification result, reads the block content, vectorizes it, and generates a block result vector. The directory structure of the persistent storage medium for storing the block result vector is shown below:

[0158] file_id 1 /

[0159] / bulk_id 1 / block result vector

[0160] / bulk_id 2 / block result vector

[0161] / bulk_id 3 / block result vector;

[0162] After distributed block partitioning, distributed classification, and distributed vectorization, the final parsing result is stored in the following persistent storage medium directory structure, as shown below. The upload identifier is the last added part, including three states: not uploaded, uploaded, and upload failed.

[0163] file_id 1 / Category result 1

[0164] / Upload ID

[0165] / bulk_id 1 / block content

[0166] Location information recording

[0167] / block result vector

[0168] / bulk_id 2 / block content

[0169] Location information recording

[0170] / block result vector

[0171] / bulk_id 3 / block content

[0172] Location information recording

[0173] / block result vector;

[0174] In this embodiment, the knowledge base can adopt RAGFlow, and Minio and ES can be used as storage media for incrementally changed files, block content and block result vectors. For files with upload status of not uploaded or upload failure, they are converted according to the data format requirements of RAGFlow knowledge base and submitted to the local knowledge base for storage, while the upload status is updated.

[0175] In this embodiment, the development of a distributed data processing parsing module is an optional implementation method for changes of type deletion: based on the file_id to be deleted, the file, block content, and block result vector in the RAGFlow local knowledge base are directly deleted, and all contents of the corresponding file in the persistent data structure are also deleted.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A local knowledge base automated construction system based on multi-source acquisition and distributed computing, characterized in that: It includes a multi-source data collector and a distributed data processing and parsing module; the multi-source data collector realizes the access, data collection, data transmission and incremental update of multi-source data, and generates incremental change records; the distributed data processing and parsing module includes multiple distributed nodes, which are used to perform distributed processing and parsing of data files based on incremental change records on the data collected by the multi-source data collector, and establish a local knowledge base. The multi-source data collector includes a data acquisition probe module and a data transmission module; the data acquisition probe module is used to realize multi-source data access, data snapshot generation, data snapshot comparison, incremental change record generation, and data transmission submission; the data transmission module is used to realize incremental change record transmission, data file transmission, and expired file cleanup; The specific method by which the distributed data processing and parsing module performs distributed processing and parsing of data files based on incremental change records is as follows: When the incremental change record records a file change type of "new", the new file is sequentially subjected to distributed block operation, distributed classification operation, distributed vectorization operation, and knowledge base writing operation. When the incremental change record records a file change type of deletion, the related files, block content, and block result vector data in the knowledge base, persistent media, etc. are cleaned up in sequence.

2. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 1, characterized in that: The data acquisition probe module enables multi-source data access by defining the data source structure and acquisition strategy, and supports data source categories including: file type, database type and API type; The data source structure defined by the data acquisition probe module is as follows:<id,name,type,sconfig,dconfig> Where: id is the identifier of the data source; name is the name of the data source, which can be configured to be any string; type is the type of data source, including file type, database type, and API type. sconfig configures the data collection strategy for the data source side, and is used to configure the collection strategy of the multi-source data collector. The format is: <srcs[src<datakey,interval,option> The data structure is defined as follows: `srcs` is a list of data source collection strategies; `src` is a specific element that identifies the collection strategy for a particular data source; `datakey` is the key parameter for data retrieval from the data source. When the data source is a file, `datakey` is the specific file storage path; when the data source is a database, `datakey` is the specific SQL statement for retrieving data; when the data source is an API, `datakey` is the corresponding API address; `interval` is the data collection time interval; and `option` is an optional configuration used to configure different parameters required when retrieving data using `datakey`. dconfig specifies the data transmission module's configuration for receiving collected data. The format is:<dtype, path, option> dtype is the method by which the data transmission module receives files, with options for "HTTP service" and "object storage"; path is the path address for the data transmission module to receive files, including the address for receiving incremental change records and the address for receiving data files; option is an optional configuration.

3. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 2, characterized in that: The data acquisition probe module periodically generates data snapshots according to the strategy defined in the data source structure. The data snapshot structure is as follows: <snapshots[snapshot<snapid,createtime,rootpath,files<path,size,updatetime, hash> The data snapshots list is defined as follows: `snapshots` represents the list of data snapshots; `snapshot` represents the data snapshot element, specifically including the snapshot's `snapid`, creation time (`createtime`), rootpath (the directory corresponding to the snapshot), and `files` (the list of snapshot files). The `files` list includes information about multiple database snapshot files, specifically the snapshot file path, size, updatetime, and hash. When the data source is a file, the snapshot is generated directly based on the file structure. When the data source is a database, the acquired data is first saved to a file, and then a snapshot is generated based on the file structure. When the data source is an API, the data obtained through the API interface is first saved to a file, and then a snapshot is generated based on the file structure.

4. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 3, characterized in that: The data acquisition probe module periodically generates data snapshots according to the strategy defined by the data source structure. It then compares the newly generated data snapshot with the most recent saved data snapshot and generates incremental change records on a file-by-file basis. The specific format of the incremental change record is as follows:<optype,optime,filepath,filetype, filesize,filehash,fileupdatetime> Where optype is the incremental change type, including addition and deletion; optimum is the incremental change time; filepath is the original file path, filetype is the file type, filesize is the file size, filehash is the hash of the data snapshot file, and fileupdatetime is the latest update time of the file; when the data snapshot comparison finds that the file has been added or deleted, a corresponding incremental change record is generated; when the snapshot comparison finds that the file has been updated, an incremental change record of type addition is first generated for the updated file, and then an incremental change record of type deletion is generated for the original file before the update.

5. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 4, characterized in that: After generating incremental change records, the data acquisition probe module, for each file in the incremental change record, when the file is a new file, submits the new file to the distributed data processing and parsing module through the data transmission module. After the file is successfully uploaded, the incremental change record is then submitted to the distributed data processing and parsing module through the data transmission module. When the file is a deleted file, only the incremental change record is submitted to the distributed data processing and parsing module through the data transmission module. If the file upload process fails, skip this file upload and wait for the next round to re-perform data snapshot comparison, file generation, and file upload; After the data transmission module completes the file and incremental change record submission operation, it regenerates and saves the data snapshot for the successfully uploaded files and incremental change records.

6. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 1, characterized in that: The data transmission module is equipped with a persistent medium to receive incremental change records uploaded by the multi-source data collector. The incremental change records are stored in the persistent medium in the form of files. It receives data files uploaded by multi-source data collectors and stores them in persistent media; Based on the incremental change log, files involved in deletion are periodically cleaned up from the persistent media.

7. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 1, characterized in that: The distributed chunking operation executes chunking tasks in a distributed manner for newly added files. Based on the Spark framework, it is configured with dynamic resource allocation and fault tolerance mechanisms, specifically including: Task division involves reading incremental change records uploaded by multi-source data collectors through the data transmission module, and dividing all files to be processed into sub-tasks using a custom file division strategy to ensure that the total file size processed by each sub-task is the same. Distributed processing: each subtask parses the file one by one to generate text parsing results, divides the parsing results into blocks, and generates the block content of the file. Results aggregation involves collecting the generated content blocks and writing each block of content to a persistent medium; The distributed classification operation uses a deep learning algorithm to classify the block content in the persistent medium and persistently stores the classification results. The distributed vectorization operation determines the corresponding embedding model for the classification results obtained from the distributed classification; all block content with the same classification result is processed using a distributed task, and there are multiple distributed tasks. Each distributed task is processed as follows: Divide the data blocks to be processed into subtasks according to the number of blocks; each subtask processes the same number of blocks. For each distributed node, initialize the embedding model corresponding to the classification result, and set the embedding model to use the CPU for block content vectorization; The subtasks are assigned to distributed nodes to run, and the embedded model is used to parse the block content and generate block result vectors in turn. Write the block result vector parsed from the embedded model into the persistent medium; The knowledge base writing operation is as follows: adding new files, block content, and block result vectors to the knowledge base.

8. The local knowledge base automated construction system based on multi-source acquisition and distributed computing according to claim 7, characterized in that: When the incremental change record's change type is deletion, the specific method for sequentially cleaning up the knowledge base, related files, block content, and block result vector data in the persistent media is as follows: Based on the acquired incremental change records, the file to be deleted, the content of the block to be deleted, and the result vector of the block to be deleted are determined in sequence, and the relevant data in the knowledge base is deleted. After a successful knowledge base deletion operation, the block result vector, blocks, and content in the persistent medium are deleted sequentially.

Images