Local knowledge base automatic construction system based on multi-source acquisition and distributed computing

Abstract

The invention provides a local knowledge base automatic construction system based on multi-source acquisition and distributed computing, and relates to the technical field of computer systems. The system comprises a multi-source data collector and a distributed data processing and analyzing module, wherein the multi-source data collector realizes access, data collection, data transmission and incremental updating of multi-source data, and generates an incremental change record; and the distributed data processing and analyzing module comprises a plurality of distributed nodes and is used for carrying out distributed processing and analysis on data files according to incremental change records on the data collected by the multi-source data collector and establishing a local knowledge base. According to the system, automatic construction of the local knowledge base can be realized, the knowledge base construction efficiency is improved, the knowledge base construction period is shortened, the knowledge data freshness is ensured, and the data quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems, and in particular to a local knowledge base automatic construction system based on multi-source acquisition and distributed computing. Background Art

[0002] With the rapid development of the artificial intelligence industry, the application of new-generation general-purpose AI models, represented by DeepSeek, is gradually expanding and deepening across various industries thanks to its powerful natural language processing and logical reasoning capabilities. Finance, healthcare, manufacturing, and government services are among the first sectors to integrate DeepSeek. By building and integrating dedicated localized knowledge bases, DeepSeek is enabling core application scenarios such as intelligent customer service, decision support, and process automation.

[0003] The market size of smart application projects based on large language models is growing. However, current industry practices have exposed significant technical bottlenecks. The local knowledge base that supports intelligent applications has core pain points such as "long construction cycle, low parsing efficiency, delayed data updates, and low data quality."

[0004] Therefore, it is necessary to establish a more general and efficient technical method process based on distributed computing to realize the automated construction of local knowledge bases, improve the efficiency of knowledge base construction, shorten the knowledge base construction cycle, ensure the freshness of knowledge data, and improve data quality. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and provide a local knowledge base automatic construction system based on multi-source acquisition and distributed computing to realize the automatic construction of the local knowledge base.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: a local knowledge base automatic construction system based on multi-source collection and distributed computing, including a multi-source data collector and a distributed data processing and parsing module; the multi-source data collector implements multi-source data access, data collection, data transmission and incremental update, and generates incremental change records; the distributed data processing and parsing module includes multiple distributed nodes for distributed processing and parsing data files collected by the multi-source data collector according to the incremental change records, and establishing a local knowledge base; The multi-source data collector includes a data acquisition probe module and a data transmission module; the data acquisition probe module is used to implement multi-source data access, data snapshot generation, data snapshot comparison, incremental change record generation and data transmission submission; the data transmission module is used to implement incremental change record transmission, data file transmission and expired file cleaning; The specific method for the distributed data processing and parsing module to perform distributed processing and parsing of data files based on incremental change records is as follows: When the incremental change record records that the file change type is new, the new file is sequentially subjected to distributed block operations, distributed classification operations, distributed vectorization operations, and knowledge base writing operations; When the incremental change record records that the file change type is deletion, the knowledge base, related files in the persistent medium, block content, and block result vector data are cleaned up in turn.

[0007] Furthermore, the data acquisition probe module implements multi-source data access by defining data source structure and acquisition strategy, and supports data source categories including: file type, database type and API type.

[0008] Furthermore, the data source structure defined by the data acquisition probe module is<id,name,type,sconfig,dconfig> , where: id is the identifier of the data source; name is the name of the data source, which can be customized to any string; type is the data source type, including file type, database type, and API type; sconfig is the data source collection strategy configuration, which is used to configure the collection strategy of the multi-source data collector. The format is: <srcs[src<datakey,interval,option> ]>, where srcs is a list of data source collection strategies; src is a specific element that identifies the collection strategy of a data source; datakey is the key parameter for the data source to obtain data. When the data source is a file type, datakey is the specific file storage path; when the data source is a database type, datakey is the specific SQL statement for obtaining data; when the data source is an API type, datakey is the corresponding API interface address; interval is the data source collection time interval; option is an optional configuration used to configure different parameters required for datakey to obtain data; dconfig is the specific configuration for the data transmission module to receive collected data. The format is:<dtype, path,option> , where dtype is the way the data transmission module receives files, and the optional values ​​are "HTTP service" and "object storage"; path is the path address of the data transmission module to receive files, including the incremental change record receiving address and the data file receiving address; option is an optional configuration.

[0009] Furthermore, the data acquisition probe module generates data snapshots regularly according to the strategy defined by the data source structure. The data snapshot structure is <snapshots[snapshot<snapid,createtime,rootpath,files<path,size,updatetime, hash> >], where snapshots is a list of data snapshots; snaphot is a data snapshot element, specifically including data snapshot snapshot, data snapshot creation time createtime, data snapshot corresponding directory rootpath, and data snapshot file list files; the data snapshot file list files includes information about multiple database snapshot files, specifically data snapshot file path path, data snapshot file size size, data snapshot file update time updatetime, and data snapshot file hash; when the data source is a file class, a data snapshot is directly generated according to the file structure; when the data source is a database class, the acquired data is first saved as a file, and then a data snapshot is generated according to the file structure; when the data source is an API class, the data acquired through the API interface is first saved as a file, and then a data snapshot is generated according to the file structure.

[0010] Furthermore, the data acquisition probe module generates data snapshots according to the strategy defined by the data source structure, compares the latest generated data snapshot with the last saved data snapshot, and generates incremental change records in file units; the specific format of the incremental change record is<optype,optime,filepath,filetype,filesize,filehash,fileupdatetime> , where optype is the incremental change type, including addition and deletion; optime is the incremental change time; filepath is the original file path, filetype is the file type, filesize is the file size, filehash is the data snapshot file hash, and fileupdatetime is the latest update time of the file; when the data snapshot is compared and a file is added or deleted, a corresponding incremental change record is generated; when the snapshot comparison finds that a file is updated, an incremental change record of the addition type is generated for the updated file first, and then an incremental change record of the deletion type is generated for the original file before the update.

[0011] Furthermore, after the data acquisition probe module generates the incremental change record, for each file in the incremental change record, when the file is of a newly added type, the newly added file is submitted and uploaded to the distributed data processing and parsing module through the data transmission module. After the file is uploaded successfully, the incremental change record is submitted and uploaded to the distributed data processing and parsing module through the data transmission module; when the file is of a deleted type, only the incremental change record is submitted and uploaded to the distributed data processing and parsing module through the data transmission module; if a failure occurs during the file upload process, the file upload is skipped, and the data snapshot comparison, file generation and file upload are performed again in the next round; after the data transmission module completes the file and incremental change record submission operation, it regenerates the data snapshot and saves it for the successfully uploaded file and incremental change record.

[0012] Furthermore, the data transmission module is configured with a persistent medium to receive the incremental change records uploaded by the multi-source data collector, and the incremental change records are stored in the persistent medium in the form of files; Receive data files uploaded by multi-source data collectors and store them in persistent media; Based on the incremental change records, the deleted files are regularly cleaned up from the persistent media.

[0013] Furthermore, the distributed block operation performs block tasks for newly added files in a distributed manner, and configures dynamic resource allocation and fault tolerance mechanisms based on the Spark framework, specifically including: Task division: read the incremental change records uploaded by the multi-source data collector through the data transmission module, and divide all the files to be processed into subtasks through a custom file division strategy to ensure that the total file size processed by each subtask is the same; Distributed processing: each subtask parses the files one by one to generate text parsing results, divides the parsing results into blocks, and generates the block content of the file; Result aggregation: collect the generated blocks of content and write them to persistent media.

[0014] Furthermore, the distributed classification operation uses a deep learning algorithm to classify the block content in the persistent medium and persistently stores the classification results.

[0015] Furthermore, the distributed vectorization operation determines a corresponding embedding model for the classification results obtained by distributed classification; all block contents of the same classification result are processed using one distributed task, and there are multiple distributed tasks. The following processing is performed for each distributed task: Divide the block data to be processed into subtasks according to the number of block contents; each subtask processes the same number of blocks; For each distributed node, initialize the embedding model corresponding to the classification result, and set the embedding model to use the CPU for block content vectorization; Assign subtasks to distributed nodes to run, and use the embedding model to parse the block content in turn to generate block result vectors; Write the block result vector parsed by the embedded model to the persistent medium.

[0016] Furthermore, the knowledge base writing operation is: writing the newly added file, block content and block result vector into the knowledge base.

[0017] Furthermore, when the incremental change record change type is deletion, the specific method of clearing the knowledge base, related files in the persistent medium, block content, and block result vector data is as follows: Based on the acquired incremental change records, the files to be deleted, the block contents to be deleted, and the block result vectors to be deleted are determined in sequence, and the relevant data in the knowledge base is deleted. After the knowledge base deletion operation is successful, the block result vectors, blocks, and contents in the persistent medium are deleted in sequence.

[0018] The beneficial effects of adopting the above technical solution are: the local knowledge base automatic construction system based on multi-source acquisition and distributed computing provided by the present invention can realize the automatic construction of the local knowledge base, improve the efficiency of knowledge base construction, shorten the knowledge base construction cycle, ensure the freshness of knowledge data, and improve data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A structural diagram of a local knowledge base automatic construction system based on multi-source acquisition and distributed computing provided by an embodiment of the present invention; Figure 2 A diagram showing the working principle of a multi-source data collector provided by an embodiment of the present invention; Figure 3 This is a diagram of the working principle of the distributed data processing and parsing module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0021] In this embodiment, a local knowledge base automatic construction system based on multi-source acquisition and distributed computing is constructed. Figure 1As shown, it includes a multi-source data collector and a distributed data processing and analysis module; the multi-source data collector realizes the access, data collection, data transmission and incremental update of multi-source data, and generates incremental change records; the distributed data processing and analysis module includes multiple distributed nodes, which are used to perform distributed processing and analysis on the data files collected by the multi-source data collector according to the incremental change records, and establish a local knowledge base.

[0022] In this embodiment, the multi-source data collector is as follows: Figure 2 As shown, it includes a data acquisition probe module and a data transmission module; the data acquisition probe module is used to realize multi-source data access, data snapshot generation, data snapshot comparison, incremental change record generation and data transmission submission; the data transmission module is used to realize incremental change record transmission, data file transmission and expired file cleaning.

[0023] The data acquisition probe module implements multi-source data access by defining the data source structure and acquisition strategy, and supports data source categories including: file class (file), database class (db) and API class (api).

[0024] The data source structure defined by the data acquisition probe module is:<id,name,type,sconfig,dconfig> , where: id is the identifier of the data source; name is the name of the data source, which can be customized to any string; type is the data source type, including file type (file), database type (db), and API type (api); sconfig is the data source collection strategy configuration, which is used to configure the collection strategy of the multi-source data collector. The format is: <srcs[src<datakey,interval,option> ]>, where srcs is a list of data source collection strategies; src is a specific element that identifies the collection strategy of a data source; datakey is a key parameter for the data source to obtain data. When the data source is a file type, datakey is a specific file storage path; when the data source is a database type, datakey is a specific SQL statement for obtaining data; when the data source is an API type, datakey is the corresponding API interface address; interval is the data source collection time interval; option is an optional configuration used to configure the different parameters required for datakey to obtain data, including authentication method, request and response format definition, etc.

[0025] dconfig is the specific configuration for the data transmission module to receive collected data. The format is:<dtype, path,option> , where dtype is the way the data transfer module receives files, and the optional values ​​are "HTTP service" and "object storage (MinIO)"; path is the path address of the data transfer module to receive files, including the incremental change record receiving address and the data file receiving address; option is an optional configuration, including information such as the data transfer module authentication method.

[0026] In this embodiment, the data acquisition probe module generates data snapshots regularly according to the strategy defined by the data source structure. The data snapshot structure is <snapshots[snapshot<snapid,createtime,rootpath,files<path,size,updatetime, hash> >], where snapshots is a list of data snapshots; snaphot is a data snapshot element, specifically including data snapshot snapshot, data snapshot creation time createtime, data snapshot corresponding directory rootpath, and data snapshot file list files; the data snapshot file list files includes information about multiple database snapshot files, specifically data snapshot file path path, data snapshot file size size, data snapshot file update time updatetime, and data snapshot file hash; when the data source is a file class, a data snapshot is directly generated according to the file structure; when the data source is a database class, the acquired data is first saved as a file, and then a data snapshot is generated according to the file structure; when the data source is an API class, the data acquired through the API interface is first saved as a file, and then a data snapshot is generated according to the file structure.

[0027] The data acquisition probe module generates data snapshots according to the strategy defined by the data source structure, compares the latest generated data snapshot with the most recent saved data snapshot, and generates incremental change records in file units; the specific format of the incremental change record is<optype,optime,filepath,filetype, filesize,filehash,fileupdatetime> , where optype is the incremental change type, including addition and deletion; optime is the incremental change time; filepath is the original file path, filetype is the file type, filesize is the file size, filehash is the data snapshot file hash, and fileupdatetime is the latest update time of the file; when the data snapshot is compared and a file is added or deleted, a corresponding incremental change record is generated; when the snapshot comparison finds that a file is updated, an incremental change record of the addition type is generated for the updated file first, and then an incremental change record of the deletion type is generated for the original file before the update.

[0028] After the data acquisition probe module generates the incremental change record, for each file in the incremental change record, when the file is of a newly added type, the newly added file is submitted and uploaded to the distributed data processing and parsing module through the data transmission module. After the file is uploaded successfully, the incremental change record is submitted and uploaded to the distributed data processing and parsing module through the data transmission module; when the file is of a deleted type, only the incremental change record is submitted and uploaded to the distributed data processing and parsing module through the data transmission module; if a failure occurs during the file upload process, the file upload is skipped, and the data snapshot comparison, file generation and file upload are performed again in the next round; after the data transmission module completes the file and incremental change record submission operation, it regenerates the data snapshot for the successfully uploaded file and incremental change record and saves it.

[0029] The data transmission module is configured with a persistent medium, receives incremental change records uploaded by multi-source data collectors, and stores the incremental change records in the form of files in the persistent medium; receives data files uploaded by multi-source data collectors and stores them in the persistent medium; based on the incremental change records, regularly cleans up the files involved in the deletion from the persistent medium.

[0030] The distributed data processing and parsing module performs distributed processing and parsing on the data files according to the incremental change records, such as Figure 3 As shown, the specific method is: When the incremental change record records that the file change type is new, the new file is sequentially subjected to distributed block operations, distributed classification operations, distributed vectorization operations, and knowledge base writing operations; When the incremental change record records that the file change type is deletion, the knowledge base, related files in the persistent medium, block content, and block result vector data are cleaned up in turn.

[0031] The distributed block operation performs block tasks for newly added files in a distributed manner. Based on the Spark framework, it configures dynamic resource allocation and fault tolerance mechanisms, specifically including: Task division: read the incremental change records uploaded by the multi-source data collector through the data transmission module, and divide all the files to be processed into subtasks through a custom file division strategy to ensure that the total file size processed by each subtask is basically the same; Distributed processing: each subtask parses the files one by one to generate text parsing results, divides the parsing results into blocks, and generates the block content of the file; Result aggregation: collect the generated blocks of content and write them to persistent media.

[0032] Distributed classification operation uses deep learning algorithms to classify block content in persistent media and persistently stores the classification results.

[0033] Distributed vectorization operation determines the corresponding embedding model for the classification results obtained by distributed classification. All block contents of the same classification result are processed using one distributed task. There are multiple distributed tasks, and the following processing is performed for each distributed task: Divide the block data to be processed into subtasks according to the number of block contents; each subtask processes the same number of blocks; For each distributed node, initialize the embedding model corresponding to the classification result, and set the embedding model to use the CPU for block content vectorization; Assign subtasks to distributed nodes to run, and use the embedding model to parse the block content in turn to generate block result vectors; Write the block result vector parsed by the embedded model to the persistent medium.

[0034] The knowledge base writing operation is to write the newly added file, block content and block result vector into the knowledge base.

[0035] When the incremental change record change type is deletion, the specific method for cleaning up the knowledge base, related files in the persistent medium, block content, and block result vector data is as follows: Based on the acquired incremental change records, the files to be deleted, the block contents to be deleted, and the block result vectors to be deleted are determined in sequence, and the relevant data in the knowledge base is deleted. After the knowledge base deletion operation is successful, the block result vectors, blocks, and contents in the persistent medium are deleted in sequence.

[0036] In this embodiment, the multi-source data collector supports the definition and collection of file-type, database-type, and API-type data source structures. The definition of a file-type data source structure is shown below as an example: { "id": 1, "name": "Formal Policy and System Data Collection", "type": "file", "sconfig": [ { "datakey": " / data1 / rule1", "interval": "30min" }, { "datakey": " / data1 / rule2", "interval": "45min" } ], "dconfig": { "type": "httpservice", "path": "http: / / host1:8080 / rule", "option": { "user": "user1", "password": "password" } } };

[0037] In the above data source structure definition, the multi-source data collector reads the files in / data1 / rule1 and / data1 / rule2 and periodically reads the directories to generate data snapshots according to the configuration policy. The specific data snapshot files are as follows: { "id": "id1", "createtime": "2025-05-14T10:30:45Z", "rootpath": " / data1 / rule1", "files": [ { "path": "rule1.pdf", "size": 1024567, "lastModified": "2023-05-10T14:25:30Z", "hash":"sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" }, { "path": "rule2.pdf", "size": 2048152, "lastModified": "2023-05-12T09:15:22Z", "hash": "md5:098f6bcd4621d373cade4e832627b4f6" } ] };

[0038] After the data snapshot is generated, the difference is compared with the most recent saved data snapshot, and an incremental change record is generated per file, as shown below: [ { "optype":"add", "optime":"2025-05-14T10:30:45Z", "filepath":" / data1 / rule1 / rule1.pdf", "filetype":"pdf", "filesize":1024567, "filehash":"sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", "fileupdatetime":"2023-05-10T14:25:30Z" }, { "optype":"add", "optime":"2025-05-14T10:30:45Z", "filepath":" / data1 / rule1 / rule2.pdf", "filetype":"pdf", "filesize":2048152, "filehash":"md5:098f6bcd4621d373cade4e832627b4f6", "fileupdatetime":"2023-05-12T09:15:22Z" } ];

[0039] After generating the incremental change record, the incremental change record and the changed file are uploaded through the data transmission module. At the same time, the upload status of the file parsing results generated by the distributed data processing and parsing module is regularly monitored. When it is in the uploaded status, all the parsing results corresponding to the file are deleted.

[0040] Utilize the distributed data processing and parsing module to regularly capture changes, perform distributed computing and parsing on incremental change files based on incremental change records, and feed back to the local knowledge base; In this embodiment, a distributed data processing and parsing module is developed as an optional implementation for new changes: distributed processing based on the Spark framework, with dynamic resource allocation and fault tolerance. The entire processing flow includes distributed partitioning, distributed classification, distributed vectorization, and knowledge base updates. When the number of incremental changes reaches a certain threshold (time or file number reaches a threshold), the Spark distributed processing program starts: 1. Distributed block division: When the distributed node starts, the deep learning model and various parsers used are initialized; the formats supported by the parser include: PDF, DOCX, JSON, Markdown, HTML, etc. Task division strategy: According to the file size, the threshold is set to 512MB in this embodiment to divide the subtasks. When processing subtasks, the file needs to be divided into blocks, and the block size threshold is 512B. When processing adjacent blocks, if the last sentence of the previous block is truncated, the sentence will be included in the next block, and the block location information record will be generated at the same time, generating the following persistent storage medium directory structure, as shown below: file_id 1 / / bulk_id 1 / bulk content / Location information history / bulk_id 2 / bulk content / Location information history / bulk_id 3 / bulk content / Location information records; In this embodiment, the location information of the generated block is recorded as follows, where start represents the starting position in the document, and end represents the ending position in the document. file_id represents the file ID, and bulk_id represents the block ID; { "start": { "x":0, "y":0 }, "end": { "x":100, "y":80 }, "file_id": "a1", "bulk_id":"1" }; 2. Distributed classification: Use deep learning models to analyze block content, classify files based on the analysis results, and output the classification results. The directory structure stored in the persistent storage medium is as follows: file_id 1 / classification result 1; file_id 2 / classification result 2; 3. Distributed vectorization: Blocks with the same classification results will use the same embedding model and be processed using the same distributed vectorization task. Therefore, when a distributed node starts, it initializes the corresponding embedding model according to the classification results, reads the block content for vectorization, and generates a block result vector. The directory structure of the block result vector stored in the persistent storage medium is as follows: file_id 1 / / bulk_id 1 / Block result vector / bulk_id 2 / Block result vector / bulk_id 3 / block result vector; After distributed segmentation, distributed classification, and distributed vectorization, the final parsing results are stored in the following persistent storage medium directory structure, as shown below. The upload mark indicates the last part added, including three states: not uploaded, uploaded, and upload failed. file_id 1 / Classification result 1 / Upload logo / bulk_id 1 / bulk content / Location information history / block result vector / bulk_id 2 / bulk content / Location information history / block result vector / bulk_id 3 / bulk content / Location information history / block result vector;

[0041] In this embodiment, the knowledge base can adopt RAGFlow, use Minio and ES as the storage medium for incremental change files, block content and block result vectors, and for files with upload status of not uploaded and upload failed, they are converted according to the data format requirements of the RAGFlow knowledge base and submitted to the local knowledge base for storage, and the upload status is updated at the same time.

[0042] In this embodiment, a distributed data processing and parsing module is developed as an optional implementation method for when the change type is deletion: according to the file_id to be deleted, the file, block content and block result vector in the RAGFlow local knowledge base are directly deleted, and all the contents of the corresponding file in the persistent data structure are deleted at the same time.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A local knowledge base automatic construction system based on multi-source acquisition and distributed computing, characterized by: It includes a multi-source data collector and a distributed data processing and analysis module; the multi-source data collector implements multi-source data access, data collection, data transmission and incremental update, and generates incremental change records; the distributed data processing and analysis module includes multiple distributed nodes for distributed processing and analysis of data files collected by the multi-source data collector based on incremental change records, and establishes a local knowledge base; The multi-source data collector includes a data acquisition probe module and a data transmission module; the data acquisition probe module is used to implement multi-source data access, data snapshot generation, data snapshot comparison, incremental change record generation and data transmission submission; the data transmission module is used to implement incremental change record transmission, data file transmission and expired file cleaning; The specific method for the distributed data processing and parsing module to perform distributed processing and parsing of data files based on incremental change records is as follows: When the incremental change record records that the file change type is new, the new file is sequentially subjected to distributed block operations, distributed classification operations, distributed vectorization operations, and knowledge base writing operations; When the incremental change record records that the file change type is deletion, the knowledge base, related files in the persistent medium, block content, and block result vector data are cleaned up in turn.

2. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 1 is characterized by: The data acquisition probe module implements multi-source data access by defining the data source structure and acquisition strategy, and supports data source categories including: file class, database class and API class; The data source structure defined by the data acquisition probe module is:<id,name,type,sconfig,dconfig> , where: id is the identifier of the data source; name is the name of the data source, which can be customized to any string; type is the data source type, including file type, database type, and API type; sconfig is the data source collection strategy configuration, which is used to configure the collection strategy of the multi-source data collector. The format is: <srcs[src<datakey,interval,option> ]>, where srcs is a list of data source collection strategies; src is a specific element that identifies the collection strategy of a data source; datakey is the key parameter for the data source to obtain data. When the data source is a file type, datakey is the specific file storage path; when the data source is a database type, datakey is the specific SQL statement for obtaining data; when the data source is an API type, datakey is the corresponding API interface address; interval is the data source collection time interval; option is an optional configuration used to configure different parameters required for datakey to obtain data; dconfig is the specific configuration for the data transmission module to receive collected data. The format is:<dtype, path, option> , where dtype is the way the data transfer module receives files, with optional values ​​of "HTTP service" and "object storage"; path is the path address of the data transfer module receiving files, including the incremental change record receiving address and the data file receiving address; option is an optional configuration.

3. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 2 is characterized by: The data acquisition probe module generates data snapshots regularly according to the strategy defined by the data source structure. The data snapshot structure is <snapshots[snapshot<snapid,createtime,rootpath,files<path,size,updatetime, hash> >], where snapshots is a list of data snapshots; snaphot is a data snapshot element, specifically including data snapshot snapshot, data snapshot creation time createtime, data snapshot corresponding directory rootpath, and data snapshot file list files; the data snapshot file list files includes information about multiple database snapshot files, specifically data snapshot file path path, data snapshot file size size, data snapshot file update time updatetime, and data snapshot file hash; when the data source is a file class, a data snapshot is directly generated according to the file structure; when the data source is a database class, the acquired data is first saved as a file, and then a data snapshot is generated according to the file structure; when the data source is an API class, the data acquired through the API interface is first saved as a file, and then a data snapshot is generated according to the file structure.

4. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 3 is characterized by: The data acquisition probe module generates data snapshots according to the strategy defined by the data source structure, compares the latest generated data snapshot with the most recent saved data snapshot, and generates incremental change records in file units; the specific format of the incremental change record is<optype,optime,filepath,filetype, filesize,filehash,fileupdatetime> , where optype is the incremental change type, including addition and deletion; optime is the incremental change time; filepath is the original file path, filetype is the file type, filesize is the file size, filehash is the data snapshot file hash, and fileupdatetime is the latest update time of the file; when the data snapshot is compared and a file is added or deleted, a corresponding incremental change record is generated; when the snapshot comparison finds that a file is updated, an incremental change record of the addition type is generated for the updated file first, and then an incremental change record of the deletion type is generated for the original file before the update.

5. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 4 is characterized in that: After the data acquisition probe module generates the incremental change record, for each file in the incremental change record, if the file is a newly added type, the newly added file is submitted and uploaded to the distributed data processing and parsing module through the data transmission module. After the file is successfully uploaded, the incremental change record is submitted and uploaded to the distributed data processing and parsing module through the data transmission module; if the file is a deleted type, only the incremental change record is submitted and uploaded to the distributed data processing and parsing module through the data transmission module; If a failure occurs during the file upload process, the current file upload will be skipped and the data snapshot comparison, file generation, and file upload will be repeated in the next round; After the data transmission module completes the file and incremental change record submission operation, it regenerates the data snapshot and saves it for the successfully uploaded files and incremental change records.

6. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 1 is characterized in that: The data transmission module is configured with a persistent medium to receive incremental change records uploaded by the multi-source data collector, and the incremental change records are stored in the persistent medium in the form of files; Receive data files uploaded by multi-source data collectors and store them in persistent media; Based on the incremental change records, the deleted files are regularly cleaned up from the persistent media.

7. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 1 is characterized in that: The distributed block operation performs block tasks for newly added files in a distributed manner, configures dynamic resource allocation and fault tolerance mechanisms based on the Spark framework, and specifically includes: Task division: read the incremental change records uploaded by the multi-source data collector through the data transmission module, and divide all the files to be processed into subtasks through a custom file division strategy to ensure that the total file size processed by each subtask is the same; Distributed processing: each subtask parses the files one by one to generate text parsing results, divides the parsing results into blocks, and generates the block content of the file; Result aggregation: collect the generated blocks of content and write them to persistent media; The distributed classification operation classifies the block content in the persistent medium using a deep learning algorithm and persistently stores the classification results; The distributed vectorization operation determines the corresponding embedding model for the classification results obtained by distributed classification. All block contents of the same classification result are processed using one distributed task. There are multiple distributed tasks, and the following processing is performed for each distributed task: Divide the block data to be processed into subtasks according to the number of block contents; each subtask processes the same number of blocks; For each distributed node, initialize the embedding model corresponding to the classification result, and set the embedding model to use the CPU for block content vectorization; Assign subtasks to distributed nodes to run, and use the embedding model to parse the block content in turn to generate block result vectors; Write the block result vector parsed by the embedded model into a persistent medium; The knowledge base writing operation is to write the newly added file, block content and block result vector into the knowledge base.

8. The local knowledge base automatic construction system based on multi-source acquisition and distributed computing according to claim 7 is characterized in that: When the incremental change record change type is deletion, the specific method for cleaning up the knowledge base, related files in the persistent medium, block content, and block result vector data is as follows: Based on the acquired incremental change records, the files to be deleted, the contents of the blocks to be deleted, and the result vectors of the blocks to be deleted are determined in sequence, and the relevant data in the knowledge base is deleted; After the knowledge base deletion operation is successful, the block result vector, block, and content in the persistent medium are deleted in sequence.

Images