Multi-modal AI knowledge base construction system oriented to privatized deployment

Abstract

The invention provides a private deployment-oriented multi-modal AI knowledge base construction system. The private deployment-oriented multi-modal AI knowledge base construction system comprises a knowledge storage module, an intelligent document loading module, a document partitioning engine module, a data enhancement engine module, a multi-language semantic vector alignment module and a private deployment module, the knowledge storage module comprises a knowledge authority management sub-module and a knowledge source management sub-module, the knowledge authority management sub-module is used for managing and storing knowledge from different sources, and the knowledge source management sub-module is used for managing electronic documents and multimedia documents; according to the method, intelligent identification, partitioning, vectorization and source file storage can be carried out on different types of electronic files, knowledge graph construction is carried out for specific fields, semantic relevance between texts and topics and key entities is fully considered, the method has wider applicability, higher robustness and controllability, the data leakage risk is effectively reduced, and the method is suitable for popularization and application. The method is suitable for enterprise sensitive data protection and personal user elastic computing power requirements.

Description

Technical Field

[0001] This invention belongs to the field of archives management, and in particular relates to a multimodal AI knowledge base construction system for private deployment. Background Technology

[0002] With the rapid development of artificial intelligence and information technology, knowledge management and information retrieval are becoming increasingly important in various fields, especially in application scenarios such as enterprise archive knowledge management, data mining, and personalized information recommendation. Traditional knowledge management systems often rely on structured data. This approach suffers from low information security levels and difficulty in achieving intelligent utilization when faced with multi-source heterogeneous data, unstructured information, and rich multimodal content (such as text, images, and videos). For example, existing chunking methods are not adaptable to multimodal / complex documents, and it is difficult to deeply integrate fine-grained security control with the knowledge base construction process.

[0003] Currently, the main challenges in building AI knowledge bases include: First, how to effectively integrate heterogeneous data from different sources to achieve integrated knowledge representation is an urgent problem to be solved. In addition, in a private deployment environment, system security, customizability, and maintenance costs are also key factors to consider when building a multimodal AI knowledge base.

[0004] In light of the current situation, this paper introduces multimodal AI technology for private deployment, combined with advanced methods such as deep learning and natural language processing, to provide new ideas for solving these problems. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to build a multimodal AI knowledge base construction system for private deployment, which addresses the shortcomings of the existing technology. The system includes a knowledge storage module, an intelligent document loading module, a document chunking engine module, a data augmentation engine module, a multilingual semantic vector alignment module, and a private deployment module.

[0006] The knowledge storage module includes a knowledge access management submodule and a knowledge source management submodule. The knowledge access management submodule is used to manage and store knowledge from different sources, and the knowledge source management submodule is used to manage electronic documents and multimedia documents.

[0007] The intelligent document loading module preprocesses different types of files using a document loader. By preloading files and analyzing their formats, it identifies and extracts high-quality content from the documents.

[0008] The document segmentation engine module automatically divides large blocks of text into smaller, more reasonable segments by building a lightweight document segmenter.

[0009] The data augmentation engine module uses data augmentation technology and knowledge graph methods to augment the data of the segmented content and add semantic features.

[0010] The multilingual semantic vector alignment module utilizes the capabilities of the Embedding model and the multimodal model to convert text and multimedia files into vector representations;

[0011] The private deployment module provides a localized deployment interface for enterprise private data and AI models, allowing only authorized personnel to access sensitive information.

[0012] The knowledge access management submodule supports the uploading, updating, deletion, and sharing of electronic documents and multimedia documents, and has access control to ensure information security: the knowledge access management submodule adopts a fine-grained access control mechanism that combines role-based access control and attribute-based access control to achieve document-level access control;

[0013] The knowledge access management submodule uses TLS / SSL encryption protocol during data transmission; for sensitive data stored in the system, the system uses a private file cabinet mechanism for encrypted storage, uses encryption algorithms to encrypt sensitive data, and implements key segmentation management; the system also establishes a data backup and recovery mechanism.

[0014] The knowledge access management submodule has an operation recording function, which permanently records the operations of the entire knowledge management process. The recorded content includes operation time, operation type, operator identity, operation content and operation result. The recorded content is stored in an encrypted manner to ensure that it cannot be tampered with, and is archived and backed up regularly.

[0015] The knowledge access control submodule integrates system user behavior monitoring functions to monitor users' system operation behavior in real time and identify abnormal operations. The system user behavior monitoring function can periodically detect issues such as overlapping permissions, missing permissions, and lack of permissions, and optimize permission allocation through permission conflict detection mechanisms and periodic permission reviews. At the same time, the knowledge access control submodule also provides real-time alarm and post-event traceability functions.

[0016] The intelligent document loading module uses a document loader to preload documents. The intelligent document loading module implements a cross-modal document structured parsing technology. The document loader includes a PDF Loader, an ImageLoader, a VideoLoader, and an AudioLoader.

[0017] The PDF Loader is responsible for loading document file content, and the content it identifies includes: text, titles, images, tables, and formulas.

[0018] The ImageLoader is responsible for recognizing the text content and description contained in the image;

[0019] The VideoLoader is responsible for extracting keyframes from video files using software or hardware acquisition devices, identifying and generating corresponding images and descriptions.

[0020] The AudioLoader is responsible for recognizing audio files, including: text, summary and topic analysis, keywords, entity and custom tag extraction.

[0021] The PDF Loader fine-tunes the layout detection model using annotations from PDF documents in two or more formats. The fine-tuned model achieves accurate extraction results on diverse PDF documents such as papers, textbooks, research reports, and financial reports, and demonstrates high robustness against challenges such as blurring and watermarks. The layout detection model used by the PDF Loader is the LayoutLMv3 model, and the following steps are performed:

[0022] Step 1-1: Read the PDF file page by page;

[0023] Steps 1-2: Locate different elements in the document, including images, tables, text, titles, and formulas, based on the fine-tuned LayoutLMv3 model;

[0024] Steps 1-3: Use UniMERNet to recognize the formula image as LaTeX (a typesetting system based on TeX) source code. UniMERNet is an algorithm designed for recognizing various formulas in real-world scenarios. By building large-scale training data and carefully designed results, it achieves excellent recognition performance for complex long formulas, handwritten formulas, and noisy screenshot formulas.

[0025] Steps 1-4 involve detecting and performing OCR (Optical Character Recognition) on the text and text boxes. The OCR recognition is based on a PaddleOCR model that has been trained and fine-tuned to return the text box coordinates and recognition results. The OCR recognition includes the following steps:

[0026] Step 1-4-1: Perform text detection on the document and obtain the coordinates of the text boxes;

[0027] Step 1-4-2: Rotate and crop each text box;

[0028] Step 1-4-3: Perform text recognition on the cropped document;

[0029] Step 1-4-4: Filter out recognition results with a confidence level lower than the specified threshold (default is 0.5).

[0030] Steps 1-5 involve performing layout analysis on the document image to identify different types of regions;

[0031] Steps 1-6 involve parsing the document content and processing the identified regions, specifically including the following steps:

[0032] Step 1-6-1: Using the LORE table structure recognition model, elements in the table region are identified. The LORE table structure recognition model combines the spatial and logical positions of table cells for end-to-end modeling and prediction. The LORE table structure recognition model uses a keypoint segmentation network based on a convolutional neural network to extract visual features, and predicts the spatial and logical positions of cells through two regression heads respectively. Finally, the LORE table structure recognition model reconstructs the row and column structure of the entire table and outputs a machine-readable representation.

[0033] Step 1-6-2: Output the cell content and the cell's spatial and logical positions;

[0034] Steps 1-7 integrate the results of layout analysis and OCR recognition, introduce a layout restoration process, and convert the document into Markdown (a lightweight markup language) for output.

[0035] Steps 1-5 include:

[0036] Step 1-5-1: Group the tables, titles, and paragraphs output from the layout analysis;

[0037] Step 1-5-2: Merge the text within the group according to spatial and logical positions;

[0038] Step 1-5-3: Convert the merged text to Markdown according to its type.

[0039] The lightweight document segmenter includes a recursive character text segmenter, a text semantic segmenter, a fixed-size character text segmenter, a document structure segmenter, a Large Language Model (LLM) segmenter, and a custom segmenter. The lightweight document segmenter is used to divide the text output by the intelligent document loading module into various small blocks.

[0040] The recursive character text segmenter pre-segments the text by specifying a block length (default is 1024) and a set of delimiters, according to the priority of the delimiters, dividing the text into block documents with a length smaller than the original document, then merging the block documents that do not exceed the specified block length, and recursively splitting the text that exceeds the specified block length until the desired block size is obtained.

[0041] The text semantic segmenter segments text by identifying breakpoints. The breakpoints are determined as follows: if the distance between the embedding vectors of two consecutive paragraphs exceeds a specified threshold (0.5 by default), a breakpoint is set between the two consecutive paragraphs to segment the text.

[0042] The fixed-size character text segmenter cuts the text into small segments of the same size according to a preset number of characters, words, or tokens, and preserves the contextual coherence by overlapping.

[0043] The document structure segmenter uses the inherent structure of the document (such as headings, chapters, tables, etc.) to divide it, with each structural unit as a block. When the content of the document chapters is of different sizes and exceeds the block size limit, it needs to be split and merged in conjunction with a recursive character text segmenter. The document structure segmenter maintains structural integrity by aligning with the logical parts of the document.

[0044] The LLM segmenter directly inputs the original document into the Large Language Model (LLM), which intelligently generates semantic blocks; and dynamically segments the text using the semantic understanding capabilities of the LLM segmenter.

[0045] The custom segmenter dynamically segments the document using user-defined characters or strategies; for example, if the video segmentation strategy is set to split by second, the custom segmenter extracts keyframes from the video file in units of seconds, thereby completing the video file segmentation.

[0046] The AudioLoader performs the following steps:

[0047] Step 2-1: Load the speech recognition model and set up the hot word dictionary; the speech recognition model performs the following steps:

[0048] Step 2-1-1: Extract the acoustic feature vector corresponding to the target text using the built-in Predictor module based on the Continuous Integrate-and-Fire (CIF) predictor.

[0049] Step 2-1-2: Sampling is performed using the built-in Sampler to transform the acoustic feature vector and the target text vector into feature vectors containing semantic information. This is combined with a bidirectional decoder to enhance the model's ability to model the context.

[0050] Step 2-1-3: Train the speech recognition model based on the MWER (Minimum Word Error Rate) training criterion of negative sample sampling;

[0051] Step 2-2: Load the audio file, perform text recognition on the audio using a speech recognition model, and output the recognized text results;

[0052] Steps 2-3: Correct the text results;

[0053] Steps 2-4 involve summarizing and analyzing the text results, extracting keywords and entities, using prompt word engineering.

[0054] Steps 2-5: Extract system tag information contained in the text results.

[0055] The data augmentation engine module performs the following steps:

[0056] Step 3-1, extract metadata information, specifically including the following steps:

[0057] Step 3-1-1: Extract the page number, file name, author, and timestamp corresponding to the segment;

[0058] Step 3-1-2, Text preprocessing: Remove stop words and punctuation marks;

[0059] Step 3-1-3: Generate summary information for the segmented fragments;

[0060] Step 3-1-4: Use BERT (Bidirectional Encoder Representations from Transformers) to extract document embeddings, obtain document-level representations, extract word embeddings of N-gram words or phrases, and use cosine similarity to find the words or phrases most similar to the document.

[0061] Step 3-1-5: Generate a specified number of keywords (default is 3);

[0062] Step 3-1-6 generates a specified number (default is 3) of questions related to the chunk fragments;

[0063] Step 3-2, cross-page table merging technology, combines the LORE table structure recognition model to handle cross-page table merging problems, specifically including the following steps:

[0064] Step 3-2-1, based on the layout recognition judgment method, requires correctly identifying the layout and removing the header and footer of each page, and determining the type of the last element of the current block and the first element of the next block. If both are table types, it is determined to be a continuation of a cross-page table.

[0065] Step 3-2-2, Visual continuity analysis: Use an object detection model to identify table bounding boxes and calculate the geometric overlap rate between adjacent pages. If the overlap rate is greater than a threshold of 60%, it is determined to be a cross-page table. The overlap rate X1 is calculated as follows:

[0066] X1 = X2 / min(X3, X4),

[0067] Where X2 represents the area of ​​the overlapping region, X3 represents the area of ​​the table on the previous page, and X4 represents the area of ​​the table on the next page;

[0068] Step 3-2-3, Logical Continuity Verification: Through semantic similarity verification of the table header, when the semantic similarity between the description information of the first table and the description information of the second page table is greater than the threshold of 85%, it is determined to be a cross-page table.

[0069] Step 3-2-4: Merge the tables corresponding to the segments that satisfy the judgment rules of Step 3-2-1 to Step 3-2-3 into a complete table and store it as a segment;

[0070] Step 3-3: Supplement each segment with contextual information. Based on the Large Language Model (LLM), use prompt word engineering to provide each document segment with an additional description (including summary, keywords, and related questions) to make it more "self-contained context". The context includes: summary, keywords, and related questions.

[0071] Steps 3-4, Raptor (recursive abstract processing of tree-structured retrieval) enhancement: Through recursive embedding, clustering, and summarizing, the original fragmented data is processed to create an information tree containing different granularities and semantic spans. Specifically, the steps include:

[0072] Step 3-4-1: Convert the segmented pieces into vector form using an embedding model;

[0073] Step 3-4-2 involves grouping and clustering the segmented text, then summarizing the original text using cue word engineering to create new segments. This method employs a Gaussian Mixture Model (GMM), a clustering method based on maximum likelihood estimation and expectation-maximization (EM) algorithms. The algorithm assumes that data points are generated from one or more Gaussian distributions and estimates the parameters of the Gaussian distribution for each cluster using maximum likelihood estimation. The probability density function of the Gaussian distribution is:

[0074]

[0075] Where f(x; μ, ∑) is the Gaussian probability density function, x is the target data point, n represents the dimension of the variable x, μ is the mean vector, ∑ is the covariance matrix, |∑| is the determinant of the covariance matrix; exp is the natural exponential function; T represents the matrix transpose; this formula describes the probability density function of x given the mean μ and the covariance matrix ∑.

[0076] The goal of a Gaussian mixture model (GMM) is to maximize the likelihood function of the data.

[0077]

[0078] Where L represents the likelihood function, N represents the number of data points, K represents the number of Gaussian distributions, and x i For the i-th data point, μ j Let π represent the mean vector of the j-th Gaussian distribution. j Let ∑ represent the weights of the j-th Gaussian distribution. j Let represent the covariance matrix of the j-th Gaussian distribution. The Expectation-Maximization (EM) algorithm is used to solve for the parameters of the Gaussian Mixture Model (GMM). The Expectation-Maximization (EM) algorithm includes the following steps:

[0079] Step 3-4-2-1: Initialize the parameters of the Gaussian Mixture Model (GMM);

[0080] Step 3-4-2-2: Calculate the responsivity of all data points;

[0081] Step 3-4-2-3: Update the parameters of each Gaussian distribution;

[0082] Step 3-4-2-4: Calculate the likelihood function of the model;

[0083] Step 3-4-2-5: Determine whether the model parameters have converged. If they have converged, end the iteration; otherwise, return to step 3-4-2-2.

[0084] Through continuous iteration, the data is clustered using the GMM clustering algorithm;

[0085] Step 3-4-3: Repeat step 3-4-2 until the text can no longer be divided, that is, the number of blocks is 1. Each execution of step 3-4-2 is the construction of a layer of nodes. For non-leaf nodes, each non-leaf node stores its own index, child nodes, the summarized original text and the corresponding vector.

[0086] Steps 3-5 involve defining a domain ontology for a specific domain, such as manufacturing. Entity relationships are extracted from the segmented data using a large language model. Based on deep learning and large model technology, a domain knowledge graph is constructed. Leiden technology is used to perform hierarchical clustering on the initial knowledge graph and generate community summaries. The construction of the domain knowledge graph includes the following steps:

[0087] Step 3-5-1, Define the domain ontology: Establish the ontology to describe the various concepts and relationships in the domain, using the knowledge of domain experts and publicly available domain literature to define the hierarchical relationships, attributes, and constraints of concepts;

[0088] Step 3-5-2, Entity Recognition and Relationship Extraction: Using information extraction and large model techniques, identify entities of a specified type from segmented text and multimodal data, and extract potential relationships between entities;

[0089] Step 3-5-3, Construct a knowledge graph: Generate a preliminary knowledge graph based on the extracted entities and relationships, and use a graph database to store entities and relationships in the form of nodes and edges to maintain the dynamic update and scalability of the graph;

[0090] Steps 3-5-4, Hierarchical clustering of the knowledge graph: The Leiden algorithm is applied to perform hierarchical clustering analysis on the nodes of the preliminary knowledge graph to form a concept set with a community structure;

[0091] Step 3-5-5: Generate community summaries: Automatically generate summaries for community nodes based on hierarchical clustering results, and use language models to summarize the main themes and important information of each community.

[0092] The multilingual semantic vector alignment module performs the following steps:

[0093] Step 4-1: After loading the PDF file using the PDF Loader, use the corresponding chunker to chunk the text content and enhance the context.

[0094] Step 4-2: After loading the image and video files based on the ImageLoader and VideoLoader, generate the corresponding text descriptions, and use the corresponding chunkers to segment the text content and enhance the context.

[0095] Step 4-3: Based on the BGE (BAAI General Embedding) model, the segmented documents are used as input to generate corresponding semantic vectors. The BGE model uses an instruction fine-tuning approach, performing the following steps:

[0096] Step 4-3-1: Generate the corpus and fine-tune the dataset format as [query, positive sample set, negative sample set], where query is the query text, the positive sample set represents the text related to the query, and the negative sample set represents the text unrelated to the query.

[0097] Step 4-3-2, set the fine-tuning parameters;

[0098] Step 4-3-3: Fine-tuning the Embedding model treats the semantic relevance task as a binary classification task, uses the cross-entropy loss function as the loss function, and employs backpropagation to update the model parameters, outputting the fine-tuned model; the cross-entropy loss function is:

[0099]

[0100] Where H(p,q) is the cross-entropy between probability distribution q and probability distribution p, and p(x) i q(x) represents the probability value of the i-th sample under the probability distribution. i ) represents the probability value of the i-th sample under probability distribution q, where n is the total number of samples;

[0101] Step 4-4: Store the semantic vector and the original text in the vector library and file repository.

[0102] The private deployment module supports the localized deployment of enterprise internal data and AI models, ensuring that data does not cross the enterprise's security boundaries. Combined with the data storage module, the system provides a flexible user management mechanism, allowing enterprises to customize access permissions according to their own needs, ensuring that only authorized users can access sensitive information. A logging and auditing mechanism is introduced to monitor and record system and user behavior in real time, enhancing the system's internal control and security. System backup and recovery functions are provided to ensure data integrity and rapid system recovery in case of emergencies.

[0103] The multilingual semantic vector conversion module converts text content into vector representations, facilitating efficient semantic retrieval. To improve the model's performance in specific domains, the general model is fine-tuned using data from the vertical domain. This step ensures that the generated semantic vectors are closer to the domain-specific semantics. This module supports text conversion in multiple languages, ensuring it can handle documents in different languages ​​and providing broader applicability. Finally, the text is converted into fixed-length vectors for subsequent data utilization.

[0104] The private deployment module refers to deploying physical servers or virtualized clusters in an enterprise's own data center or server room. All components run on this local hardware. This deployment method offers the highest level of physical isolation and provides complete control over the hardware and network. Enterprises can independently define access permissions and user management mechanisms to ensure that only authorized personnel can access sensitive information, thereby improving internal control capabilities and guaranteeing the security, customizability, and maintenance costs of the knowledge management system. Deploying the system within a local network reduces or avoids public network transmission latency, making it particularly suitable for internal applications with extremely high response speed requirements. Private deployment utilizes open-source models and proprietary infrastructure, significantly reducing dependence on a single public cloud or closed-source API provider, resulting in lower migration costs.

[0105] This invention is a platform built using artificial intelligence technology for centralized storage, intelligent management, and efficient utilization of core enterprise knowledge assets. Its core function is to address the challenges of traditional knowledge management, transforming a static knowledge base into a dynamic intelligent hub, empowering employees, optimizing processes, and improving decision-making quality. The system fully leverages the characteristics of multimodal information, enhancing knowledge discoverability and utilization to meet increasingly complex knowledge management needs. The innovations of this invention lie in its cross-modal document structured parsing and cross-page table merging technology, its dynamic semantic layering technology based on Raptor enhancement and knowledge graph fusion, a specific combination and optimization of existing technologies, and the synergistic effects and technical capabilities of this combination in solving the problem of building private multimodal knowledge bases. This invention aims to solve the following problems:

[0106] Semantic understanding of multimodal heterogeneous data: Existing systems struggle to effectively integrate and understand the deep semantic information from multi-source heterogeneous data such as text, images, audio, video, tables, and formulas, and cannot achieve unified knowledge representation across modalities. By using an intelligent document loading module, multilingual semantic vector alignment, and processing of multimodal information, we can achieve unified parsing, semantic extraction, and vectorized representation of multi-source heterogeneous data content.

[0107] Automated knowledge extraction: Utilizing advanced parsing technology, it seamlessly processes documents of various formats (text, audio, video, etc.) and automatically extracts and integrates key knowledge content.

[0108] Deep knowledge mining in specific scenarios: By leveraging multi-level summary trees and structured relationship networks provided by knowledge graphs, combined with high-quality vectors generated by multilingual semantic models, the system supports more complex semantic matching, multi-level relationship reasoning, and cross-domain deep knowledge mining, meeting users' precise and in-depth information needs.

[0109] Secure storage and access control: Deploy efficient and secure system modules, combined with granular permission management, to ensure the controllability and ease of use of core knowledge assets.

[0110] The present invention has the following beneficial effects:

[0111] (1) Intelligent document and segmentation technology: It can intelligently read multimodal archive files and perform various efficient and accurate segmentation processes on electronic archives of different formats, thereby improving the flexibility and convenience of document processing.

[0112] (2) Multilingual and multimodal compatibility: The multilingual semantic vector conversion module can process documents in multiple languages, such as text and images. By fine-tuning the model, it enhances the adaptability to different embedding environments.

[0113] (3) Data augmentation engine: Through various data augmentation methods, the diversity of samples is increased. Through knowledge graph construction and other methods, the system can better understand and associate information, improve the overall quality of the knowledge base, increase semantic features, and facilitate knowledge management and subsequent data utilization.

[0114] (4) Private deployment capability: Enterprises have complete control over hardware and network. Enterprises can define access permissions and user management mechanisms to ensure that only authorized personnel can access sensitive information, improve internal control capabilities, and ensure the security, customizability and maintenance cost of the knowledge management system.

[0115] (5) Improve work efficiency: Overall, this invention significantly improves the efficiency of document knowledge extraction and management through automated and intelligent processing, reduces the need for manual intervention, solves the challenges faced by traditional knowledge management, transforms the static knowledge base into a dynamic intelligent hub, empowers enterprise employees, optimizes processes and improves decision-making quality. Attached Figure Description

[0116] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0117] Figure 1 This is a structural diagram of the system of the present invention.

[0118] Figure 2 This is a flowchart of the system of the present invention. Detailed Implementation

[0119] In this embodiment of the invention, a multimodal AI knowledge base construction system for private deployment is specifically provided, such as... Figure 1 As shown, it includes a knowledge storage module, a document loading module, a document chunking engine module, a data augmentation engine module, a multilingual semantic vector alignment module, and a private deployment module.

[0120] The knowledge storage module includes a knowledge access control submodule and a knowledge source management submodule. The knowledge access control submodule is used to manage and store knowledge from different sources, and the knowledge source management submodule is used to manage electronic documents and multimedia documents.

[0121] The knowledge access control submodule supports the uploading, updating, deletion, and sharing of electronic documents (such as PDF, Word, Markdown, TXT, HTML) and multimedia documents (audio and video files), and has access control to ensure information security: The knowledge access control submodule adopts a fine-grained access control mechanism that combines role-based access control (RBAC) and attribute-based access control (ABAC) to achieve document-level access control.

[0122] The knowledge access management submodule uses TLS / SSL encryption protocol during data transmission; for sensitive data stored in the system, the system uses a private file cabinet mechanism for encrypted storage, uses encryption algorithms to encrypt sensitive data, and implements key segmentation management; the system also establishes a data backup and recovery mechanism.

[0123] The knowledge access management submodule has an operation recording function, which permanently records the operations of the entire knowledge management process. The recorded content includes operation time, operation type, operator identity, operation content and operation result. The recorded content is stored in an encrypted manner to ensure that it cannot be tampered with, and is archived and backed up regularly.

[0124] The document loading module preprocesses different types of files. By preloading files and analyzing their formats, it efficiently identifies and extracts high-quality content from complex and diverse documents. It can recognize and process electronic archives of different formats, such as MP4, MP3, PDF, Word, and Markdown, which facilitates subsequent processing.

[0125] The intelligent document segmentation utilizes natural language processing technology to automatically break down long texts into smaller, more manageable text blocks by building a lightweight document segmentation engine, so as to better understand, process, or store the content of these files.

[0126] The data augmentation described herein increases sample diversity through various data augmentation methods, and helps the system better understand and associate information through knowledge graph construction, thereby improving the overall quality of the knowledge base, adding semantic features, and facilitating knowledge management and subsequent data utilization.

[0127] The multilingual semantic vector conversion transforms text information into vector form for fast and efficient semantic search. It supports fine-tuning of the general model using industry-specific data, ensuring that the output semantic vectors better meet the semantic needs of the specific domain. Furthermore, this module possesses multilingual processing capabilities, handling documents in various languages, thus enabling its wide application range. The multilingual semantic vector conversion module can process documents in multiple languages, such as text and images, enhancing adaptability to different embedding environments.

[0128] The private deployment module refers to deploying physical servers or virtualized clusters in an enterprise's own data center or server room. All components run on this local hardware. This deployment method offers the highest level of physical isolation and provides complete control over the hardware and network. Enterprises can independently define access permissions and user management mechanisms to ensure that only authorized personnel can access sensitive information, thereby improving internal control capabilities and guaranteeing the security, customizability, and maintenance costs of the knowledge management system. Deploying the system within a local network reduces or avoids public network transmission latency, making it particularly suitable for internal applications with extremely high response speed requirements. Private deployment utilizes open-source models and proprietary infrastructure, significantly reducing dependence on a single public cloud or closed-source API provider, resulting in lower migration costs.

[0129] like Figure 2 The diagram shown is a flowchart of the process of this embodiment, which specifically includes the following steps:

[0130] Step 1-1: Define knowledge permissions and start the system. Enter the document loading module. This process ensures that different users' access permissions to knowledge can be distinguished and managed according to permission levels.

[0131] Steps 1-2 involve importing data from knowledge sources. This data can be electronic documents or multimedia documents (such as audio and video). Different document types are processed through different parsing channels to ensure that subsequent processing is optimized for the data type.

[0132] Steps 1-3: Read the electronic files. Upload the electronic file samples that need to be retrieved, such as excerpts from "Employee Handbook.pdf", "Telemarketing Records.mp3", and "Warehouse Personnel Entry and Exit Records.mp4". These text-formatted samples contain elements of tables, titles, and paragraphs.

[0133] Steps 1-4, automatically select a loader for preloading according to the document format. For the document "Yongshanqiao Employee Handbook.pdf", the PDF Loader named "MinerUParseLoader" will be used to load the document, identify tables, headings, paragraphs, pictures, etc. in it. In order to restore the results in the retrieval stage, this loader needs to convert the identified text into markdown format;

[0134] Steps 1-5, add hot words to the hot word library that need to be concerned about, such as: "Qiao Chu File". For the document "Telemarketing Record.mp3", the AudioLoader will be used for loading, and the text of the audio file will be transcribed and the vocabulary corrected before outputting;

[0135] Steps 1-6, for "Warehouse Personnel In-and-Out Record.mp4", use the VideoLoader for loading. This loader extracts key frames from the video in seconds and outputs descriptions of key frame images in chronological order through prompt engineering for generating the main content of this file;

[0136] Steps 1-7, split and chunk the loaded document. The specific steps are as follows:

[0137] Steps 1-7-1, select a suitable text splitter according to the configuration. For example, the selected tokenizer is RecursiveCharacterMarkdownSpliter, and this tokenizer will split the loaded document according to the markdown text format.

[0138] Steps 1-7-2, set the relevant parameters for file processing, including the character list, the maximum length of a single-segment document, and the overlapping length of adjacent chunks. For example, set the character list to ["\n\n","\n","",""], the maximum length of a single-segment document to 1024, and the overlapping length of adjacent chunks to 512.

[0139] Steps 1-7-3, execute the chunking logic. The processing results for "Employee Handbook.pdf" and "Telemarketing Record.mp3" are shown in Table 1 below.

[0140] Table 1

[0141]

[0142] It can be seen that the current document is split into two chunked documents, using Markdown as the output format for document parsing, and its recognition results are consistent with the content of the original document.

[0143] Steps 1-8, add context information to the text chunks. The specific steps are as follows:

[0144] Step 1-8-1: Based on the context window size of 2, select the information of the two blocks before and after the current block.

[0145] Step 1-8-2: The selected information is merged and stored as the context metadata information of the current block. The purpose is to reduce the semantic loss caused by the block splitting.

[0146] Step 1-8-3: Add metadata information to the text blocks, specifically including the following steps:

[0147] Step 1-8-4: Set the number of generated keywords to 5, and generate keywords corresponding to the blocks;

[0148] Step 1-8-5: Generate the summary information corresponding to the block;

[0149] Step 1-8-6: Add filename information to the block information;

[0150] Step 1-8-7: Add topic-related information to the segmented information;

[0151] Steps 1-9: After the above steps, the final block information generated for the documents "Employee Handbook.pdf" and "Telemarketing Records.mp3" is shown in Table 2 below.

[0152] Table 2

[0153]

[0154]

[0155] Steps 1-9: Based on the BGE model, the segmented documents are used as input to generate corresponding semantic vectors.

[0156] Steps 1-10: Use a vector library to store the generated semantic vectors.

[0157] As can be seen, the results include segmentation information, and the hot words set in the audio loader are also correctly identified. Since the construction of the knowledge graph requires domain knowledge as a foundation, the above example does not involve the knowledge graph construction process.

[0158] To more clearly illustrate the technical solution of this embodiment, the following describes in detail the specific implementation of an enterprise-level knowledge management system based on a large language model, using a real-world application case from a large manufacturing enterprise:

[0159] 1. Knowledge Storage Module: After system deployment, the first step is to import the company's existing document data into the knowledge storage module. The company's document types include PDF, Word, audio (MP3 format), and video (MP4 format), with a total data volume of 2.8TB. The administrator categorizes these documents by type and usage scenario, mainly into the following categories: technical manuals, customer feedback, operation guides, and internal training materials.

[0160] 1.1 Data types: PDF documents (5000 documents, approximately 800GB), Word documents (2000 documents, approximately 200GB), audio files (300 hours, approximately 550GB), video files (100 hours, approximately 530GB).

[0161] 1.2 Access Control: The system assigns role-based access permissions to different departments. For example, the technical support department can access all technical manuals and operating guides, but cannot view sensitive customer feedback records; while management can access all documents, including customer feedback.

[0162] 1.2.1 The access control is based on a role-based chain structure, ensuring that different departments and users can access documents appropriate to their roles. By subdividing roles, such as ordinary employees, department managers, and project administrators, the system can achieve more precise access control. Ordinary employees can only access public documents, such as company announcements and basic operating procedures. They cannot access any documents involving customer privacy or internally sensitive information. Department managers can access all documents within their department, including technical manuals and customer feedback. Their access level is higher than that of ordinary employees, but they still do not have the full access rights of management. Project administrators are responsible for managing specific projects and therefore can access technical manuals, operating guidelines, and feedback information related to that project.

[0163] 1.2.2 This module supports the creation and management of custom roles. Administrators can define different access levels and assign corresponding permissions according to enterprise needs. Furthermore, changes and deletions of roles will automatically trigger corresponding permission adjustments, ensuring the dynamic nature and security of permission management.

[0164] 1.2.3 The knowledge access management module implements multi-layered security measures, including TLS / SSL encryption, authentication mechanisms, and access auditing functions. All data access and modification operations are logged and monitored to ensure that improper behavior can be detected and dealt with in a timely manner.

[0165] 1.3 Results: All documents were successfully imported into the system, and the system generated metadata for each document, including document type, upload time, file size, etc. The documents were stored in encrypted form to ensure data security.

[0166] 2. Document Loading Module: After the system connects to the knowledge parsing module, it begins to parse and process documents of different formats.

[0167] 2.1 Electronic Document Parsing: Taking the "Product Model A Operation Manual" (PDF format, 100 pages) as an example, the system uses the PDFLoader to parse the PDF file, extracting technical steps, warning messages, and experimental data tables. The document has a complex structure, containing a mix of images, tables, and text. The system employs OCR technology to recognize the text in the images, achieving an accuracy of 98.5% and a recognition speed of approximately 1.5 seconds per page, completing the entire document parsing in 145 seconds.

[0168] Analysis parameters: The character recognition accuracy of the OCR model is 98.5%, and the processing speed is 1 page / second; the table extraction accuracy is 95%, and the extraction accuracy for complex tables (such as tables with merged cells) is 90%.

[0169] Storage Results: The system generated an index entry for this document, extracting key information including: motor maintenance steps, voltage adjustment methods, and troubleshooting suggestions. The extracted tabular data includes technical specifications, such as "the motor's maximum power is 5kW, and its rated voltage is 220V."

[0170] 2.2 Multimedia Document Parsing: For customer feedback audio (MP3 format, 30 minutes in length), the system uses a speech-to-text model to convert the customer dialogue in the audio file into text with a recognition accuracy of 95%, and extracts the speaker's identity and timestamp.

[0171] Speech recognition parameters: The speech recognition accuracy of the speech-to-text model is 95%, and the processing time is 1 / 10 of the multimedia duration, that is, each minute of audio is processed within 6 seconds.

[0172] Result: The system generated customer feedback in text format, and marked the speaker's identity and timestamp information.

[0173] 2.3 Video Parsing and Processing: For video file parsing, such as "Employee Training Video.mp4" (45 minutes long), the system uses the VideoLoader to process the video content through frame extraction and OCR recognition technology. The system divides the video into frames per second and uses an OCR model to extract text content from the text information in each frame. Simultaneously, the system uses video analysis technology to identify key figures and actions appearing in the video for subsequent content retrieval.

[0174] Processing parameters: Video frame extraction rate is 1 frame / second; OCR recognition accuracy is 98.5%, consistent with PDF text. Recognition of key elements in the video is based on a deep learning model, with an accuracy rate of 90%.

[0175] Results: Video parsing generated multiple document entries, including key dialogues, important presentation charts, and relevant video timeline segments. This processed information will be stored in the system as text and images for subsequent searching and knowledge structure construction.

[0176] 3. Document Chunking Module: After the documents are loaded, the system performs detailed chunking processing on each document. Based on the file content, processing needs, and context information, the system divides technical manuals, customer feedback, and training video content into smaller text blocks for easier subsequent management and retrieval.

[0177] 3.1 Processing Mechanism: The system divides the text into blocks according to a preset character length (usually 1024) and context identifiers. For example, for the technical steps section of the "Product Model A Operation Manual", the block result is as follows:

[0178] Section 1: "For motor maintenance, the first step is to disconnect the power and use specialized tools for cleaning and inspection to avoid damaging the equipment."

[0179] Segment 2: "The motor's maximum power is 5kW, its rated voltage is 220V, and its maintenance cycle is once a month." This approach ensures that users don't have to carry all the data from every document when conducting a topic search, allowing them to quickly obtain the information they need.

[0180] 4. Data Augmentation Engine Module: After parsing all documents, the system uses the data augmentation engine module to organize and optimize the information.

[0181] 4.1 Knowledge Graph Construction: The system combines the enterprise's domain knowledge to construct domain knowledge graphs for different types of documents. Terms and definitions in technical manuals are linked to experimental data, and common problems in customer feedback are mapped to solutions, forming efficient information retrieval and problem-solving paths.

[0182] During the knowledge graph construction process, the system will structure different types of document data through information extraction and relationship modeling, thereby forming a graph that supports intelligent retrieval and knowledge association. This process includes the following steps:

[0183] 4.1.1 The system first extracts key information from the imported documents, forming a knowledge representation based on triples (subject, predicate, object). For example, in a technical manual, the following triples might be extracted:

[0184] (Motor A, maximum power, 5kW), (Motor A, rated voltage, 220V), (Motor A, maintenance cycle, once a month), (Troubleshooting, recommended tools, special tools), (Technical manual, belongs to, product knowledge base)

[0185] 4.1.2 After extracting the triples, the system will define the relationships between the entities. These relationships help users better understand the connections between documents. For example:

[0186] "Motor A" may have a "need" relationship with "trouble handling", that is: (Motor A, need, trouble handling).

[0187] "Motor A" can also be linked to related "Frequently Asked Questions" to help users quickly find solutions when searching.

[0188] 4.1.3 The system will construct a knowledge graph from the extracted triples and their relationships. In the graph, different nodes represent entities (such as motors, operation manuals, fault handling solutions, etc.), while edges represent the relationships between them (such as containment, need, belonging, etc.).

[0189] 4.1.4 As new documents are added to the knowledge base, the system will automatically update the knowledge graph. For example, after adding a new technical manual about "Motor B", the system will extract its triples and associate them with relevant nodes in the existing knowledge graph. In this way, the knowledge graph will continuously evolve and be enriched, making it a dynamic carrier of enterprise knowledge.

[0190] 4.2 Alignment and Translation: As the company has expanded into overseas markets, the system will perform semantic vector alignment on multilingual documents to enable global employees to access shared knowledge. For example, product manuals in French and German will be aligned with the English version to ensure consistency.

[0191] 5. Multilingual Processing Results: Through a semantic alignment model, the system automatically aligns the French and German versions of the user manual to the English version, ensuring that all language versions have the same semantics and content structure. The translation accuracy is close to that of a human, with a semantic accuracy rate of 95%.

[0192] The split document blocks, enhanced by the data augmentation engine, are then input into a semantic vector generation module based on the BGE model, achieving fast and accurate semantic representation. The semantic vector of each text block not only incorporates contextual information but can also be fine-tuned based on user-injected industry-specific vocabulary to meet the needs of different business scenarios.

[0193] 6. Private Deployment and Security: The system is deployed in a fully private environment within the enterprise's own data center, ensuring physical data security and access control. Administrators can flexibly define access permissions and user roles as needed to comply with the enterprise's internal security policies.

[0194] Ultimately, the large-scale application of the entire system not only improved the efficiency of knowledge management within the enterprise but also significantly reduced the operational and maintenance burden. While ensuring information security, employees at all levels of the enterprise can quickly search for and share the information they need, significantly improving work efficiency and cross-departmental collaboration capabilities.

[0195] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention content of the multimodal AI knowledge base construction system for private deployment provided by this invention, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0196] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0197] This invention provides a multimodal AI knowledge base construction system for private deployment. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A multi-modal AI knowledge base construction system for private deployment, characterized in that, The system comprises a knowledge storage module, an intelligent document loading module, a document segmentation engine module, a data enhancement engine module, a multi-language semantic vector alignment module, and a private deployment module. The knowledge storage module comprises a knowledge permission management submodule and a knowledge source management submodule. The intelligent document loading module pre-processes different types of files through a document loader, preloads files, and analyzes formats to identify and extract high-quality content from documents. The document segmentation engine module automatically segments large blocks of text into smaller, more reasonable segments by constructing a lightweight document segmenter. The data enhancement engine module uses data enhancement techniques and knowledge graph methods to enhance the data of segmented content and increase semantic features. The multi-language semantic vector alignment module uses Embedding models and multi-modal model capabilities to convert text and multimedia files into vector representations. The private deployment module provides a localized deployment interface for enterprise private data and AI models, and only authorized personnel can access sensitive information.

2. The system of claim 1, wherein, The knowledge permission management submodule supports uploading, updating, deleting, and sharing of electronic documents and multimedia documents, and has permission control to ensure information security. The knowledge permission management submodule uses a fine-grained permission control mechanism that combines role-based access control and attribute-based access control to implement document-level access control. The knowledge permission management submodule uses TLS / SSL encryption protocols during data transmission. For sensitive data stored in the system, the system uses a private file cabinet mechanism for encrypted storage, uses encryption algorithms to encrypt sensitive data, and implements key segmentation management. The system also has a data backup and recovery mechanism.

3. The system of claim 2, wherein, The knowledge permission management submodule has an operation record function that permanently records the entire knowledge management process, including operation time, operation type, operator identity, operation content, and operation results. The recorded content is stored in an encrypted manner to ensure that it cannot be tampered with and is regularly archived and backed up. The knowledge permission management submodule integrates a system user behavior monitoring function to monitor user system operation behavior in real time and identify abnormal operations. The system user behavior monitoring function can periodically detect permission conflicts, missing permissions, and lack of permissions, optimize permission allocation through a permission conflict detection mechanism and regular permission review, and provide real-time alerts and post-event tracing functions. The intelligent document loading module uses a document loader to preload documents. The intelligent document loading module implements a cross-modal document structured analysis technology, and the document loader includes a PDF Loader, an ImageLoader, a VideoLoader, and an AudioLoader. The PDF Loader loader is responsible for loading the content of the document class file, and the content includes text, title, picture, table, title, formula; The ImageLoader loader is responsible for identifying the text content and description contained in the picture; The VideoLoader loader is responsible for key frame extraction of video files through software or hardware acquisition devices, and identifies and generates corresponding pictures and descriptions; The AudioLoader loader is responsible for identifying audio files, and the content includes text, summary and topic analysis, keywords, entity and custom tag extraction.

4. The system of claim 3, wherein, The PDF Loader loader fine-tunes the layout detection model using annotations of PDF documents in two or more formats; the layout detection model used by the PDF Loader loader is the LayoutLMv3 model, which performs the following steps: Step 1-1, perform PDF file page reading; Step 1-2, locate different elements in the document based on the fine-tuned LayoutLMv3 model, including images, tables, text, titles, and formulas; Step 1-3, identify the formula image as LaTeX source code through UniMERNet; Step 1-4, detect and OCR recognize the text and text box, wherein the OCR recognition is trained and fine-tuned based on the model of PaddleOCR, and returns the text box coordinates and recognition result, and the OCR recognition includes the following steps: Step 1-4-1, text detection of the document to obtain text box coordinates; Step 1-4-2, rotate and crop each text box; Step 1-4-3, text recognition of the cropped document; Step 1-4-4, filter out the recognition results with confidence lower than the specified threshold; Step 1-5, perform layout analysis on the document image to identify different types of regions; Step 1-6, parse the document content and process the identified regions, including the following steps: Step 1-6-1, use the LORE table structure recognition model to identify the elements of the table region, the LORE table structure recognition model combines the spatial position and logical position of the table cells together for end-to-end modeling and prediction, the LORE table structure recognition model uses a key point segmentation network based on a convolutional neural network to extract visual features, and through two regression heads, the spatial position and logical position of the cells are predicted; finally, the LORE table structure recognition model restores the row and column structure of the entire table and outputs a machine-readable representation; Step 1-6-2, output the content of the cell and the spatial position and logical position of the cell; Step 1-7, fuse the results of layout analysis and OCR recognition, and introduce a layout restoration process to convert the document into markdown format for output.

5. The system of claim 4, wherein, Step 1-5 includes: Step 1-5-1, group the tables, titles, and paragraphs output by layout analysis; Step 1-5-2, merge the text in the group according to the spatial position and logical position; Step 1-5-3, convert the merged text into markdown according to the type.

6. The system of claim 5, wherein, The lightweight document segmenter includes a recursive character text segmenter, a text semantic segmenter, a fixed-size character text segmenter, a document structure segmenter, a large language model (LLM) segmenter, and a custom segmenter, and is configured to divide the text output by the intelligent document loading module into small blocks; The recursive character text segmenter divides the text into block documents with a length smaller than that of the original document by specifying a block length and a set of delimiters, pre-segmenting the text according to the priority order of the delimiters, merging the block documents with a length not exceeding the specified block length, and recursively splitting the text exceeding the specified block length until the desired block size is obtained. The text semantic segmenter divides the text by identifying breakpoints, and the breakpoints are determined as follows: if the distance between the embedding vectors of two consecutive paragraphs exceeds a specified threshold, a breakpoint is set between the two consecutive paragraphs to divide the text. The fixed-size character text segmenter divides the text into small segments with a consistent size according to a predetermined number of characters, words, or tokens, and preserves the context continuity by overlapping. The document structure segmenter divides the text according to the inherent structure of the document, with each structural unit serving as a block. When the size of the document chapters varies and exceeds the size limit of the block, the recursive character text segmenter is used to split and merge the text. The document structure segmenter maintains the structural integrity by aligning with the logical parts of the document. The LLM segmenter directly inputs the original document into a large language model (LLM) to intelligently generate semantic blocks. The LLM segmenter dynamically divides the text based on its semantic understanding capabilities. The custom segmenter dynamically divides the document based on user-defined characters or strategies.

7. The system of claim 6, wherein, The AudioLoader loader performs the following steps: Step 2-1, load the speech recognition model and set the hotword library; the speech recognition model performs the following steps: Step 2-1-1, extract the acoustic feature vector corresponding to the target text by the built-in Predictor module based on the continuous integration predictor; Step 2-1-2, sample the acoustic feature vector and target text vector into a feature vector containing semantic information using the built-in Sampler, and use a bidirectional decoder to enhance the model's ability to model the context; Step 2-1-3, perform speech recognition model training based on the MWER training criterion for negative sample sampling; Step 2-2, load the audio file and perform text recognition on the audio using the speech recognition model to output the recognized text result; Step 2-3, correct the text result; Step 2-4, perform summary and topic analysis, keyword and entity extraction on the text result using the prompt word engineering; Step 2-5, extract the system tag information contained in the text result.

8. The method of claim 7, wherein, The data augmentation engine module performs the following steps: Step 3-1, extract metadata information, including the following steps: Step 3-1-1, extract the page number, file name, author, and timestamp corresponding to the block segment; Step 3-1-2, text preprocessing: removing stop words and punctuation; Step 3-1-3, generating summary information of the segmented fragment; Step 3-1-4, using BERT to extract document embedding to get document-level representation, extracting word embedding of N-gram word or phrase, and using cosine similarity to find the most similar word or phrase to the document; Step 3-1-5, generating a specified number of keywords; Step 3-1-6, generating a specified number of questions related to the segmented fragment; Step 3-2, cross-page table merging technology, combined with LORE table structure recognition model to handle cross-page table merging problem, including the following steps: Step 3-2-1, based on layout recognition judgment method, requiring correct identification of layout and removal of header and footer of each page, judging the type of the last element of the current block and the first element of the next block, if both are table type, it is determined to be a continuation of the cross-page table; Step 3-2-2, visual continuity analysis, using object detection model to identify table bounding box, calculating the geometric overlap rate of the upper and lower page tables, when the overlap rate is greater than the threshold, it is determined to be a cross-page table, wherein the calculation method of the overlap rate X1 is: X1=X2 / min(X3,X4), Where X2 represents the area of the overlapping region, X3 represents the area of the upper page table, and X4 represents the area of the lower page table; Step 3-2-3, logical continuity verification: through table header semantic similarity check, when the semantic similarity of the description information of the first table and the description information of the second page table is greater than the threshold, it is determined to be a cross-page table; Step 3-2-4, merging the table corresponding to the segmented fragment that meets the determination rules of steps 3-2-1 to 3-2-3 into a complete table and storing it as a block; Step 3-3, supplementing context information for each segmented fragment, based on large language model LLM, providing each document fragment with an additional explanation through prompt word engineering; Step 3-4, Raptor enhancement: through recursive embedding, clustering and summarization, the original segmented fragment is processed to realize an information tree containing different granularities and different semantic spans, including the following steps: Step 3-4-1, converting the segmented fragment into vector form through the embedding model; Step 3-4-2, grouping and clustering the segmented fragments, then using prompt word engineering to summarize the original text and integrate new blocks, using Gaussian Mixture Model GMM, setting the data points to be generated by one or more Gaussian distributions, and estimating the parameters of the Gaussian distribution of each cluster by maximum likelihood estimation method, the probability density function of the Gaussian distribution is: Where f(x; μ,∑) is the Gaussian probability density function, x is the target data point, n represents the dimension of variable x, μ is the mean vector,∑ is the covariance matrix, |∑| is the determinant of the covariance matrix; exp is the natural exponential function; T represents matrix transpose; this formula describes the probability density function of x under the given mean μ and covariance matrix∑; The goal of Gaussian Mixture Model GMM is to maximize the likelihood function of the data: where L denotes a likelihood function, N denotes a number of data points, K denotes a number of Gaussian distributions, x i is the i-th data point, μ j denotes a mean vector of the j-th Gaussian distribution, π j denotes a weight of the j-th Gaussian distribution, ∑ j denotes a covariance matrix of the j-th Gaussian distribution, and the parameters of the Gaussian Mixture Model (GMM) are solved using an Expectation Maximization algorithm; the Expectation Maximization algorithm comprises the following steps: Step 3-4-2-1, initialize the parameters of the Gaussian Mixture Model (GMM); Step 3-4-2-2, calculate the response of all data points; Step 3-4-2-3, update the parameters of each Gaussian distribution; Step 3-4-2-4, calculate the likelihood function of the model; Step 3-4-2-5, determine whether the parameters of the model have converged, if converged, end the iteration; otherwise, return to step 3-4-2-2; Through continuous iteration, the data is clustered by the GMM clustering algorithm; Step 3-4-3, repeat step 3-4-2 until the text cannot be further divided, i.e., the number of blocks is 1, and each execution of step 3-4-2 is the construction of a layer of nodes. For non-leaf nodes, non-leaf nodes store their own index, child nodes, summarized original text, and corresponding vector; Step 3-5, for a specific domain, define the domain ontology, extract entities and relationships from the segmented data through a large language model, construct a domain knowledge graph based on deep learning and large model technology, use Leiden technology to perform hierarchical clustering on the initial knowledge graph and generate community summaries. The construction of the domain knowledge graph includes the following steps: Step 3-5-1, define the domain ontology: establish an ontology to describe the concepts and relationships in the domain, use the knowledge of domain experts and publicly available domain literature to define the hierarchical relationships, attributes, and constraints of the concepts; Step 3-5-2, entity recognition and relationship extraction: use information extraction and large model technology to identify specified types of entities from segmented text and multi-modal data, and extract potential relationships between entities; Step 3-5-3, construct the knowledge graph: generate a preliminary knowledge graph based on the extracted entities and relationships, use a graph database to store entities and relationships in the form of nodes and edges, and maintain the dynamic updating and scalability of the graph; Step 3-5-4, hierarchical clustering of the knowledge graph: apply the Leiden algorithm to perform hierarchical clustering analysis on the nodes of the preliminary knowledge graph to form a community-structured concept set; Step 3-5-5, generate community summaries: automatically generate summaries for community nodes based on hierarchical clustering results, and use language models to summarize the main themes and important information of each community.

9. The system of claim 8, wherein, The multi-language semantic vector alignment module performs the following steps: Step 4-1, after loading the PDF file based on the PDF Loader loader, use the corresponding block to block the text content and enhance the context; Step 4-2, after loading the picture and video files based on the ImageLoader loader and VideoLoader loader, generate corresponding text descriptions, use the corresponding block to block the text content and enhance the context; Step 4-3, based on the BGE model, the segmented document is used as input to generate the corresponding semantic vector. The BGE model uses instruction fine-tuning, which performs the following steps: Step 4-3-1, generate corpus, fine-tuning dataset format is [query, positive sample set, negative sample set], wherein, query is query text, positive sample set represents text related to query, and negative sample set represents text unrelated to query; Step 4-3-2, set fine-tuning parameters; Step 4-3-3, fine-tune Embeding model to regard semantic correlation task as a binary classification task, use cross-entropy loss function as loss function, use back propagation for model parameter update, and output fine-tuned model; the cross-entropy loss function is: where H(p, q) is the cross-entropy of the probability distribution q and the probability distribution p, p(x i ) represents the probability value of the i-th sample under the probability distribution, q(x i ) represents the probability value of the i-th sample under the probability distribution q, and n is the total number of samples; Step 4-4, store semantic vectors and original texts into vector library and file storage.

10. The system of claim 9, wherein, The private deployment module supports the localization deployment of enterprise internal data and AI models; combined with the data storage module, the system provides a user management mechanism, allows enterprises to customize access permissions according to their own needs, ensures that only authorized users can access sensitive information, introduces a log recording and auditing mechanism to monitor and record system and user behavior in real time, and provides system backup and recovery functions.

Images