Book data processing and intelligent service system based on artificial intelligence

Abstract

The invention discloses a book data processing and intelligent service system based on artificial intelligence, and relates to the technical field of artificial intelligence and digital library crossing, and the technical scheme is characterized in that a multi-format document intelligent conversion and regular cleaning mechanism is adopted, a multi-level metadata extraction algorithm is combined, and a structured knowledge base is accurately constructed; vector semantic retrieval, BM25 keyword retrieval and a cross encoder reordering model are creatively fused, and high-precision context retrieval is achieved through dynamic weight configuration and locality sensitive hash de-duplication. A parallel data processing architecture supporting GPU acceleration is constructed based on a Chroma vector database, the knowledge response capability of a local large model is enhanced in combination with an RAG technology, and verifiable standardized content is generated. The system is compatible with a Linux / Windows platform and containerized deployment, has hundred million-level data throughput efficiency, can be widely applied to the fields of intelligent libraries, knowledge questions and answers and personalized recommendation, and remarkably improves the retrieval accuracy of book data and the intellectualization level of knowledge services.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of artificial intelligence and digital libraries, and more specifically, to a book data processing and intelligent service system based on artificial intelligence. Background Art

[0002] As the digital transformation of the book industry deepens, book knowledge base systems face new technical bottlenecks in knowledge services and data governance. The current goal of building book knowledge bases is to optimize the entire process of collecting, organizing, and providing knowledge services for book resources through semantic understanding, multimodal retrieval, and knowledge enhancement technologies. After years of informatization, the book industry has accumulated a vast amount of unstructured data (such as reader search logs, full-text e-books, and multi-version metadata), which contains complex semantic associations and specialized terminology. Achieving efficient data classification, precise term location, and cross-modal semantic relationship mining has become a core challenge in the integration of natural language processing and library information technology.

[0003] Current library knowledge base systems generally adopt a single search modality, making it difficult to balance the needs of semantic understanding and precise term matching. Keyword-based Boolean search technology can quickly locate explicit terms, but its generalization capabilities for semantically related content are insufficient, making it prone to missed detections, especially when dealing with metaphorical expressions or cross-domain synonyms. For example, when a user queries "neural network-based reading behavior analysis," traditional keyword search may not be able to effectively capture semantically related but differentially expressed literature such as "deep learning-driven user profile construction." On the other hand, while neural embedding-based vector search technology improves recall through semantic similarity calculations, its recognition accuracy for structured fields (such as ISBN codes and publisher names) is insufficient, resulting in a high false positive rate in specific scenarios. This disconnect between semantic understanding and precise term matching has become a core bottleneck restricting service efficiency in complex query scenarios.

[0004] Existing search-enhanced generation systems typically feed document content into large models as unstructured text snippets, ignoring the semantically constrained value of book metadata (such as title, author, and publication information). When processing queries involving multiple versions of a work or authors with the same name, the system's lack of correlation analysis of structured fields can easily lead to content confusion or misquotes. Furthermore, traditional prompting systems use fixed templates to stitch together search results, without dynamically adjusting context organization strategies based on query type. In open-domain question-answering scenarios, this coarse-grained context integration approach can easily introduce redundant information or result in missing key details, leading to logical deviations and factual errors in the generated content, seriously impacting its credibility in professional contexts.

[0005] Building a large-scale library knowledge base faces significant constraints on data processing efficiency and hardware resources. In traditional pipelines, processes such as multi-format document conversion (e.g., PDF / Word to Markdown), data cleaning (e.g., removing annotation tags), and vectorized calculations often rely on serial processing, failing to fully utilize distributed computing resources. This results in excessively long processing times for terabyte-level data. Furthermore, locally deployed large model services rely on high-end GPUs for real-time inference, and the high memory requirements (typically exceeding 80GB) significantly increase the entry cost for small and medium-sized institutions. This imbalance between resource consumption and computing efficiency makes it difficult for existing technical solutions to support business demands for dynamic updates and real-time responses.

[0006] To address the above issues, recent technological improvements have attempted to seek breakthroughs through hybrid retrieval strategies and model lightweighting, but there are still significant limitations. Although the multi-stage retrieval framework combines the advantages of vector and keyword retrieval, the re-ranking process is inefficient for processing long texts and cannot meet the needs of real-time interaction; although the dynamic weight allocation mechanism can theoretically adaptively adjust the retrieval weight of semantics and terms, in actual operation it still relies on manual experience to set parameters and lacks the automated ability to recognize intent; in addition, although model compression technology lowers the hardware threshold, it sacrifices the depth of semantic understanding, resulting in a double decline in retrieval quality and generation accuracy. These compromise solutions expose the deep contradictions between efficiency, cost and quality in the current technology system.

[0007] Therefore, the present invention aims to provide a book data processing and intelligent service system based on artificial intelligence to solve the above problems. Summary of the Invention

[0008] The purpose of the present invention is to provide a book data processing and intelligent service system based on artificial intelligence. The present invention addresses the problems of low retrieval efficiency, insufficient semantic understanding and limited knowledge generation accuracy in traditional book information management systems, and proposes an "AI+book" intelligent application technology solution based on hybrid retrieval and retrieval enhancement generation. By integrating multimodal data processing, dynamic weighted hybrid retrieval and structured knowledge enhancement generation mechanism, it aims to achieve automated management of book resources, high-precision semantic retrieval and intelligent knowledge services, and significantly improve the efficiency of book information processing and the quality of generated content.

[0009] The above technical objectives of the present invention are achieved through the following technical solutions: an artificial intelligence-based book data processing and intelligent service system, comprising a multi-format document processing module, a hybrid retrieval module, a vector database module and a RAG generation engine module;

[0010] The multi-format document processing module is used to automatically convert the format of uploaded book documents and generate structured data of book documents through regular expression cleaning and multi-priority metadata extraction algorithm;

[0011] The hybrid retrieval module is used to receive user query requests, obtain semantic complexity by analyzing the query requests, and then output high-precision retrieval results using dynamic weight allocation and local sensitive hashing deduplication mechanism;

[0012] The vector database module is used to build a distributed vector database that supports GPU acceleration, and realizes real-time storage and retrieval optimization of hundreds of millions of book data;

[0013] The RAG generation engine module is used to deploy a localized large model, generate metadata-enhanced contextual prompts in query requests by calling high-precision retrieval results in the hybrid retrieval module, and output verifiable knowledge response content.

[0014] The present invention is further configured as follows: the multi-format document processing module includes a priority configuration unit and a regular cleaning unit;

[0015] The priority configuration unit is used to assign priority extraction weights to the structured data of the book document according to preset rules;

[0016] The regular expression cleaning unit is used to remove redundant characters and unstructured noise in the book document through regular expression matching.

[0017] The present invention is further configured as follows: the hybrid retrieval module includes a vector semantic retrieval unit, a keyword retrieval unit and a cross encoder reordering unit;

[0018] The vector semantic retrieval unit is used to dynamically adjust the weight ratio of the vector retrieval according to the semantic complexity of the user query;

[0019] The keyword search unit is used to dynamically adjust the weight ratio of keyword search according to the semantic complexity of the user query;

[0020] The cross-encoder re-ranking unit is used to re-rank the retrieval results by relevance scores through a cross-encoder to screen the Top-K candidate sets.

[0021] The present invention is further configured as follows: the vector database module includes a fast deduplication unit and a metadata indexing unit;

[0022] The fast deduplication unit is based on a local sensitive hash deduplication mechanism to eliminate duplicate block data;

[0023] The metadata indexing unit is used to establish a multi-level index through fields in the structured data to speed up retrieval.

[0024] The present invention is further configured as follows: the RAG generation engine module includes a context enhancement unit and a verification interface unit;

[0025] The context enhancement unit is used to inject metadata in the search results into the prompt template;

[0026] The verification interface unit is used to call a preset rule library to verify the logical consistency and factual accuracy of the generated content.

[0027] The present invention also provides a book data processing method based on artificial intelligence, comprising the following steps:

[0028] S1. Receive heterogeneous book document input, perform format conversion and metadata extraction in the book documents, and generate a structured knowledge base;

[0029] S2, triggering hybrid search based on user queries, generating candidate sets by combining semantic and keyword results;

[0030] S3, call the vector database to retrieve relevant data blocks and build context-enhanced prompts;

[0031] S4. Generate response content through the local large model and return the verified normalized output.

[0032] The present invention is further configured as follows: the process of format conversion in step S1 includes the following steps:

[0033] S11, receiving uploaded book documents and storing them in a designated directory;

[0034] S12, calling a document conversion tool to batch convert non-Markdown format files in the book document into unified Markdown format files;

[0035] S13. Convert the Markdown format file into a structured HTML format file, and then use regular expressions to clear comments in the HTML format file.

[0036] The present invention is further configured such that: the metadata in the book document in step S1 includes the title, author, publisher, ISBN and publication date.

[0037] The present invention is further configured as follows: the weight distribution formula for the mixed search in step S2 is:

[0038] Score final =α·sim vec +(1-α)·BM25(α∈[0.5,0.8])

[0039] Among them, Score final Indicates the final score of hybrid retrieval, sim vec represents vector retrieval similarity, BM25 represents keyword retrieval score, and α represents the dynamic weight coefficient.

[0040] The present invention is further configured as follows: the construction of the vector database in step S3 includes the following steps:

[0041] S31. Generate text vectors using pre-trained semantic encoding models;

[0042] S32. Build a vector database based on Chroma and define a similarity threshold;

[0043] S33. Filter low-correlation data through similarity threshold and use local sensitive hashing algorithm to achieve document deduplication.

[0044] The present invention also provides an artificial intelligence-based book data processing device, comprising at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement an artificial intelligence-based book data processing method.

[0045] The present invention also provides a computer-readable storage medium, which stores computer instructions. The computer instructions are used to be executed by the computer to implement a book data processing method based on artificial intelligence.

[0046] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, an artificial intelligence-based book data processing method is implemented.

[0047] In summary, the present invention has the following beneficial effects:

[0048] 1. This invention uses hybrid retrieval technology to dynamically integrate the advantages of vector retrieval and BM25 keyword retrieval, combined with a cross-encoder reranking model, to achieve an MRR@10 of 0.82 for retrieval results, a 35% improvement over single retrieval methods. It also uses locality-sensitive hashing for efficient deduplication, introduces threshold filtering and a degraded return mechanism, and ensures that retrieval results have both high recall and high relevance.

[0049] 2. This invention combines metadata-enhanced prompt templates with a local large model to effectively suppress the hallucination phenomenon of the language model, achieving a 92% factual consistency in the generated content. At the same time, the dynamic prompt template integrates book attributes (such as title, author, and publisher) with the text content, constraining the generation logic of the language model and avoiding content confusion or incorrect citations.

[0050] 3. This invention uses a 4-bit quantization model to increase inference speed by 2.3 times and reduce video memory usage by 58%, significantly reducing the hardware cost of deploying large models for small and medium-sized institutions. The vector database module supports GPU acceleration and implements document deduplication through the MinHash algorithm, effectively ensuring retrieval efficiency and data consistency.

[0051] 4. The system of the present invention is compatible with Linux and Windows platforms and supports containerized deployment. It has a data throughput efficiency of billions of data points and can meet the dynamic update and real-time response requirements of large-scale book knowledge bases. Its modular architecture design provides flexibility for multi-scenario adaptation and can be widely used in smart libraries, knowledge quizzes, personalized recommendations and other fields.

[0052] 5. This invention uses a multi-round interactive context consistency maintenance mechanism implemented through a dialogue management module to ensure the logic and consistency of the question-answering system. The dynamic adjustment strategy of the prompt template can optimize the context organization according to the query type, avoiding redundant information interference or missing key details.

[0053] 6. The present invention unifies heterogeneous documents into a standardized Markdown format through multi-format document conversion and regular cleaning mechanisms, and combines multi-level metadata extraction algorithms to significantly improve the processing efficiency of unstructured data. The parallel processing technology and distributed vector database design support the rapid processing and storage of TB-level data, which can meet the needs of large-scale knowledge base construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a schematic diagram of the module structure of an artificial intelligence-based book data processing and intelligent service system in Example 1 of the present invention;

[0055] Figure 2 1 is a flow chart of a method for processing book data based on artificial intelligence in Example 2 of the present invention;

[0056] Figure 3 1 is a schematic diagram of the steps of book data processing in Example 2 of the present invention;

[0057] Figure 4 Schematic diagram of the process of constructing a vector database in Example 2 of the present invention;

[0058] Figure 5 This is a schematic diagram of the hybrid search process in Example 2 of the present invention;

[0059] Figure 6 Schematic diagram of the workflow of the RAG system in Example 1 of the present invention. DETAILED DESCRIPTION

[0060] The following is combined with Figure 1-6 The present invention is described in further detail.

[0061] Example 1: A book data processing and intelligent service system based on artificial intelligence, including a multi-format document processing module, a hybrid retrieval module, a vector database module and a RAG generation engine module; the multi-format document processing module is used to automatically convert the format of uploaded book documents, and generate structured data of book documents through regular expression cleaning and multi-priority metadata extraction algorithms; the hybrid retrieval module is used to receive user query requests, derive semantic complexity by analyzing the query requests, and then use dynamic weight allocation and local sensitive hashing deduplication mechanism to output high-precision retrieval results; the vector database module is used to build a distributed vector database that supports GPU acceleration to achieve real-time storage and retrieval optimization of hundreds of millions of book data; the RAG generation engine module is used to deploy a localized large model, generate metadata-enhanced context prompts in the query request by calling the high-precision retrieval results in the hybrid retrieval module, and output verifiable knowledge response content.

[0062] Preferably, in this embodiment, the multi-format document processing module includes a priority configuration unit and a regular cleaning unit; the priority configuration unit is used to assign priority extraction weights to the structured data of the book document according to preset rules; the regular cleaning unit is used to remove redundant characters and unstructured noise in the book document through regular expression matching.

[0063] Preferably, in this embodiment, the hybrid retrieval module includes a vector semantic retrieval unit, a keyword retrieval unit and a cross-encoder reordering unit; the vector semantic retrieval unit is used to dynamically adjust the weight ratio of the vector retrieval according to the semantic complexity of the user query; the keyword retrieval unit is used to dynamically adjust the weight ratio of the keyword retrieval according to the semantic complexity of the user query; the cross-encoder reordering unit is used to reorder the retrieval results by relevance scoring through a cross encoder to screen the Top-K candidate set.

[0064] Preferably, in this embodiment, the vector database module includes a fast deduplication unit and a metadata index unit; the fast deduplication unit is based on a local sensitive hash deduplication mechanism to eliminate duplicate block data; the metadata index unit is used to establish a multi-level index through fields in the structured data to accelerate retrieval.

[0065] Preferably, in this embodiment, the RAG generation engine module includes a context enhancement unit and a verification interface unit; the context enhancement unit is used to inject metadata in the retrieval results into the prompt template; the verification interface unit is used to call the preset rule library to verify the logical consistency and factual accuracy of the generated content.

[0066] This embodiment consists of five core modules: hardware environment configuration, local large model deployment, knowledge base construction, hybrid retrieval optimization, and RAG integration. At the hardware level, high-performance GPUs and large-scale storage devices are used to build a computing power foundation to support the efficient operation of localized large language models and vector databases. The model deployment link integrates the Ollama and vLLM frameworks to achieve low-latency reasoning and resource optimization management of open source large models (such as the deepseek series), providing stable computing power support for semantic understanding and generation tasks.

[0067] The knowledge base construction module adopts a multi-stage data processing flow: first, heterogeneous documents are unified into a standardized Markdown format through format conversion and regular expression cleaning; then, a multi-level metadata extraction algorithm is designed to automatically extract key information such as book titles, authors, and publishers from the text based on regular expressions and heuristic rules; finally, vectorized storage of documents is implemented based on the Chroma vector database, and a pre-trained semantic encoding model (such as BAAI / bge-large-zh-v1.5) is combined with parallel processing technology to build a structured knowledge base that supports fast similarity retrieval.

[0068] Hybrid retrieval technology dynamically combines the advantages of vector retrieval and BM25 keyword retrieval, employing an adaptive allocation strategy for semantic and keyword weights, combined with a reranking model to refine and optimize candidate results. During the retrieval process, the system uses locality-sensitive hashing (LSH) for efficient deduplication and introduces threshold filtering and a degraded return mechanism to ensure that search results have both high recall and high relevance. For specific scenarios such as summary generation, the system can dynamically expand the search scope and adjust the weight configuration to further enhance the completeness of contextual coverage.

[0069] The RAG system integrates hybrid retrieval, prompt engineering, and language model interfaces through a modular architecture: the hybrid searcher provides multimodal knowledge input; the prompt engineering module innovatively constructs metadata-enhanced structured templates, integrating book attributes and text content into standardized prompts to constrain the generation logic of the language model; the language model interface calls local large models (such as deepseek-r1) based on standardized APIs, generates diversity through temperature parameter regulation, and combines with the dialogue management module to maintain contextual coherence in multiple rounds of interaction.

[0070] The present invention significantly improves the intelligent level of book information processing through the collaborative optimization of multiple technologies. The automated metadata extraction and vectorized storage mechanism of the knowledge base construction module solves the problem of low efficiency in unstructured data processing; the hybrid retrieval technology significantly improves the retrieval accuracy compared to the single mode through the dynamic fusion of semantics and keywords; the RAG system combines structured prompts with metadata enhancement to effectively suppress language model hallucinations, and significantly enhances the factuality and verifiability of the generated content. In addition, localized model deployment and GPU acceleration design ensure the real-time response capability of the system, and the scalable modular architecture provides flexibility for multi-scenario adaptation, providing efficient and reliable technical support for applications such as library digital services and intelligent question-answering systems.

[0071] Example 2: A book data processing method based on artificial intelligence, comprising the following steps:

[0072] S1. Receive heterogeneous book document input, perform format conversion and metadata extraction in book documents, and generate a structured knowledge base.

[0073] In this embodiment, the specific operation process of multi-format data conversion is as follows:

[0074] Receive PDF, Word, Excel, TXT, and Markdown format files uploaded by users and store them in a specified directory; call document conversion tools (such as `docling`);

[0075] Convert non-Markdown files into unified Markdown format in batches;

[0076] Use regular expression `re.sub(r'<!--.*?-->',',text,flags=re.DOTALL)` to remove HTML comments.

[0077] In metadata extraction, book title extraction is performed according to the priority matching rules:

[0078] Explicit label: `r"Book Title[::]\s*([^\n]+)"`;

[0079] First level heading: `r"^#\s*(.*?)[\n]"`;

[0080] · First paragraph interception: `text[:200].split('\n')[0]`;

[0081] ISBN extraction is performed according to the matching standard format;

[0082] `r"ISBN\s*[::]?\s*([0-9\-]{10,17})";

[0083] Finally, the extracted metadata (book title, author, publisher, ISBN, publication date) is stored.

[0084] In this embodiment, the specific operation process of constructing the vector database is as follows:

[0085] Generate text vectors using the pre-trained model `BAAI / bge-large-zh-v1.5`:

[0086] v=Encoder(text)∈R 768

[0087] Where v represents the generated 768-dimensional semantic vector, Encoder(·) represents the pre-trained semantic encoding model, and text represents the encoded text.

[0088] Build a vector database based on Chroma and define the similarity threshold:

[0089] sim min =0.65

[0090] Among them, sim min Indicates the minimum similarity threshold. Search results below this value will be filtered.

[0091] Document deduplication is achieved through the MinHash algorithm:

[0092]

[0093] Among them, A and B represent the feature sets of two documents respectively, and Jaccard(A,B) is used to calculate document similarity.

[0094] S2. Trigger hybrid retrieval based on user queries and generate candidate sets by combining semantic and keyword results.

[0095] In this embodiment, the hybrid search method includes multimodal search and re-ranking optimization, wherein the operation process of multimodal search is as follows:

[0096] Vector retrieval: Calculate the cosine similarity between the query and the document:

[0097]

[0098] Where q represents the query vector, d represents the document vector, · represents the vector dot product, and |||| represents the vector modulus.

[0099] BM25 search: Calculate relevance based on statistical model:

[0100]

[0101] (Parameters: k1 = 1.2, b = 0.75)

[0102] Among them, f t,d represents the frequency of term t in document d, d| represents the document length (number of words), avgdl represents the average document length of the corpus, k1=1.2, b=0.75 represents the adjustment parameters, Represents the inverse document frequency, N is the total number of documents, n t Represents the number of documents containing the term t.

[0103] Mixed weight distribution: Comprehensive score formula:

[0104] Score final =0.7·sim vec +0.3 BM25

[0105] Among them, Score final Indicates the final score of hybrid retrieval, sim vec represents vector retrieval similarity, and BM25 represents keyword retrieval score.

[0106] The operation process of reordering optimization is as follows:

[0107] Use the cross encoder `BAAI / bge-reranker-large` to fine-rank the Top-K results:

[0108] logits=Sigmoid(W T Encoder([q;d])

[0109] Among them, [q;d] represents the concatenation vector of the query and document, W represents the trainable weight matrix, and the output value range [0,1] represents the relevance probability.

[0110] Combined with TF-IDF keyword matching score:

[0111]

[0112] Among them, TF(t,d) represents the term frequency of term t in document d, DF(t) represents the number of documents containing term t, and N represents the total number of documents in the corpus.

[0113] S3. Call the vector database to retrieve relevant data blocks and build context-enhanced prompts.

[0114] In this embodiment, the operation process of dynamic prompt construction is:

[0115] Generate structured prompt template by combining search results and metadata:

[0116] ```text

[0117] Generate answers based on the following book information:

[0118] Title: {title}

[0119] Author:{author}

[0120] Publisher: {publisher}

[0121] Matching content:{content}

[0122] Question: {query}

[0123] ```

[0124] S4. Generate response content through the local large model and return the verified normalized output.

[0125] In this embodiment, the answer is generated by inputting it into a language model (such as `deepseek-r1:70b`).

[0126] Example 3: A book data processing device based on artificial intelligence, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the processor, the instructions being used to be executed by the processor to implement a book data processing method based on artificial intelligence, the method comprising: receiving heterogeneous book document input, performing format conversion and metadata extraction in the book documents, and generating a structured knowledge base; triggering hybrid retrieval based on user queries, and generating a candidate set by combining semantic and keyword results; calling a vector database to retrieve relevant data blocks and constructing context-enhanced prompts; generating response content through a local large model, and returning verified normalized output.

[0127] Example 4: A computer-readable storage medium stores computer instructions, which are used to be executed by a computer to implement an artificial intelligence-based book data processing method, the method comprising: receiving heterogeneous book document input, performing format conversion and metadata extraction in the book documents, and generating a structured knowledge base; triggering hybrid retrieval based on user queries, and generating a candidate set by combining semantic and keyword results; calling a vector database to retrieve relevant data blocks and constructing context-enhanced prompts; generating response content through a local large model, and returning verified normalized output.

[0128] Embodiment 5: A computer program product includes a computer program, which, when executed by a processor, implements an artificial intelligence-based book data processing method, the method including: receiving heterogeneous book document input, performing format conversion and metadata extraction in the book documents, and generating a structured knowledge base; triggering hybrid retrieval based on user queries, and generating a candidate set by combining semantic and keyword results; calling a vector database to retrieve relevant data blocks and constructing context-enhanced prompts; generating response content through a local large model, and returning verified normalized output.

[0129] This specific embodiment is merely an explanation of the present invention and is not intended to limit the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as such modifications are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A book data processing and intelligent service system based on artificial intelligence, characterized by: It includes multi-format document processing module, hybrid retrieval module, vector database module and RAG generation engine module; The multi-format document processing module is used to automatically convert the format of uploaded book documents and generate structured data of book documents through regular expression cleaning and multi-priority metadata extraction algorithm; The hybrid retrieval module is used to receive user query requests, obtain semantic complexity by analyzing the query requests, and then output high-precision retrieval results using dynamic weight allocation and local sensitive hashing deduplication mechanism; The vector database module is used to build a distributed vector database that supports GPU acceleration, and realizes real-time storage and retrieval optimization of hundreds of millions of book data; The RAG generation engine module is used to deploy a localized large model, generate metadata-enhanced contextual prompts in query requests by calling high-precision retrieval results in the hybrid retrieval module, and output verifiable knowledge response content.

2. The artificial intelligence-based book data processing and intelligent service system according to claim 1, characterized in that: The multi-format document processing module includes a priority configuration unit and a regular cleaning unit; The priority configuration unit is used to assign priority extraction weights to the structured data of the book document according to preset rules; The regular expression cleaning unit is used to remove redundant characters and unstructured noise in the book document through regular expression matching.

3. The artificial intelligence-based book data processing and intelligent service system according to claim 1, characterized in that: The hybrid retrieval module includes a vector semantic retrieval unit, a keyword retrieval unit and a cross encoder reordering unit; The vector semantic retrieval unit is used to dynamically adjust the weight ratio of the vector retrieval according to the semantic complexity of the user query; The keyword search unit is used to dynamically adjust the weight ratio of keyword search according to the semantic complexity of the user query; The cross-encoder re-ranking unit is used to re-rank the retrieval results by relevance scores through a cross-encoder to screen the Top-K candidate sets.

4. The artificial intelligence-based book data processing and intelligent service system according to claim 1, characterized in that: The vector database module includes a fast deduplication unit and a metadata indexing unit; The fast deduplication unit is based on a local sensitive hash deduplication mechanism to eliminate duplicate block data; The metadata indexing unit is used to establish a multi-level index through fields in the structured data to speed up retrieval.

5. The artificial intelligence-based book data processing and intelligent service system according to claim 1, characterized in that: The RAG generation engine module includes a context enhancement unit and a verification interface unit; The context enhancement unit is used to inject metadata in the search results into the prompt template; The verification interface unit is used to call a preset rule library to verify the logical consistency and factual accuracy of the generated content.

6. A method for processing book data based on artificial intelligence, applied to the book data processing and intelligent service system based on artificial intelligence according to claim 1, characterized in that: The following steps are involved: S1. Receive heterogeneous book document input, perform format conversion and metadata extraction in the book documents, and generate a structured knowledge base; S2, triggering hybrid search based on user queries, generating candidate sets by combining semantic and keyword results; S3, call the vector database to retrieve relevant data blocks and build context-enhanced prompts; S4. Generate response content through the local large model and return the verified normalized output.

7. The method for processing book data based on artificial intelligence according to claim 1, characterized in that: The format conversion process in step S1 includes the following steps: S11, receiving uploaded book documents and storing them in a designated directory; S12, calling a document conversion tool to batch convert non-Markdown format files in the book document into unified Markdown format files; S13. Convert the Markdown format file into a structured HTML format file, and then use regular expressions to clear comments in the HTML format file.

8. The method for processing book data based on artificial intelligence according to claim 1, characterized in that: The metadata in the book document in step S1 includes the title, author, publisher, ISBN and publication date.

9. The method for processing book data based on artificial intelligence according to claim 1, characterized in that: The weight distribution formula for the hybrid search in step S2 is: Score final =α·sim vec +(1-α)·BM25(α∈[0.5,0.8]) Among them, Score final Indicates the final score of hybrid retrieval, sim vec represents vector retrieval similarity, BM25 represents keyword retrieval score, and α represents the dynamic weight coefficient.

10. The method for processing book data based on artificial intelligence according to claim 1, characterized in that: The construction of the vector database in step S3 includes the following steps: S31. Generate text vectors using pre-trained semantic encoding models; S32. Build a vector database based on Chroma and define a similarity threshold; S33. Filter low-correlation data through similarity threshold and use local sensitive hashing algorithm to achieve document deduplication.

11. A book data processing device based on artificial intelligence, characterized in that: It includes at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement an artificial intelligence-based book data processing method according to any one of claims 6 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement an artificial intelligence-based book data processing method according to any one of claims 6 to 10.

13. A computer program product, characterized in that: The invention comprises a computer program, which, when executed by a processor, implements a book data processing method based on artificial intelligence according to any one of claims 6 to 10.

Images