Book content retrieval method, device and computer equipment based on intelligent word segmentation

Abstract

The present application relates to a book content retrieval method, device and computer equipment based on intelligent word segmentation. The method includes: obtaining the word segmentation results of newly added books, dividing the word segmentation results into blocks to obtain multiple blocks; obtaining the extractive summary of each block, adjusting the original prompt template according to the extractive summary to obtain the first semantic extraction prompt template, using the fine-tuned local semantic extraction large model and the first semantic extraction prompt template to perform semantic extraction on each block to obtain the output text of each block; obtaining the generative summary of the newly added books, constructing the second semantic extraction prompt template according to the generative summary and the original prompt template, inputting the output text of each block and the second semantic extraction prompt template into the global semantic extraction large model, outputting the semantic introduction of the newly added books and storing it in the semantic introduction library; searching in the semantic introduction library according to the user's search statement. The use of this method can improve the efficiency and accuracy of intelligent retrieval.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a book content retrieval method, device and computer equipment based on intelligent word segmentation. Background Art

[0002] With the advancement of digitalization, the management and retrieval of books and document resources has become one of the core issues of enterprise knowledge management. Enterprises usually need to quickly locate business-related information in large-scale document libraries to support decision-making, R&D and operations. However, traditional technologies in the field of information retrieval currently face a common problem in practical applications: how to strike a balance between information coverage and semantic consistency.

[0003] Most traditional methods are based on keyword matching or statistical model technology, such as TF-IDF model or simple word vector method. These methods can cover more potential relevant information, but they are limited in deep semantic understanding, which often leads to the following problems: First, the information coverage is too high, resulting in insufficient relevance. When improving the search scope, traditional methods are prone to introduce a lot of noise information. Since they only rely on keywords or shallow semantics, the search results may contain a large number of irrelevant documents or fragments, and users need to spend extra time to screen truly relevant information. Second, semantic consistency is difficult to guarantee. The content in the search results usually lacks contextual consistency, and there may be logical disconnection or semantic conflicts between fragments, making it difficult to form a clear knowledge chain. This problem is particularly obvious when it is necessary to integrate multi-level information, such as comprehensive analysis from phrases to sentences to paragraphs. Third, the ability to process long documents is insufficient. Faced with complex long documents, traditional methods cannot effectively block and extract multi-level semantic information, and can only provide scattered document fragments, which cannot meet users' requirements for semantic integrity and logical coherence.

[0004] The above problems highlight the contradiction between information coverage and semantic consistency: on the one hand, improving information coverage may lead to a decrease in semantic consistency; on the other hand, excessive pursuit of semantic consistency may lead to insufficient information coverage and omission of potential relevant content. Traditional information retrieval methods are difficult to find an effective balance between the two and cannot fully meet the needs of enterprises for efficient and accurate information retrieval. Summary of the invention

[0005] Based on this, it is necessary to provide a book content retrieval method, device and computer equipment based on intelligent word segmentation to address the above technical problems.

[0006] A book content retrieval method based on intelligent word segmentation, the method comprising:

[0007] Performing intelligent word segmentation processing on the newly added books to obtain corresponding word segmentation results, and performing block processing on the word segmentation results based on semantic association rules to obtain multiple blocks;

[0008] Obtain an extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template, perform semantic extraction on each block using the fine-tuned local semantic extraction large model and the first semantic extraction prompt template, and obtain output text for each block;

[0009] Obtain a generative summary of the newly added book, construct a second semantic extraction prompt template based on the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic introduction of the newly added book and store it vectorized into the semantic introduction library;

[0010] The user search sentence is obtained, similarity matching is performed in the semantic introduction library according to the vector representation of the user search sentence, the matching result is output, and the top-ranked matching result and the corresponding semantic introduction are fed back to the user interface.

[0011] A book content retrieval device based on intelligent word segmentation, the device comprising:

[0012] A block processing module is used to perform intelligent word segmentation processing on the newly added books to obtain corresponding word segmentation results, and to perform block processing on the word segmentation results based on semantic association rules to obtain multiple blocks;

[0013] A block semantic extraction module is used to obtain an extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template, and use the fine-tuned local semantic extraction large model and the first semantic extraction prompt template to perform semantic extraction on each block to obtain an output text of each block;

[0014] A semantic summary extraction module is used to obtain a generative summary of the newly added book, construct a second semantic extraction prompt template according to the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic summary of the newly added book and store it vectorized in the semantic summary library;

[0015] The retrieval output module is used to obtain the user's search sentence, perform similarity matching in the semantic introduction library according to the vector representation of the user's search sentence, output the matching result, and feed back the top-ranked matching result and the corresponding semantic introduction to the user interface.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Performing intelligent word segmentation processing on the newly added books to obtain corresponding word segmentation results, and performing block processing on the word segmentation results based on semantic association rules to obtain multiple blocks;

[0018] Obtain an extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template, perform semantic extraction on each block using the fine-tuned local semantic extraction large model and the first semantic extraction prompt template, and obtain output text for each block;

[0019] Obtain a generative summary of the newly added book, construct a second semantic extraction prompt template based on the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic introduction of the newly added book and store it vectorized into the semantic introduction library;

[0020] The user search sentence is obtained, similarity matching is performed in the semantic introduction library according to the vector representation of the user search sentence, the matching result is output, and the top-ranked matching result and the corresponding semantic introduction are fed back to the user interface.

[0021] The above-mentioned book content retrieval method, device and computer equipment based on intelligent word segmentation refine the newly added book text into semantic units through intelligent word segmentation, achieve extensive information coverage, and divide the blocks according to the semantic association rules to ensure that the semantics of each block are relatively unified, preliminarily balance the information coverage and semantic consistency, and lay the foundation for long document processing. Through the extractive summary, the key information of the block is accurately refined, and the information is compressed while maintaining the information importance coverage. The semantic extraction template is adjusted accordingly, so that the subsequent extraction revolves around the core semantics and strengthens the semantic consistency, which is conducive to the balanced advancement of the two when assisting multi-level semantic extraction. Through the generative summary, the core of the book is grasped from the whole, and the semantic introduction is constructed in combination with the block output, the overall semantic direction can be controlled, the information coverage can be improved, and the semantic introduction is generated according to the template constructed by the model, which is conducive to ensuring the coordination with the semantics of each level. Thereby, the semantic matching of the user during the retrieval can go deep into the semantic level, and a comprehensive search and precise positioning can be achieved. The embodiment of the present invention can improve the efficiency and accuracy of the intelligent retrieval service. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a flow chart of a book content retrieval method based on intelligent word segmentation in one embodiment;

[0023] Figure 2 A schematic diagram of a process of constructing a first semantic extraction prompt template in an embodiment;

[0024] Figure 3 A schematic diagram of a process for obtaining output text in blocks in one embodiment;

[0025] Figure 4 A schematic diagram of a process for obtaining a semantic introduction of a book in one embodiment;

[0026] Figure 5 is a structural block diagram of a book content retrieval device based on intelligent word segmentation in one embodiment;

[0027] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] In one embodiment, Figure 1 As shown, a book content retrieval method based on intelligent word segmentation is provided, comprising the following steps:

[0030] Step 102, intelligent word segmentation processing is performed on the newly added books to obtain corresponding word segmentation results, and the word segmentation results are segmented based on semantic association rules to obtain multiple blocks.

[0031] Newly added books refer to various types of book resources added to the platform. The scope they cover is related to the various business models involved in the platform. Taking the proposal management business as an example, newly added books may include guidance books on how to write high-quality proposals, research works that conduct in-depth analysis of various policies to provide reference for proposals, or case collections that introduce past excellent proposal cases, etc. Newly added books are added to the platform in the form of electronic documents.

[0032] It can be understood that intelligent word segmentation processing can decompose text content into basic semantic units, ensuring that each word in the text can be processed separately. The word segmentation results are processed in blocks, so that the book content can be divided into different parts according to a certain logic, thereby ensuring the coverage of the book content as a whole and avoiding the omission of important information. The block operation itself is based on certain semantic associations. Specifically, semantically consistent content is divided into the same block, so that the semantics within each block is relatively consistent, providing a relatively semantically concentrated unit for subsequent processing.

[0033] Step 104, obtain the extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain the first semantic extraction prompt template, use the fine-tuned local semantic extraction large model and the first semantic extraction prompt template to perform semantic extraction on each block, and obtain the output text of each block.

[0034] The extractive summary contains the logical relationship between the main content and key information in the document block. By guiding the model to focus on key information through the extractive summary, the model can understand the key information in a relatively complete semantic environment. Obtaining the extractive summary of the block can extract key information from each block. Without losing the core content of the block, the information is compressed to a certain extent, so that subsequent processing can focus on important information. At the same time, it also ensures that the main information of the block can be extracted and the information coverage is maintained.

[0035] The original prompt template is adjusted according to the extractive summary to obtain the first semantic extraction prompt template, so that the semantic extraction template is more in line with the actual content of the book block. During the semantic extraction process, this template can guide the model to extract according to the core semantic direction of the block, ensuring that the output text is consistent with the original semantics of the block, and maintaining the semantic coherence and consistency.

[0036] Step 106, obtain the generative summary of the newly added book, construct a second semantic extraction prompt template based on the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic introduction of the newly added book and store it vectorized in the semantic introduction library.

[0037] The generative summary can summarize the main content and core ideas of the document, provide a macro-semantic guide for the generation of prompt templates, and obtain the generative summary of the newly added book. It can grasp the core content of the book as a whole, and build a semantic introduction based on the output text of each block. The semantic introduction not only contains the macro-core content of the book, but also integrates the detailed information of each block, comprehensively covering the important information of the book.

[0038] The second semantic extraction prompt template is constructed based on the generative summary and the original prompt template, so that the template can extract semantics closely around the characteristics of the book itself. It ensures that when generating the semantic introduction, the model can generate around the theme and core semantics of the book, ensuring the semantic consistency between the semantic introduction and the overall content of the book and the semantic consistency of each block content.

[0039] In step 104, the local semantic extraction big model is used to carry out semantic mining locally (i.e., at the block level), while the global semantic extraction big model is to build a complete, coherent, and global semantic framework from the perspective of the entire book, that is, the generated semantic introduction can cover the main content structure and logical relationships of the book. Both the local semantic extraction big model and the global semantic extraction big model can adopt the LLM big model.

[0040] Step 108, obtain the user's search sentence, perform similarity matching in the semantic profile library according to the vector representation of the user's search sentence, output the matching result, and feed back the top-ranked matching result and the corresponding semantic profile to the user interface.

[0041] In the above-mentioned book content retrieval method based on intelligent word segmentation, the newly added book text is refined into semantic units through intelligent word segmentation to achieve extensive information coverage, and the blocks are divided according to the semantic association rules to ensure that the semantics of each block are relatively unified, and the information coverage and semantic consistency are initially balanced, laying the foundation for long document processing. The extractive summary accurately extracts the key information of the block, compresses the information while maintaining the information importance coverage, and adjusts the semantic extraction template accordingly, so that the subsequent extraction revolves around the core semantics, strengthens the semantic consistency, and is conducive to facilitating the balanced advancement of the two during multi-level semantic extraction. Through the generative summary, the core of the book is grasped from the whole, and the semantic introduction is constructed in combination with the block output, which can control the overall semantic direction and improve information coverage. The template constructed by the model guides the generation of the semantic introduction, which is conducive to ensuring coordination with the semantics of each level. As a result, the semantic matching of the user during retrieval can go deep into the semantic level, realize comprehensive search and precise positioning. The embodiment of the present invention can improve the efficiency and accuracy of intelligent retrieval services.

[0042] In one embodiment, the newly added book content is intelligently segmented to obtain the corresponding segmentation results, including: obtaining the document text corresponding to the newly added book content, and using the intelligent segmenter pre-trained based on the domain word library to perform intelligent segmentation on the document text to obtain the corresponding segmentation results. In this embodiment, after obtaining the document text, it is input into the intelligent segmenter pre-trained based on the domain word library. The intelligent segmenter is built based on a neural network architecture, and its core neural network model can use a recurrent neural network (RNN) or a convolutional neural network (CNN) and its variants, such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU). This neural network structure can effectively learn the semantic and grammatical information in the text sequence, so as to perform the segmentation operation more accurately. The domain word library includes relevant content under the application field. For example, in business models such as proposal management, the domain word library includes business-related vocabulary, legal and regulatory vocabulary, core terms, and emerging technology vocabulary.

[0043] In one embodiment, the word segmentation results are segmented based on semantic association rules to obtain multiple blocks, including: generating multiple initial blocks according to paragraph symbols in the word segmentation results, traversing each initial block, calculating the similarity between the current initial block and the next initial block, and if the similarity is greater than a preset threshold, merging the next initial block into the current initial block to obtain a new block, until all initial blocks are traversed, outputting the current segmentation result, and obtaining multiple blocks.

[0044] In this embodiment, first, the word segmentation results of the book content after intelligent word segmentation processing are obtained, and then, according to the common paragraph symbols (such as line breaks, etc.) in the natural language text, preliminary division is performed to divide the text into relatively independent parts from the perspective of vision and structure, and each part is an initial block. Each initial block is stored as an element in the list, and each element contains all the word segmentation results in the block and the corresponding position information (such as the starting and ending positions in the original book text, etc.), which is convenient for subsequent accurate calculation of the similarity between the blocks and merging operations. All words in the block are converted into corresponding word vectors, and then the cosine similarity between the word vector sets of the two blocks is calculated to measure their semantic similarity. Each initial block is taken out in turn from the data structure storing the initial blocks, and then the next initial block is obtained, and the similarity score between them is calculated. When it is determined that the similarity of the two initial blocks is greater than the preset threshold and needs to be merged, all the word segmentation results and related position information in the next initial block are merged into the current initial block. After the above traversal and merging operations, the blocks that finally remain in the data structure are the block results after processing based on semantic association rules. These blocks are relatively independent and coherent in semantics. Each block contains a part of the relevant patterns such as proposal management, providing a suitable semantic unit basis for subsequent operations such as semantic extraction.

[0045] In one embodiment, Figure 2 As shown, a schematic diagram of a process of constructing a first semantic extraction prompt template is provided, and adjusting the original prompt template according to the extractive summary to obtain the first semantic extraction prompt template includes:

[0046] Step 202, obtaining a keyword set of the extractive summary and an importance score of each keyword;

[0047] Step 204, converting the basic information guidance in the original prompt template into a vector representation, calculating the correlation score between each keyword and each dimension of the information guidance vector, adjusting the basic information guidance according to each keyword and the corresponding correlation score, and obtaining an adjusted information guidance vector;

[0048] Step 206 , adjusting the original prompt template according to the natural language description corresponding to the adjusted information guidance vector, and filling the key information and detail information of the extractive summary into the original prompt template to obtain a first semantic extraction prompt template.

[0049] Specifically, in step 202, the importance score of the word is calculated by the TF-IDF method (term frequency-inverse document frequency), and the keyword set is determined according to the importance score. The importance score is calculated by the following formula:

[0050] ;

[0051] in, , , For words The number of occurrences in the extractive summary, is the total number of words in the extractive summary, is the total number of documents in the document library, To contain words Through the above calculation process, the words are sorted according to their TF-IDF values, and several words with higher TF-IDF values ​​are selected to form a keyword set. By extracting the keyword set of the extractive summary and calculating its importance score in step 202, the core words in the extractive summary and their relative importance can be accurately grasped, providing a key basis for subsequent adjustment of information guidance.

[0052] In step 204, the correlation score between each keyword and each dimension of the information guidance vector is calculated as:

[0053] ;

[0054] in, is the basic information guidance, and the adjusted information guidance vector is:

[0055] ;

[0056] in, is the first The original information guide vector dimension, The basic information guidance in the original prompt template is converted into a vector representation and the correlation score with the keyword is calculated. This operation realizes the deep integration of extractive summary and original information guidance.

[0057] In step 206, the original prompt template includes the original information guidance, and the adjusted information guidance vector is converted into a natural language description Replace the "information guidance" position of the original prompt template. Fill the selected key phrase pairs and their logical relationships into the corresponding positions, and at the same time determine and fill the content of "specific context or theme" according to the theme of the extractive summary to obtain the first semantic extraction prompt template.

[0058] In this embodiment, the specific context or topic is firstly clarified, so that the big model has more background support when analyzing key phrases and their logical relationships, and will not view the phrase relationships in isolation from the overall context. When designing the first semantic extraction prompt template, it is considered that in [document topic or specific context], the core phrases and their logical associations related to [key information 1], [key information 2], etc. are first locked, and their core meanings in the overall context are understood; then, all elements in the phrase text are comprehensively sorted out, including modifiers, qualifiers, supplementary explanatory phrases and other detailed information, and how these details enrich the semantic connotation around the core information is analyzed, and special attention is paid to those details that supplement, refine, correct or expand the core information; finally, the core semantics of the key information and the detailed information are integrated to present a complete and meticulous semantic analysis result, focusing on how the detailed information affects the expression and understanding of the key information in a specific context, so as to guide the big model to pay attention to both the key information and the detailed information when processing the text, thereby improving the completeness and accuracy of semantic extraction in book content retrieval.

[0059] For example, first, extract key information from the extractive summary. For the "proposal for balanced allocation of educational resources", the key information is "educational resources" and "balanced allocation". Then extract key phrases based on syntactic analysis. Key phrases such as "urban-rural educational resource gap" and "balanced allocation of teaching staff" will be extracted. The model locks these key phrases according to the prompt template and analyzes their logical associations. For example, there is a causal relationship between "urban-rural educational resource gap" and "balanced allocation of teaching staff". Because there is a gap in urban and rural educational resources, balanced allocation of teaching staff is needed to promote balanced allocation of educational resources, so as to understand its core meaning in the context of the entire historical proposal. Then, the detailed information in the extractive summary, such as "distance teaching technology" in "narrowing the gap between urban and rural educational resources through distance teaching technology", "increasing investment in education funds in rural areas to help balanced allocation of teaching staff" and other detailed phrases will also be comprehensively sorted out. The model will analyze how these details enrich the semantic connotation of the core information. For example, the detail "distance teaching technology" describes a specific means to narrow the gap in urban and rural educational resources, while "rural areas" and "education funding investment" clarify the key areas and implementation methods for the balanced allocation of teaching staff. These detailed information further elaborates on the specific content and implementation path of the proposal around the core information. The model integrates the core semantics of the key information with these detailed information according to the prompt template, thereby presenting the information about the historical proposal in a complete and detailed manner, allowing users to have an in-depth understanding of the overall picture of the historical proposal, grasping the key points and knowing the specific details.

[0060] In one embodiment, semantic extraction is performed on each block using a fine-tuned local semantic extraction large model and a first semantic extraction prompt template, and the output text of each block is obtained, including: obtaining a multi-level semantic text of each block; the multi-level semantic text includes phrase text, sentence text and paragraph text; using a fine-tuned large language model to interact with the first semantic extraction prompt template, gradually extracting semantic information at all levels in the block, and obtaining the output text of each block; the output text includes phrase semantic information, sentence semantic information and paragraph semantic information. In this embodiment, for each block, a preliminary text level division is first performed based on punctuation marks. Common punctuation marks such as commas, semicolons, periods, question marks, exclamation marks, etc. are used as separation marks. For phrase text, commas, semicolons and some conjunctions are further used as the basis for extraction, and the entire block itself is used as the paragraph text, and sentences and phrases in each paragraph are extracted in the above manner. After the preliminary division, the extracted semantic texts at all levels are checked for semantic integrity and coherence. Adjustments are made for some situations where the division may be inaccurate due to irregular use of punctuation or complex semantics.

[0061] In one embodiment, Figure 3 As shown, a flowchart of obtaining block output text is provided, using the fine-tuned large language model to interact with the first semantic extraction prompt template to gradually extract semantic information at all levels in the block, and the output text of each block includes:

[0062] Step 302, input the phrase text of the current block and the first semantic extraction prompt template into the large language model, and output the phrase semantic information of the current block;

[0063] Step 304, constructing a corresponding sentence semantic extraction prompt template according to the phrase semantic information of the current block and the original prompt template, inputting the sentence text and sentence semantic extraction of the current block into the large language model, and outputting the sentence semantic information of the current block;

[0064] Step 306, constructing a paragraph semantic extraction prompt template according to the sentence semantic information of the current block and the original prompt template, inputting the paragraph text of the current block and the paragraph semantic extraction prompt template into the large language model, and outputting the paragraph semantic information of the current block;

[0065] Step 308, obtaining the output text of the current block according to the phrase semantic information, sentence semantic information and paragraph semantic information of the current block;

[0066] Step 310, traverse all blocks to obtain the output text of each block.

[0067] Specifically, in step 302, for each block of phrase text, it is organized into a format suitable for input into the large language model, and the corresponding first semantic extraction prompt template is obtained to ensure that the key elements in the template (such as key phrase placeholders, logical relationship types, etc.) are clear and can be used to guide semantic analysis, and the organized phrase text and the first semantic extraction prompt template are input into the fine-tuned large language model together. Based on its pre-trained language knowledge and adaptability to specific tasks after fine-tuning, the large language model performs semantic analysis on each phrase in the phrase text according to the guidance of the first semantic extraction prompt template, and outputs the phrase semantic information of each block according to the analysis results, which includes the specific meaning of each key phrase (combined with the context), the logical relationship between key phrases, and other modifying or supplementary semantic content around these phrases.

[0068] In step 304, the phrase semantic information of each block is deeply analyzed to extract the core key phrases and the types of logical relationships between them, to clarify the theme and main semantic focus of the phrase semantics of the block, and to integrate the key phrases, logical relationships, themes and semantic focus extracted from the phrase semantic information into the original prompt template.

[0069] In step 306, the logical structure between sentences is sorted out, including cause-effect relationship, progressive relationship, and parallel relationship, the theme and core idea of ​​the entire paragraph are extracted, and the logical structure between sentences, paragraph theme and core idea are integrated into the original prompt template.

[0070] In this embodiment, the extracted phrase semantic information, sentence semantic information and paragraph semantic information are integrated to form the output text of each block. This output text comprehensively reflects the semantic content of the block in the platform business context, including multi-level information from micro phrase semantics to macro paragraph semantics, providing rich and accurate materials for subsequent semantic integration and the generation of the overall semantic introduction of the book, which helps to better utilize the semantic knowledge resources in the book in various business application scenarios of the platform, such as content retrieval, knowledge graph construction, etc.

[0071] In one embodiment, constructing a second semantic extraction prompt template based on a generative summary and an original prompt template includes: obtaining a set of key subject words of the generative summary and a set of logical relationships between the key subject words, filling the key subject words and the corresponding logical relationships into the original prompt template, assigning an initial weight to each keyword according to the importance of each key subject word, and obtaining a second semantic extraction prompt template.

[0072] In this embodiment, the proposal management business model is taken as an example, for example, "the proposal management system plays a key role in proposal collection, review, classification and subsequent follow-up processing, and achieves its goals by establishing online submission channels, setting review standard processes, using intelligent classification algorithms and building tracking feedback mechanisms". First, a keyword extraction algorithm (such as TF-IDF algorithm) in natural language processing technology is used to extract a set of key subject words, including "proposal management system", "proposal collection", "review", "classification", "follow-up processing", "online submission channels", "review standard processes", "intelligent classification algorithms" and "tracking feedback mechanisms". Then, the logical relationship set between key subject words is determined by semantic analysis tools, such as "proposal collection" and "online submission channels" have a method and purpose relationship, because online submission channels are an important way to realize proposal collection; "review" and "review standard processes" have a basis relationship, and the review is carried out according to a specific standard process; "classification" and "intelligent classification algorithm" have a technical support relationship, and the intelligent classification algorithm helps proposal classification, etc. These key subject words and corresponding logical relationships are filled into the original prompt template. For example, the original prompt template is "Under [topic], focus on analyzing the relationship between [key element 1] and [key element 2] and the impact on [goal]", and after filling it becomes "Under the topic of proposal management system, focus on analyzing the relationship between proposal collection and online submission channels and the impact on improving the efficiency of proposal acquisition", etc. Then, an initial weight is assigned to each keyword based on its importance in the proposal management business. For example, if "audit" is regarded as the core business focus in the overall proposal management business, then a relatively high initial weight, such as 0.8, is assigned to "audit" and its related keywords, while other relatively auxiliary keywords such as "intelligent classification algorithm" can be assigned an initial weight of 0.6. The determination of weights can be combined with multiple factors such as the actual process focus, business goals, and past case analysis of the proposal management business, so as to obtain the second semantic extraction prompt template. When constructing the second semantic extraction prompt template, a set of key subject terms and a set of logical relationships are extracted from the generative abstracts of books related to proposal management, filled into the original prompt template, and the keywords are assigned initial weights according to their importance. This lays the foundation for subsequent semantic processing and preliminarily determines the focus and weight distribution of semantic analysis, so that semantic extraction can be carried out around core business elements.

[0073] In one embodiment, Figure 4 As shown, a flowchart of obtaining a semantic introduction to a book is provided. The output text of each block and the second semantic extraction prompt template are input into the global semantic extraction model. The output semantic introduction to the newly added book includes:

[0074] Step 402, traverse each block, obtain the corresponding semantic information set according to the output text of the current block, initialize the semantic window, input the semantic information set of the current block and the second semantic extraction prompt template into the large language model, obtain the semantic introduction and store it in the semantic window;

[0075] Step 404, the semantic information set of the next block is merged with the semantic brief in the semantic window. If a subject word related to the existing semantic information in the semantic window appears in the new block, the similarity between the related subject word and the key subject word in the generative summary is calculated, and the keyword weight in the second semantic extraction prompt template is updated according to the similarity score to obtain an updated second semantic extraction prompt template;

[0076] Step 406, based on the updated second semantic extraction prompt template, the semantic profile in the current semantic window is updated, and the above steps are iterated until all the blocks are processed, then the traversal is terminated and the semantic profile in the current semantic window is output.

[0077] Specifically, in step 402, for each block of the proposal management book content after block processing, the corresponding semantic information set is first obtained based on the output text of the current block (including phrase semantic information, sentence semantic information and paragraph semantic information). The semantic window is initialized, and the semantic information set of the current block and the second semantic extraction prompt template are input into the global semantic extraction large language model. The model generates a semantic introduction based on the template guidance and block semantic information, such as "This block explains the specific layout and functions of the online submission interface design of the proposal management system when collecting proposals. This interface design is an important link to facilitate proposers to submit proposals and increase the number of proposals collected", and stores it in the semantic window.

[0078] In step 404, when processing the next block, its semantic information set is merged with the semantic introduction in the semantic window. For example, the new block is about "Information security measures of the proposal management system in the process of proposal collection", where "proposal collection" is a keyword related to the existing semantic information in the semantic window. Calculate the similarity between "proposal collection" and the key keyword "proposal collection" in the generative summary, and update the similarity score according to the pre-set similarity score update rule. If the similarity is higher than 0.8, increase the keyword weight, and increase the weight of "proposal collection" in the second semantic extraction prompt template from 0.6 to 0.7 to obtain the updated second semantic extraction prompt template. Dynamic weight update improves the accuracy of the semantic introduction. By comparing the similarity with the key keywords of the generative summary and adjusting the weight, the model can extract semantic information more accurately based on the overall business core and the current text focus during the block processing process, avoid interference from irrelevant or secondary information, and make the semantic introduction more in line with the actual semantic connotation of the proposal management business.

[0079] In step 406, the semantic introduction in the current semantic window is updated based on the updated second semantic extraction prompt template. For example, the updated template and the new block semantic information are used to re-enter the large language model to generate a new semantic introduction "This block further elaborates on the information security measures based on the original proposal collection, and works together with the online submission interface design to improve the proposal collection system, and provide strong guarantees for the safe and efficient operation of the proposal management system", replacing the original semantic introduction in the semantic window. Then continue to iterate the processing of the next block, repeating the above fusion, template update and window update steps until all blocks are processed. Finally, the semantic introduction in the current semantic window is output. The semantic introduction comprehensively covers the overall semantics of the content of the proposal management book, and can be used for subsequent application scenarios such as knowledge retrieval and business process optimization in the proposal management business.

[0080] In this embodiment, an incremental learning method based on a dynamic semantic window is used to improve efficiency. The window stores the semantic information of previous blocks. When processing new blocks, the existing information can be used for fusion and optimization to reduce repeated calculations and analysis. As the block processing progresses, the model is continuously optimized and adjusted to gradually improve the understanding and summarization capabilities of the overall book semantics. The final output semantic profile can comprehensively and efficiently reflect the overall semantics of the proposal management book, provide high-quality semantic support for subsequent application scenarios such as knowledge retrieval and business process optimization, and effectively improve the efficiency and accuracy of text semantic information related to proposal management business processing.

[0081] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0082] In one embodiment, Figure 5 As shown, a book content retrieval device based on intelligent word segmentation is provided, comprising:

[0083] The block processing module 502 is used to perform intelligent word segmentation processing on the newly added books to obtain corresponding word segmentation results, and to perform block processing on the word segmentation results based on semantic association rules to obtain multiple blocks;

[0084] The block semantic extraction module 504 is used to obtain an extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template, and use the fine-tuned local semantic extraction large model and the first semantic extraction prompt template to perform semantic extraction on each block to obtain an output text of each block;

[0085] The semantic summary extraction module 506 is used to obtain the generative summary of the newly added book, construct a second semantic extraction prompt template according to the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic summary of the newly added book and store it vectorized in the semantic summary library;

[0086] The search output module 508 is used to obtain the user's search sentence, perform similarity matching in the semantic profile library according to the vector representation of the user's search sentence, output the matching result, and feed back the top-ranked matching result and the corresponding semantic profile to the user interface.

[0087] For the specific definition of the book content retrieval device based on intelligent word segmentation, please refer to the definition of the book content retrieval method based on intelligent word segmentation in the above text, which will not be repeated here. Each module in the above-mentioned book content retrieval device based on intelligent word segmentation can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0088] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a book content retrieval method based on intelligent word segmentation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0089] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0091] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A book content retrieval method based on intelligent word segmentation, characterized in that: The method comprises: Performing intelligent word segmentation processing on the newly added books to obtain corresponding word segmentation results, and performing block processing on the word segmentation results based on semantic association rules to obtain multiple blocks; Obtain an extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template, perform semantic extraction on each block using the fine-tuned local semantic extraction large model and the first semantic extraction prompt template, and obtain output text for each block; Obtain a generative summary of the newly added book, construct a second semantic extraction prompt template based on the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic introduction of the newly added book and store it vectorized into the semantic introduction library; Obtaining a user search sentence, performing similarity matching in the semantic profile library according to a vector representation of the user search sentence, outputting a matching result, and feeding back a top-ranked matching result and a corresponding semantic profile to a user interface; Adjusting the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template includes: Obtaining a keyword set of the extractive summary and an importance score of each keyword; The basic information guidance in the original prompt template is converted into a vector representation to obtain an information guidance vector, the correlation score between each keyword and each dimension of the information guidance vector is calculated, and the basic information guidance is adjusted according to each keyword and the corresponding correlation score to obtain an adjusted information guidance vector; The original prompt template is adjusted according to the natural language description corresponding to the adjusted information guidance vector, and the key information and detail information of the extractive summary are filled into the original prompt template to obtain a first semantic extraction prompt template.

2. The method according to claim 1, characterized in that Intelligent word segmentation is performed on the newly added book content, and the corresponding word segmentation results include: The document text corresponding to the newly added book content is obtained, and the document text is intelligently segmented using an intelligent word segmenter pre-trained based on a domain vocabulary library to obtain the corresponding word segmentation results.

3. The method according to claim 1, characterized in that The word segmentation result is processed into blocks based on semantic association rules to obtain multiple blocks including: Generate multiple initial blocks according to the paragraph symbols in the word segmentation results, traverse each initial block, calculate the similarity between the current initial block and the next initial block, and if the similarity is greater than the preset threshold, merge the next initial block into the current initial block to obtain a new block, until all initial blocks are traversed, output the current block result, and obtain multiple blocks.

4. The method according to claim 1, characterized in that: The fine-tuned local semantic extraction model and the first semantic extraction prompt template are used to perform semantic extraction on each block, and the output text of each block includes: Acquire a multi-level semantic text of each block; the multi-level semantic text includes a phrase text, a sentence text and a paragraph text; The fine-tuned large language model is used to interact with the first semantic extraction prompt template to gradually extract semantic information at all levels in the block to obtain the output text of each block; the output text includes phrase semantic information, sentence semantic information and paragraph semantic information.

5. The method according to claim 4, characterized in that The fine-tuned large language model is used to interact with the first semantic extraction prompt template to gradually extract semantic information at all levels in the block, and the output text of each block includes: Input the phrase text of the current block and the first semantic extraction prompt template into the large language model, and output the phrase semantic information of the current block; Constructing a corresponding sentence semantic extraction prompt template according to the phrase semantic information of the current block and the original prompt template, inputting the sentence text of the current block and the sentence semantic extraction into the large language model, and outputting the sentence semantic information of the current block; Constructing a paragraph semantic extraction prompt template according to the sentence semantic information of the current block and the original prompt template, inputting the paragraph text of the current block and the paragraph semantic extraction prompt template into the large language model, and outputting the paragraph semantic information of the current block; Obtain the output text of the current block according to the phrase semantic information, sentence semantic information and paragraph semantic information of the current block; Traverse all blocks and get the output text of each block.

6. The method according to claim 1, characterized in that Constructing a second semantic extraction prompt template according to the generative summary and the original prompt template comprises: Obtain a set of key words and a set of logical relationships between the key words of the generative summary, fill the key words and the corresponding logical relationships into the original prompt template, assign an initial weight to each keyword according to the importance of each key word, and obtain a second semantic extraction prompt template.

7. The method according to claim 6, characterized in that The output text of each block and the second semantic extraction prompt template are input into the global semantic extraction model, and the semantic introduction of the newly added book is output, including: Traverse each block, obtain the corresponding semantic information set according to the output text of the current block, initialize the semantic window, input the semantic information set of the current block and the second semantic extraction prompt template into the large language model, obtain the semantic introduction and store it in the semantic window; The semantic information set of the next block is merged with the semantic brief in the semantic window. If a subject word related to the existing semantic information in the semantic window appears in the new block, the similarity between the related subject word and the key subject word in the generative summary is calculated, and the keyword weight in the second semantic extraction prompt template is updated according to the similarity score to obtain the updated second semantic extraction prompt template; The semantic profile in the current semantic window is updated based on the updated second semantic extraction prompt template, and the above steps are iterated until the blocks are processed, then the traversal is terminated and the semantic profile in the current semantic window is output.

8. A book content retrieval device based on intelligent word segmentation, characterized in that: The device comprises: A block processing module is used to perform intelligent word segmentation processing on the newly added books to obtain corresponding word segmentation results, and to perform block processing on the word segmentation results based on semantic association rules to obtain multiple blocks; A block semantic extraction module is used to obtain an extractive summary of each block, adjust the original prompt template according to the extractive summary to obtain a first semantic extraction prompt template, and use the fine-tuned local semantic extraction large model and the first semantic extraction prompt template to perform semantic extraction on each block to obtain an output text of each block; A semantic summary extraction module is used to obtain a generative summary of the newly added book, construct a second semantic extraction prompt template according to the generative summary and the original prompt template, input the output text of each block and the second semantic extraction prompt template into the global semantic extraction model, output the semantic summary of the newly added book and store it vectorized in the semantic summary library; A search output module is used to obtain a user search sentence, perform similarity matching in the semantic profile library according to the vector representation of the user search sentence, output the matching result, and feed back the top-ranked matching result and the corresponding semantic profile to the user interface; The block semantic extraction module is also used to obtain the keyword set of the extractive summary and the importance score of each keyword; convert the basic information guidance in the original prompt template into a vector representation to obtain an information guidance vector, calculate the correlation score between each keyword and each dimension of the information guidance vector, adjust the basic information guidance according to each keyword and the corresponding correlation score, and obtain an adjusted information guidance vector; adjust the original prompt template according to the natural language description corresponding to the adjusted information guidance vector, and fill the key information and detail information of the extractive summary into the original prompt template to obtain a first semantic extraction prompt template.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images