Data retrieval method, text database generation method and server

By using data retrieval methods in the knowledge question and answer system, complete and coherent target text content is generated based on text blocks and metadata of the text database, the problem of low accuracy in the knowledge question and answer system is solved, and the accuracy of the answer is significantly improved.

CN119025641APending Publication Date: 2024-11-26XFUSION DIGITAL TECH CO LTD

Patent Information

Application Number
CN202411044782.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The answers of the knowledge question and answer system are low in accuracy and cannot effectively solve this problem.

Method used

Through a data retrieval method, based on the similarity between the problem text and the text blocks in the text database, text blocks are extracted and metadata are combined, associated text blocks are determined, sorted and integrated, target text content is generated, and it is pushed to Large language model to improve the accuracy of answers.

Benefits of technology

Ensure the completeness and coherence of prompt content and improve the accuracy of the answers of the knowledge Q&A system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025641A_ABST
    Figure CN119025641A_ABST
Patent Text Reader

Abstract

The invention provides a data retrieval method, a text database generation method and a server, and the data retrieval method comprises the steps: extracting a first number of text blocks from a text database based on a first similarity between a problem text and the text blocks in the text database, based on a preset context range and the first metadata of each text block in the first number of text blocks, determining an associated text block of each text block, and based on the first metadata and the second metadata, determining an associated text block of each text block; and sorting and integrating each text block and the associated text block of each text block to obtain target text content corresponding to each text block, and determining a reply text corresponding to the question text based on the target text content corresponding to each text block. According to the data retrieval method provided by the embodiment of the invention, the accuracy of knowledge questions and answers is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data retrieval method, a text database generation method and a server. Background Art

[0002] Currently, the accuracy of answers given by the knowledge question-answering system is relatively low. Summary of the invention

[0003] The embodiments of the present application provide a data retrieval method, a text database generation method and a server, which are intended to solve the problem of answer accuracy in a knowledge question and answer system.

[0004] In a first aspect, an embodiment of the present application provides a data retrieval method, comprising:

[0005] Based on a first similarity between the question text and a text block in a text database, extracting a first number of text blocks from the text database, wherein the text database includes a plurality of text blocks and metadata corresponding to each of the text blocks, wherein the metadata includes a title path of the corresponding text block, a first sequence identifier of a text content where the text block is located in a first document, and a second sequence identifier of the text block in the text content, wherein the text content is a text content in the first document that belongs to any last-level title;

[0006] Determining an associated text block of each text block based on a preset context range and first metadata of each text block in the first number of text blocks, wherein the first metadata is metadata corresponding to each text block in the first number of text blocks;

[0007] Based on the first metadata and the second metadata, each text block and the associated text blocks of each text block are sorted and integrated to obtain the target text content corresponding to each text block, wherein the second metadata is the metadata of the associated text blocks of each text block;

[0008] Based on the target text content corresponding to each of the text blocks, the answer text corresponding to the question text is determined.

[0009] In a possible implementation, determining the answer text corresponding to the question text based on the target text content corresponding to each of the text blocks includes:

[0010] Based on the first similarity, extracting a second amount of the target text content from the target text content corresponding to each of the text blocks;

[0011] The second amount of target text content is pushed to the large language model as prompt text and question text to obtain a reply text for the question text.

[0012] The data retrieval method provided in the embodiment of the present application is a complete component of the target text content obtained based on the metadata in the document, which ensures the integrity and coherence of the prompt content, thereby improving the accuracy of the knowledge question and answer. In a possible implementation, the first metadata of each text block in the first number of text blocks and the preset context range are used to determine the associated text blocks of each text block, including:

[0013] Based on the preset context range, determining the maximum number of associated text blocks of each text block;

[0014] The maximum number is used as the length of the context window interval of each text block to determine the associated text block of each text block.

[0015] In a possible implementation, the data retrieval method further includes:

[0016] Splitting the text content of the first document belonging to the last-level title into at least one text block, wherein the first document includes a directory, and the directory includes at least one level of titles;

[0017] The text block index data is stored to generate the text database, wherein the text block index data includes: the text block and the metadata.

[0018] In a possible implementation, splitting the text content of the first document belonging to the last-level title into at least one text block includes:

[0019] Marking the text content to obtain text content with markers;

[0020] Based on a preset segmentation length and a preset text block segmentation judgment rule, the text content with the marker is segmented to obtain one or more text blocks.

[0021] In a possible implementation, segmenting the text content with the marker based on a preset segmentation length and a preset text block segmentation judgment rule includes:

[0022] The text content is segmented using the markers selected from the markers in a preset order, and it is determined whether the length of each text block obtained by each segmentation is not greater than a preset segmentation length, and whether the lengths of two adjacent text blocks are not less than the preset segmentation length.

[0023] In a possible implementation, segmenting the text content using the markers selected from the markers in a preset order includes:

[0024] Selecting the markers in order of priority from high to low, wherein the priority of the marker currently selected is lower than the priority of the marker previously selected;

[0025] The text content obtained from the previous segmentation is segmented based on the markers selected each time.

[0026] In a possible implementation, the marking process on the text content to obtain the text content with a marker includes:

[0027] Convert the text content into markdown format based on a preset conversion tool;

[0028] The text in the markdown format is input into a json converter to process the markup information to obtain the text content with the marker.

[0029] In a second aspect, an embodiment of the present application provides a text database generation method, which is applied to a first document, wherein the first document includes a directory, and the directory includes at least one level of titles, and the method includes:

[0030] Splitting the text content belonging to the last-level title in the first document into at least one text block;

[0031] Store text block index data, the text block index data includes: the text block and metadata, the metadata includes: the title path of the text block, the first sequence identifier of the text content of the text block in the first document, and the second sequence identifier of the text block.

[0032] In a third aspect, an embodiment of the present application provides a server, comprising at least one processor and a memory connected to the processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program so that the server can implement the data retrieval method described in the first aspect or any implementation of the first aspect, or the text database generation method described in the second aspect.

[0035] In a third aspect, an embodiment of the present application provides a text database generation device, which is applied to a first document, wherein the first document includes a directory, and the directory includes at least one level of titles, and the device includes:

[0036] A document segmentation module, configured to split the text content belonging to the last-level title in the first document into at least one text block; and

[0037] A data storage module is used to store text block index data, wherein the text block index data includes: the text block and metadata, wherein the metadata includes: the title path of the text block, the first sequence identifier of the text content of the text block in the first document, and the second sequence identifier of the text block.

[0038] In a fourth aspect, an embodiment of the present application provides a data retrieval device, comprising:

[0039] A text block retrieval module is used to extract a first number of text blocks from a text database based on a first similarity between a question text and a text block in a text database, wherein the text database includes a plurality of text blocks and metadata corresponding to each of the text blocks, wherein the metadata includes a title path of the corresponding text block, a first sequence identifier of a text content where the text block is located in a first document, and a second sequence identifier of the text block in the text content, wherein the text content is the text content in the first document that belongs to any last-level title;

[0040] an associated text determination module, configured to determine an associated text block of each text block based on a preset context range and first metadata of each text block in the first number of text blocks, wherein the first metadata is metadata corresponding to each text block in the first number of text blocks;

[0041] a text block integration module, configured to sort and integrate each text block and the text blocks associated with each text block based on the first metadata and the second metadata to obtain target text content corresponding to each text block, wherein the second metadata is the metadata of the text blocks associated with each text block; and

[0042] The reply text generation module is used to determine the reply text corresponding to the question text based on the target text content corresponding to each text block.

[0043] In a fifth aspect, an embodiment of the present application provides a server, comprising at least one processor and a memory connected to the processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program so that the server can implement the data retrieval method as described in any one of the first aspects, or the text database generation method as described in the second aspect.

[0046] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are used by an electronic device, such as the data retrieval method described in any one of the first aspects, or the text database generation method described in the second aspect.

[0047] In the seventh aspect, an embodiment of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data retrieval method as described in any one of the first aspect, or the text database generation method as described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of this embodiment, the drawings required for use in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of a database generation or retrieval system provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0051] Figure 3 A flowchart of a data retrieval method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a storage process of a text block provided in an embodiment of the present application;

[0053] Figure 5 A flowchart of a text database generation method provided in an embodiment of the present application;

[0054] Figure 6 A flowchart of a specific application of the data retrieval method provided in an embodiment of the present application;

[0055] Figure 7 A flowchart of restoring relevant documents provided for the embodiment of the present application;

[0056] Figure 8 A structural diagram of a text database generating device provided in an embodiment of the present application;

[0057] Fig. 9 This is a structural diagram of a data retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] It should be noted that the embodiments described in this application are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] In order to make the following embodiments clear, the technical terms involved in this application are first introduced.

[0060] A query is a statement used to retrieve data from a database. It allows users to select, filter, sort, and aggregate data from a database based on specific criteria.

[0061] Prompts are guiding texts provided when interacting with artificial intelligence (AI) dialogue systems.

[0062] See also Figure 1 , Figure 1 1 shows a schematic diagram of a data index or retrieval system architecture. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0063] Among them, a database generation or data retrieval application can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant questions input by the user on the interactive interface and send the above questions to the server 200. The server 200 can obtain processing results based on the received questions and return the processing results to the terminal 100.

[0064] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received question by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0065] Next describe Figure 1 The product form of the mid-terminal 100;

[0066] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0067] Next describe Figure 1 The product form of the server 200;

[0068] Figure 2 A structural diagram of a server 200 is provided, such as Figure 2 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0069] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0070] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0071] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0072] The memory 204 may be used to store software codes related to data indexing or data retrieval methods, and the processor 202 may execute the steps of the data indexing or data retrieval methods of the chip, and may also schedule other units to implement corresponding functions.

[0073] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0074] The embodiment of the present application provides a data retrieval method. The method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0075] Reference Figure 3 An embodiment of the present application provides a data retrieval method, which is applied to a text database, wherein the text database includes a document and a text block and metadata of the document obtained by the text database generation method described in the above embodiment. The data retrieval method specifically includes the following steps:

[0076] 301. Based on a first similarity between the question text and a text block in a text database, a first number of text blocks are extracted from the text database. The text database includes multiple text blocks and metadata corresponding to each text block. The metadata includes a title path of the corresponding text block, a first sequence identifier of the text content where the text block is located in the first document, and a second sequence identifier of the text block in the text content. The text content is the text content in the first document that belongs to any last-level title.

[0077] Specifically, the title path of a text block is the title path corresponding to the text content to which the text block belongs, that is, the path from the last-level title corresponding to the text content to the root node title. Specifically, the text block and the corresponding metadata can be stored in a pre-set text database. The text block is the various components obtained after the text content is segmented. In order to facilitate the restoration of the text content and the document content, each text block will correspond to a metadata, in which the title path of the text block is the path from the root node of the directory to the last-level title. If the last-level title of the text block is Title 3, Title 1 is its root node, and Title 2 is the parent node directly under Title 3, then the title path of the text block is: Title 1 / Title 2 / Title 3. It is necessary to add a first order identifier to the text content contained under Title 3, so as to determine the order of the text content in the entire first document. Similarly, for each text block under each text content, add a second order identifier of the text block in the text content.

[0078] According to the question text input by the user, the BM25 algorithm can be used to retrieve from the text database, and the K text blocks with the highest similarity can be obtained based on the similarity.

[0079] 302. Determine an associated text block of each text block based on a preset context range and first metadata of each text block in the first number of text blocks, where the first metadata is metadata corresponding to each text block in the first number of text blocks.

[0080] Specifically, considering that the length of text content that can be input into the large language model is limited, the length of the related text used in the constructed prompt content cannot be too long. The maximum number of associated text blocks of each text block can be determined according to the preset context range. Then, the maximum number is used as the length of the context window interval of each text block to determine the associated text blocks of each text block.

[0081] Taking the input context length of 4k supported by the large language model as an example, it is required that the retrieved reference text, user questions, the prompt content constructed by the reference text and user questions, and the expected output answer cannot exceed 4K.

[0082] That is, the length of the reference text + the length of the user question + the length of the prompt content + the length of the answer is less than 4K. For example, when the length of the user question + the prompt content + the answer is 1k, the length of the reference text is at most 3k. If the text block is set to 512 bytes, a sliding window is required to obtain the context based on the metadata, and the entire long text cannot be obtained. For example, when there is only one relevant document, the paragraph_id_l2 of the text block with the highest similarity is 5, then the sliding window can be set to [begin->5->end], where begin and end are the context window intervals of text block 5. The length of the text from begin to end after concatenation cannot be greater than 3k, that is, end–begin+1<6, and the text in this interval is used as the reference text. This ensures that the length of the returned reference content can meet the input requirements of the large language model.

[0083] 303. Based on the first metadata and the second metadata, each text block and the associated text blocks of each text block are sorted and integrated to obtain the target text content corresponding to each text block, wherein the second metadata is the metadata of the associated text blocks of each text block.

[0084] Specifically, based on the above text blocks, the text blocks are spliced ​​according to the first sequence identifier, the second sequence identifier and the title path, and then the text content arranged in the order of the document is obtained. Since the spliced ​​and restored text content is a complete component in the document, the integrity and coherence of the prompt content are guaranteed.

[0085] 304. Determine the answer text corresponding to the question text based on the target text content corresponding to each text block.

[0086] Specifically, after multiple target text contents are determined, they can be input into the language model to generate the reply text, because the target text content as the language model prompt text is a complete component in the document, which ensures the integrity and coherence of the prompt content, thereby improving the accuracy of the knowledge question and answer.

[0087] It is understandable that those skilled in the art may also use other search algorithms to search for documents and text blocks, which will not be described in detail here.

[0088] In a possible implementation, in order to further improve the accuracy of the answer content and the speed of answering, the data retrieval method further includes the following steps:

[0089] Based on the first similarity, a second number (generally 1 to 2) of target text contents are extracted from the target text contents corresponding to each text block, and then the second number of target text contents are pushed to the large language model as prompt texts and question texts to obtain reply texts to the question texts, so that after filtering by the large language model, more accurate results can be output for the questions raised by users.

[0090] Specifically, refer to Figure 6 As shown, after the processing of step 301, a total of k text blocks are obtained, and the k text blocks are arranged in descending order of similarity, and then the top q text blocks are taken out. The text content is then restored based on the metadata of the q text blocks, and finally all the obtained text content is sent to the large language model for use.

[0091] Specifically, after searching with vectors and keywords, K text blocks are obtained, which are then sorted according to similarity to obtain the most relevant text block. According to the metadata of the text block, such as paragraph_id_l1, paragraph_id_l2, paragraph_len and file name, multiple text blocks in the window context are obtained to form the corresponding text for use by the large language model.

[0092] Considering that the above data retrieval method may also be applied to purposes other than large language models, such as when it is only used to retrieve relevant documents, as a specific application of the above data retrieval method, refer to Figure 7 As shown, according to the user's question, the BM25 algorithm is applied to retrieve the most relevant K text blocks from the knowledge base, and then based on the preset context range and the first metadata of each text block in the K text blocks, the associated text blocks of each text block are determined, and based on the first metadata and the second metadata, each text block and the associated text blocks of each text block are sorted and integrated to obtain the target text content corresponding to each text block, and based on the target text content corresponding to each text block, the reply text corresponding to the question text is determined and returned as the relevant text segment for answering the question.

[0093] In a possible implementation, the process of acquiring the text block and metadata in the above text database may specifically include:

[0094] Splitting the text content of the first document belonging to the last-level title into at least one text block, wherein the first document includes a directory, and the directory includes at least one level of titles;

[0095] Store text block index data to generate a text database. The text block index data includes: text blocks and metadata

[0096] Specifically, for the first document with directory information, such as documents in the formats of html, chm, word, etc., the directories of these documents contain at least one level of titles. The directory is composed of titles arranged in a hierarchy. Generally, the directory of a document will be composed of titles of multiple levels. For each title, there is a parent title to which it belongs and subtitles contained. Of course, for the lowest level title, there is only a parent title to which it belongs, and no subtitles are included. The lowest level title can be called the last level title. The text content belonging to the last level title can be the text content under the last level title (because the last level title no longer contains a title, only the corresponding text content).

[0097] The first step is to directly determine the text content of each last-level title for any title based on the directory structure.

[0098] The second step is to segment each text content according to the actual search needs after determining the text content. The length of the text block obtained when segmenting the text content should not be too large or too small. If the text block is too large, it will take longer time to perform similarity search and reduce the search speed. If the text block is too small, it is not easy to find similar text blocks because it carries less content. Therefore, according to the actual search needs, the length of the text block can be set to 512 (number of characters) or other segmentation lengths that are easy to segment and the length of the text block obtained is more reasonable.

[0099] It should be noted that the title path of a text block is the title path corresponding to the text content to which the text block belongs, that is, the path from the last-level title corresponding to the text content to the root node title. Specifically, the text block and the corresponding metadata can be stored in a pre-set database. The text block is the various components obtained after the text content is segmented. In order to facilitate the restoration of the text content and the document content, each text block will correspond to a metadata, wherein the title path of the text block is the path from the root node of the directory to the last-level title. If the last-level title of the text block is Title 3, Title 1 is its root node, and Title 2 is the parent node directly subordinate to Title 3, then the title path of the text block is: Title 1 / Title 2 / Title 3. It is necessary to add a first order identifier to the text content contained under Title 3, so as to determine the order of the text content in the entire first document. Similarly, for each text block under each text content, add a second order identifier of the text block in the text content.

[0100] In this way, when performing similarity retrieval of text blocks, after obtaining the target text block, the first sequence identifier and the second sequence identifier of the target text block can be used to restore the complete text content under the target last-level title corresponding to the target text block. All text content under the parent title of the target last-level title can be restored according to the first sequence identifier and the title path, and all text content under the root node can be directly restored according to the title path. This ensures the integrity and coherence of the returned content in the document and improves the accuracy of the answer.

[0101] In a possible implementation, in order to improve the efficiency and accuracy of text block segmentation, the above-mentioned splitting of the text content under the last-level title into at least one text block may specifically include the following steps:

[0102] Step 1: Mark the text content to obtain the text content with markers.

[0103] Specifically, refer to Figure 4 As shown in the figure, for documents in HTML, CHM, Word and other formats with titles, first extract the text content under each title according to the directory structure, and then use the formatting conversion library to convert each text content into a text in a lightweight markup language, such as markdown format, and save it in JSON format. The obtained document content is as follows:

[0104] "id":1,

[0105] "title":" / FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost",

[0106] "content":"\n####\nAlarm Explanation\nAlarm Description:\n```\nThe heartbeat signal between the device and the system is abnormal.\n```\n When FusionOne Center detects the heartbeat of the device, this alarm is generated if the heartbeat detection fails.\n####\nAlarm Attributes\n####\nAlarm Parameters\n|\nAlarm ID\n|\nAlarm Level\n|\nAuto Clear\n|\n|---|---|---|\n|\n0x1000000101000000\n|\nEmergency\n|\nYes\n|\n|\nParameter Name\n|\nParameter Meaning\n|\n|---|---|\n|\n arg1\n|\nIP address of the device. \n|\n####\nImpact on the system\nThis alarm indicates that the heartbeat between the FusionOneCenter service and the device is lost, and the device cannot be managed normally, which will affect all user operations on the device. \n####\nPossible causes\n*The network connection between the device and FusionOne Center is abnormal. \n*Device failure. \n####\nSolution steps\n1. Log in to FusionOne Center WebUI. \n2. Check the device and FusionOne Center. Is the network communication between the centers normal? \nCheck whether the network communication is normal, for example: whether the network interface is disconnected, whether the routing configuration is normal, whether the regional network is normal, etc. \n\t*If the network communication is normal, >execute\n\t[4]\n\t. \n\t*If the network communication is abnormal, restore the network connection and check whether the alarm is restored. If the network cannot be restored, execute\n\t[4]\n\t. \n\t\t+If the alarm has been restored, the alarm processing is completed. \n\t\t+If the alarm is still >If it exists, execute\n\t\t[4]\n\t\t. \n3. Check whether the device is operating normally. \n\t*If the device is not operating normally, restore the device to normal operation and check whether the alarm is restored. \n\t\t+If the alarm is restored, the alarm processing is complete. \n\t\t+If the alarm still exists, execute\n\t\t[4]\n\t\t. \n\t*If the device is operating normally, execute\n\t[4]\n\t. \n4. Contact technical support engineers for help. \n"

[0107] The content after "content": is the text content, the content after "title": is the title path of the text block, and the content after "id": indicates the sequence number of the text content (i.e. the first order identifier).

[0108] Among them, ###, |---|, n\t, etc. are markers such as titles and table tags unique to the markdown format.

[0109] Step 2: based on a preset segmentation length and a preset text block segmentation judgment rule, segment the text content with the marker to obtain one or more text blocks.

[0110] Specifically, refer to Figure 4 As shown, when performing text segmentation, the preset segmentation length can be set to 512 bytes, that is, the length of the segmented text block is not greater than 512 bytes. When segmenting, the text content can be segmented according to the marker selected from the markers each time, and it is determined whether the length of each segmented text block is not greater than 512 bytes, and the lengths of two adjacent text blocks are not less than 512 bytes after being combined. If after the text content is segmented, the obtained text block meets the above segmentation judgment conditions, it will no longer be segmented.

[0111] In order to further improve the efficiency of text block segmentation, each time a marker selected from the markers is used to segment the content, the markers may be selected in descending order of priority, and then the text content may be segmented based on the selected markers, wherein the priority of the currently selected marker is lower than the priority of the previously selected marker.

[0112] For example, for the text content after "content": above, the text content obtained by the previous segmentation can be segmented using the selected markers in descending order of priority: '\n\n', '\n', '. ', '! ', ','. The preset segmentation length is 512 bytes. When the text block length is less than 512 bytes and the length combined with the adjacent text block exceeds the set length of 512 bytes, no segmentation is performed.

[0113] For example, after using '\n' to split, ['alarm explanation', 'alarm description:', ..., 'if the device is running normally, execute', ...] are generated. The continuous text blocks are concatenated to meet the text block length of just less than 512. After adding the file name and the title path of the text block, it is put into the result set to obtain a segmented result set.

[0114] The contents of the four text blocks finally obtained by segmentation are as follows:

[0115] 1.'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOneCenter Alarm List / 0x1000000101000000 Device Heartbeat Lost\n####Alarm Explanation\nAlarm Description:\n```\nThe heartbeat signal between the device(IP:\narg1\n)and the systemisabnormal.\n```\nWhen FusionOne Center detects the heartbeat of the device, if the heartbeat detection fails, this alarm is generated.\n####Alarm Attributes\n|Alarm ID|Alarm Level|Can be Cleared Automatically|\n|---|---|---|\n|0x1000000101000000|Urgent|Yes|\n####Alarm Parameters'.

[0116] 2.'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOneCenter Alarm List / 0x1000000101000000 Device Heartbeat Lost\n|Parameter Name|Parameter Meaning|\n|---|---|\n|\narg1\n|Device IP Address. |\n####Impact on the System\nThis alarm is generated, indicating that the heartbeat between the FusionOne Center service and the device is lost, and the device cannot be managed normally, which will affect all user operations on the device. \n####Possible Cause\nThe network connection between the device and FusionOne Center is abnormal. \nDevice failure. \n####Handling Steps\n1.Log in to the FusionOne Center WebUI.

[0117] 3.'FusionOne HCI 23 23.1.1 Alarm Processing 01\nFusionOne Center Alarm / FusionOneCenter Alarm List / 0x1000000101000000 Device Heartbeat Lost\n2.Check whether the network communication between the device and FusionOne Center is normal. Check whether the network communication is normal, for example: whether the network interface is disconnected, whether the routing configuration is normal, whether the regional network is normal, etc. \n\t\nIf the network communication is normal, execute [4]. \n\t\nIf the network communication is abnormal, restore the network connection and check whether the alarm is restored. If the network cannot be restored, execute [4]. \n\t\t+If the alarm has been restored, the alarm processing is complete. \n\t\t+If the alarm still exists, execute [4]. \n3.Check whether the device is running normally. \n\t\nIf the device is not running normally, restore the device to normal operation and check whether the alarm is restored. ',

[0118] 4.'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOneCenter Alarm List / 0x1000000101000000 Device Heartbeat Lost\n+If the alarm has been restored, alarm handling is complete. \n\t\t+If the alarm still exists, execute [4]. \n\t\nIf the device is running normally, execute [4]. \n4.]Please contact technical support engineers for help.

[0119] Sequence numbers 1 to 4 are four text blocks obtained by segmentation, and "FusionOne HCI 23 23.1.1 Alarm Processing 01" is the file name.

[0120] "FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost" is the title path of the text block.

[0121] Based on the above text blocks, it is necessary to add corresponding metadata to each text block. After adding metadata, the four parts of content stored in the database are as follows:

[0122] 1.Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost\n####Alarm Explanation\nAlarm Description:\n```\nThe heartbeat signal between the device(IP:\narg1\n)and the system is abnormal.\n```\nFusionOne When the Center detects the heartbeat of the device, if the heartbeat detection fails, this alarm is generated.\n####Alarm attributes\n|Alarm ID|Alarm level|Can be automatically cleared|\n|---|---|---|\n|0x1000000101000000|Emergency|Yes|\n####Alarm parameters',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 2323.1.1 Alarm Processing 01','title':'FusionOne Center Alarm / FusionOne Center alarm list / 0x1000000101000000 device heartbeat lost','content_embeded':array([0.00509322,-0.03263689,-0.02818436,...,-0.01265258,0.01112661,-0.01226701],dtype=float32),'paragraph_id_l1':2,'paragraph_id_l2':0,'paragraph_len':4}).

[0123] 2.Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost\n|Parameter Name|Parameter Meaning|\n|---|---|\n|\narg1\n|Device IP Address.|\n#### Impact on the System\nThis alarm indicates that the heartbeat between the FusionOne Center service and the device is lost, and the device cannot be managed normally, which affects all user operations on the device.\n#### Possible Causes\nThe network connection between the device and FusionOne Center is abnormal.\nDevice failure.\n#### Handling Steps\n1. Log in to the FusionOne Center WebUI.',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 2323.1.1 Alarm Processing 01','title':'FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss','content_embeded':array([-0.00316077,-0.02642941,-0.02343957,...,-0.01106526,0.0077391,-0.0079075],dtype=float32),'paragraph_id_l1':2,'paragraph_id_l2':1,'paragraph_len':4}).

[0124] 3.Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost\n2.Check the Device and FusionOne Is the network communication between the Centers normal? Check whether the network communication is normal, for example: whether the network interface is disconnected, whether the routing configuration is normal, whether the regional network is normal, etc. \n\t\nIf the network communication is normal, execute [4]. \n\t\nIf the network communication is abnormal, restore the network connection and check whether the alarm is restored. If the network cannot be restored, execute [4]. \n\t\t+If the alarm has been restored, the alarm processing is complete. \n\t\t+If the alarm still exists, execute [4]. \n3. Check whether the device is running normally. \n\t\nIf the device is not running normally, restore the device to normal operation and check whether the alarm is restored. ',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 23 23.1.1 Alarm Processing 01','title':'FusionOne Center Alarm / FusionOne Center alarm list / 0x1000000101000000 device heartbeat lost','content_embeded':array([0.00960491,-0.03210478,-0.02278469,...,-0.01252992,-0.02566933,-0.02070089],dtype=float32),'paragraph_id_l1':2,'paragraph_id_l2':2,'paragraph_len':4}).

[0125] 4.Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost\n+If the alarm has been restored, alarm handling is complete.\n\t\t+If the alarm still exists, execute [4].\n\t\nIf the device is running normally, execute [4].\n4.Contact technical support engineers for help.',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 23 23.1.1 Alarm Handling 01','title':'FusionOne Center Alarm / FusionOne Center alarm list / 0x1000000101000000 device heartbeat lost','content_embeded':array([0.00483514,-0.02553011,-0.01561882,...,-0.01515852,-0.01288328,-0.00948918],dtype=float32),'paragraph_id_l1':2,'paragraph_id_l2':3,'paragraph_len':4}).

[0126] The content before "metadata" in each part is the content of the above text block, and the content after "metadata" is metadata, where:

[0127] 'source':'zh-cn_topic_0000001151233327.html', indicating the source of the HTML format file;

[0128] 'file_name':'FusionOne HCI 23 23.1.1 Alarm Processing 01', indicating the file name;

[0129] 'title':'FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost', indicating the title path of the text block;

[0130] 'paragraph_id_l1', which indicates the order in which the text content is arranged in the document, i.e. the first order identifier;

[0131] 'paragraph_id_l2', which indicates the order in which the text block is arranged under the text content, i.e. the second order identifier;

[0132] In order to speed up the restoration of the text content, 'paragraph_len' is added here to indicate the number of text blocks contained in the text content. Then, when the second order identifier of the text block and the number of text blocks contained in the text content are obtained, the text content can be searched and restored more quickly.

[0133] Among them, refer to Figure 4 As shown, the obtained text block index data is finally stored to generate a text database. In some embodiments, considering that most retrieval algorithms perform similarity calculations on vectors during retrieval, the obtained text blocks can be converted into word vectors through a word embedding model and stored together with the text blocks, so that the corresponding word vectors can be directly used for retrieval during retrieval, thereby improving the retrieval speed.

[0134] The content after 'content_embeded' is the array for storing word vectors, which stores the content of the word vector corresponding to the text block.

[0135] It is understandable that those skilled in the art may select and adjust the content of the above metadata as needed, and no limitation is made here.

[0136] Reference Figure 5 , Figure 5 A flow chart of a text database generation method provided in an embodiment of the present application can be used to process a first document having directory information, such as documents in formats such as html, chm, word, etc., to generate a database, wherein the directory of the document contains at least one level of titles.

[0137] like Figure 5 As shown, the text database generation method provided in the embodiment of the present application may include steps 501 to 502, and these steps are described in detail below.

[0138] 501. Split the text content belonging to the last-level title in the first document into at least one text block.

[0139] The first document is a first document having directory information, such as a document in the format of html, chm, word, etc.

[0140] The table of contents of the first document consists of only one level of headings.

[0141] Specifically, the directory is composed of titles arranged in a hierarchy. The directory of a general document is composed of titles of multiple levels. For each title, there is a parent title to which it belongs and subtitles it contains. Of course, for the lowest-level title, there is only a parent title to which it belongs, and no subtitles are included. The lowest-level title can be called the last-level title. The text content belonging to the last-level title can be the text content under the last-level title (because the last-level title no longer contains a title, only the corresponding text content).

[0142] The first step is to directly determine the text content of each last-level title for any title based on the directory structure.

[0143] The second step is to segment each text content according to the actual search needs after determining the text content. The length of the text block obtained when segmenting the text content should not be too large or too small. If the text block is too large, it will take longer time to perform similarity search and reduce the search speed. If the text block is too small, it is not easy to find similar text blocks because it carries less content. Therefore, according to the actual search needs, the length of the text block can be set to 512 (number of characters) or other segmentation lengths that are easy to segment and the length of the text block obtained is more reasonable.

[0144] It should be noted that those skilled in the art can adjust and select the segmentation length of the text block according to the needs of actual applications, and no limitation is made here.

[0145] 302. Store text block index data, wherein the text block index data includes: the text block and metadata, wherein the metadata includes: a title path of the text block, a first sequence identifier of the text content of the text block in the first document, and a second sequence identifier of the text block.

[0146] It should be noted that the title path of a text block is the title path corresponding to the text content to which the text block belongs, that is, the path from the last-level title corresponding to the text content to the root node title. Specifically, the text block and the corresponding metadata can be stored in a pre-set database. The text block is the various components obtained after the text content is segmented. In order to facilitate the restoration of the text content and the document content, each text block will correspond to a metadata, wherein the title path of the text block is the path from the root node of the directory to the last-level title. If the last-level title of the text block is Title 3, Title 1 is its root node, and Title 2 is the parent node directly subordinate to Title 3, then the title path of the text block is: Title 1 / Title 2 / Title 3. It is necessary to add a first order identifier to the text content contained under Title 3, so as to determine the order of the text content in the entire first document. Similarly, for each text block under each text content, add a second order identifier of the text block in the text content.

[0147] In this way, when performing similarity retrieval of text blocks, after obtaining the target text block, the first sequence identifier and the second sequence identifier of the target text block can be used to restore the complete text content under the target last-level title corresponding to the target text block. All text content under the parent title of the target last-level title can be restored according to the first sequence identifier and the title path, and all text content under the root node can be directly restored according to the title path. This ensures the integrity and coherence of the returned content in the document and improves the accuracy of the answer.

[0148] It should be noted that other specific implementations of the above-mentioned text database generation method can refer to the relevant parts of the text database generation in the above-mentioned data retrieval method, which will not be repeated here.

[0149] The above describes the text database generation method and the data retrieval method provided in the embodiments of the present application. The following describes the apparatus for executing the above text database generation method and the data retrieval method.

[0150] See also Figure 8 8 is a schematic diagram of the structure of a text database generation device provided in an embodiment of the present application. Figure 8 As shown, the text database generating device comprises:

[0151] A document segmentation module 801 is used to segment the text content belonging to the last-level title in the first document into at least one text block; and

[0152] The data storage module 802 is used to store text block index data, which includes: text block and metadata, and the metadata includes: the title path of the text block, the first sequence identifier of the text content of the text block in the first document, and the second sequence identifier of the text block.

[0153] Fig. 9 A data retrieval device provided for this application includes:

[0154] The text block retrieval module 901 is used to extract a first number of text blocks from the text database based on a first similarity between the question text and the text blocks in the text database, wherein the text database includes a plurality of text blocks and metadata corresponding to each text block, wherein the metadata includes a title path corresponding to the text block, a first sequence identifier of the text content where the text block is located in the first document, and a second sequence identifier of the text block in the text content, wherein the text content is the text content in the first document belonging to any last-level title;

[0155] An associated text determination module 902 is used to determine an associated text block of each text block based on a preset context range and first metadata of each text block in the first number of text blocks, where the first metadata is metadata corresponding to each text block in the first number of text blocks;

[0156] A text block integration module 903 is used to sort and integrate each text block and the associated text blocks of each text block based on the first metadata and the second metadata to obtain the target text content corresponding to each text block, wherein the second metadata is the metadata of the associated text blocks of each text block; and

[0157] The reply text generation module 904 is used to determine the reply text corresponding to the question text based on the target text content corresponding to each text block.

[0158] In a possible implementation, the process of the reply text generating module 904 determining the reply text corresponding to the question text based on the target text content corresponding to each text block includes:

[0159] Based on the first similarity, extracting a second amount of the target text content from the target text content corresponding to each text block;

[0160] The second amount of target text content is pushed to the large language model as prompt text and question text to obtain a reply text for the question text.

[0161] In a possible implementation, the process of the associated text determination module 902 determining the associated text blocks of each text block based on the preset context range and the first metadata of each text block in the first number of text blocks includes:

[0162] Based on a preset context range, determining a maximum number of associated text blocks for each text block;

[0163] The maximum number is used as the length of the context window interval of each text block, and the associated text blocks of each text block are determined.

[0164] In a possible implementation, the method further includes: a text database generation module for

[0165] Splitting the text content of the first document belonging to the last-level title into at least one text block, wherein the first document includes a directory, and the directory includes at least one level of titles;

[0166] The text block index data is stored to generate a text database, wherein the text block index data includes: text blocks and metadata.

[0167] In a possible implementation, the text database generation module divides the text content of the first document belonging to the last-level title into at least one text block, including:

[0168] Marking the text content to obtain the text content with markers;

[0169] Based on the preset segmentation length and the preset text block segmentation judgment rule, the text content with the marker is segmented to obtain one or more text blocks.

[0170] In a possible implementation, the text database generation module segments the text content with the markers based on a preset segmentation length and a preset text block segmentation judgment rule, including:

[0171] The text content is segmented by the markers selected from the markers in a preset order, and it is determined whether the length of each text block obtained by each segmentation is not greater than the preset segmentation length, and the lengths of two adjacent text blocks are not less than the preset segmentation length.

[0172] In a possible implementation, the process of segmenting the text content by the text database generation module according to the markers selected from the markers in a preset order includes:

[0173] The markers are selected in descending order of priority, wherein the priority of the marker currently selected is lower than the priority of the marker previously selected;

[0174] The text content obtained from the previous segmentation is segmented based on the markers selected each time.

[0175] In a possible implementation, the process in which the text database generation module performs tagging processing on the text content to obtain the text content with the tag includes:

[0176] Convert the text content into markdown format based on a preset conversion tool;

[0177] Input the text in markdown format into the json converter to process the tag information and obtain the text content with tags.

[0178] An embodiment of the present application further provides a server, comprising at least one processor and a memory connected to the processor, wherein:

[0179] The memory is used to store computer programs;

[0180] The processor is used to execute the computer program so that the server can implement the text database generation method or data retrieval method described in the above embodiments.

[0181] An embodiment of the present application further provides a computer-readable storage medium, the storage medium carrying one or more computer programs, and when the one or more computer programs are executed by an electronic device, the text database generation method or data retrieval method described in the above embodiment is implemented.

[0182] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data retrieval method or text database generation method of the above embodiment.

[0183] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0184] The "first" and "second" (if any) in the names mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0185] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0186] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0187] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A data retrieval method, It is characterized in that include: Based on a first similarity between the question text and a text block in a text database, extracting a first number of text blocks from the text database, wherein the text database includes a plurality of text blocks and metadata corresponding to each of the text blocks, wherein the metadata includes a title path of the corresponding text block, a first sequence identifier of a text content where the text block is located in a first document, and a second sequence identifier of the text block in the text content, wherein the text content is a text content in the first document that belongs to any last-level title; Determining an associated text block of each text block based on a preset context range and first metadata of each text block in the first number of text blocks, wherein the first metadata is metadata corresponding to each text block in the first number of text blocks; Based on the first metadata and the second metadata, each text block and the associated text blocks of each text block are sorted and integrated to obtain the target text content corresponding to each text block, wherein the second metadata is the metadata of the associated text blocks of each text block; Based on the target text content corresponding to each of the text blocks, the answer text corresponding to the question text is determined.

2. The data retrieval method according to claim 1, It is characterized in that Determining the answer text corresponding to the question text based on the target text content corresponding to each of the text blocks includes: Based on the first similarity, extracting a second amount of the target text content from the target text content corresponding to each of the text blocks; The second amount of target text content is pushed to the large language model as prompt text and question text to obtain a reply text for the question text.

3. The data retrieval method according to claim 1 or 2, It is characterized in that The determining, based on the preset context range and the first metadata of each text block in the first number of text blocks, an associated text block of each text block includes: Based on the preset context range, determining the maximum number of associated text blocks of each text block; The maximum number is used as the length of the context window interval of each text block to determine the associated text block of each text block.

4. The data retrieval method according to any one of claims 1 to 3, It is characterized in that Also includes: Splitting the text content of the first document belonging to the last-level title into at least one text block, wherein the first document includes a directory, and the directory includes at least one level of titles; The text block index data is stored to generate the text database, wherein the text block index data includes: the text block and the metadata.

5. The data retrieval method according to claim 4, It is characterized in that The step of splitting the text content of the first document belonging to the last-level title into at least one text block includes: Marking the text content to obtain text content with markers; Based on a preset segmentation length and a preset text block segmentation judgment rule, the text content with the marker is segmented to obtain one or more text blocks.

6. The data retrieval method according to claim 5, It is characterized in that The segmenting of the text content with the marker based on the preset segmentation length and the preset text block segmentation judgment rule includes: The text content is segmented using the markers selected from the markers in a preset order, and it is determined whether the length of each text block obtained by each segmentation is not greater than a preset segmentation length, and whether the lengths of two adjacent text blocks are not less than the preset segmentation length.

7. The data retrieval method according to claim 6, It is characterized in that The segmenting of the text content by the markers selected from the markers in a preset order includes: Selecting the markers in order of priority from high to low, wherein the priority of the marker currently selected is lower than the priority of the marker previously selected; The text content obtained from the previous segmentation is segmented based on the markers selected each time.

8. The data retrieval method according to claim 5, It is characterized in that The marking process of the text content to obtain the text content with the marker includes: Convert the text content into markdown format based on a preset conversion tool; The text in the markdown format is input into a json converter to process the markup information to obtain the text content with the marker.

9. A method for generating a text database, It is characterized in that Applied to a first document, the first document includes a directory, the directory includes at least one level of titles, the method includes: Splitting the text content belonging to the last-level title in the first document into at least one text block; Store text block index data, the text block index data includes: the text block and metadata, the metadata includes: the title path of the text block, the first sequence identifier of the text content of the text block in the first document, and the second sequence identifier of the text block.

10. A server, It is characterized in that The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the server can implement the data retrieval method according to any one of claims 1 to 8, or the text database generation method according to claim 9.

Citation Information

Patent Citations

  • Document question and answer method based on multi-way tree and large-scale language model and related equipment

    CN116932730A

  • Knowledge base construction method and question and answer dialogue method and system based on generative large language model

    CN117056471A

  • Question answering method, question answering system and computer readable storage medium

    CN117520488A

  • Intelligent question answering method and device based on large language model, equipment and storage medium

    CN117556020A

  • Question answering method, apparatus and device, and medium

    CN117591639A

Cited By

  • Data retrieval method, text database generation method, and server

    WO2026026021A1