Data retrieval method, text database generation method, and server

Abstract

The present application provides a data retrieval method, a text database generation method, and a server. The data retrieval method comprises: on the basis of a first similarity between a question text and text blocks in a text database, extracting a first number of text blocks from the text database; on the basis of a preset context range and first metadata of each text block among the first number of text blocks, determining an associated text block of each text block; on the basis of the first metadata and second metadata, ranking and integrating the text blocks and the associated text blocks of the text blocks to obtain target text content corresponding to each text block; and on the basis of the target text content corresponding to each text block, determining an answer text corresponding to the question text. The data retrieval method provided in embodiments of the present application improves the accuracy of knowledge-based question answering.

Description

Data retrieval methods, text database generation methods, and servers

[0001] This disclosure claims priority to Chinese Patent Publication No. 202411044782.9, filed on July 31, 2024, entitled "Data Retrieval Method, Text Database Generation Method and Server", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a data retrieval method, a text database generation method, and a server. Background Technology

[0003] Currently, the accuracy of answers provided by knowledge-based question-and-answer systems is relatively low. Summary of the Invention

[0004] This application provides a data retrieval method, a text database generation method, and a server, aiming to solve the problem of accuracy in answering questions in knowledge-based question-and-answer systems.

[0005] In a first aspect, embodiments of this application provide a data retrieval method, including:

[0006] Based on the first similarity between the question text and the text blocks in the text database, a first number of text blocks are extracted from the text database. The text database includes multiple text blocks and metadata corresponding to each text block. The metadata includes the title path of the corresponding text block, the first order identifier of the text content in the first document, and the second order identifier of the text block in the text content. The text content is the text content in the first document that belongs to any final heading.

[0007] Based on a preset context range and the first metadata of each text block in the first number of text blocks, the associated text block of each text block is determined, wherein the first metadata is the metadata corresponding to each text block in the first number of text blocks;

[0008] Based on the first metadata and the second metadata, each text block and its associated text blocks are sorted and integrated to obtain the target text content corresponding to each text block. The second metadata is the metadata of the associated text blocks of each text block.

[0009] Based on the target text content corresponding to each text block, the response text corresponding to the question text is determined.

[0010] In one possible implementation, determining the response text corresponding to the question text based on the target text content corresponding to each text block includes:

[0011] Based on the first similarity, a second number of target text contents are extracted from the target text contents corresponding to each text block;

[0012] The second quantity of the target text content is pushed to the large language model as prompt text and question text to obtain the response text to the question text.

[0013] The data retrieval method provided in this application embodiment ensures that the target text content obtained based on metadata is a complete component of the document, thereby guaranteeing the completeness and coherence of the prompt content and improving the accuracy of knowledge-based question answering.

[0014] In one possible implementation, determining the associated text block for each text block based on a preset context scope and the first metadata of each text block in the first number of text blocks includes:

[0015] Based on the preset context range, the maximum number of associated text blocks for each text block is determined;

[0016] The maximum number is used as the length of the context window interval of each text block to determine the associated text blocks of each text block.

[0017] In one possible implementation, the data retrieval method further includes:

[0018] The text content of the first document belonging to the lowest level heading is split into at least one text block, wherein the first document contains a table of contents, and the table of contents contains at least one level of headings;

[0019] The text block index data is stored to generate the text database, and the text block index data includes the text blocks and the metadata.

[0020] In one possible implementation, splitting the text content of the first document belonging to the lowest-level heading into at least one text block includes:

[0021] The text content is marked to obtain text content with markers;

[0022] Based on a preset segmentation length and a preset text block segmentation judgment rule, the text content with markers is segmented to obtain one or more text blocks.

[0023] In one possible implementation, the segmentation of the text content with markers based on a preset segmentation length and preset text block segmentation judgment rules includes:

[0024] The text content is segmented according to the markers selected from the markers in a preset order, and it is determined whether the length of each text block obtained by each segmentation is not greater than the preset segmentation length, and whether the length of two adjacent text blocks is not less than the preset segmentation length.

[0025] In one possible implementation, the step of segmenting the text content using markers selected from the markers in a preset order includes:

[0026] The markers are selected sequentially in descending order of priority, wherein the priority of the currently selected marker is lower than the priority of the previously selected marker.

[0027] The text content obtained from the previous segmentation is segmented based on the selected markers each time.

[0028] In one possible implementation, the step of marking the text content to obtain text content with markers includes:

[0029] The text content is converted into Markdown format using a preset conversion tool.

[0030] The Markdown-formatted text is input into a JSON converter to process the markup information, resulting in the text content with the markup symbols.

[0031] Secondly, embodiments of this application provide a text database generation method applied to a first document, the first document containing a table of contents, the table of contents containing at least one level of headings, the method comprising:

[0032] The text content belonging to the lowest level heading in the first document is split into at least one text block;

[0033] The text block index data is stored, which includes the text block and metadata. The metadata includes the title path of the text block, the first order identifier of the text content in the first document, and the second order identifier of the text block.

[0034] Thirdly, embodiments of this application provide a server, including at least one processor and a memory connected to the processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is configured to execute the computer program to enable the server to implement the data retrieval method as described in the first aspect or any implementation thereof, or the text database generation method as described in the second aspect.

[0037] Fourthly, embodiments of this application provide a text database generation apparatus applied to a first document, the first document including a table of contents, the table of contents including at least one level of headings, the apparatus comprising:

[0038] The document segmentation module is used to split the text content belonging to the lowest level heading in the first document into at least one text block; and

[0039] The data storage module is used to store text block index data, which includes the text block and metadata. The metadata includes the title path of the text block, the first order identifier of the text content in the first document, and the second order identifier of the text block.

[0040] Fifthly, embodiments of this application provide a data retrieval device, comprising:

[0041] The text block retrieval module is used to extract a first number of text blocks from the text database based on the first similarity between the question text and the text blocks in the text database. The text database includes multiple text blocks and metadata corresponding to each text block. The metadata includes the title path of the corresponding text block, the first order identifier of the text content in the first document, and the second order identifier of the text block in the text content. The text content is the text content in the first document that belongs to any final heading.

[0042] The associated text determination module is used to determine the associated text block of each text block based on a preset context range and the first metadata of each text block in the first number of text blocks, wherein the first metadata is the metadata corresponding to each text block in the first number of text blocks;

[0043] A text block integration module is used to sort and integrate each text block and its associated text blocks based on the first metadata and the second metadata to obtain the target text content corresponding to each text block; the second metadata is the metadata of the associated text blocks of each text block; and

[0044] The response text generation module is used to determine the response text corresponding to the question text based on the target text content corresponding to each text block.

[0045] Sixthly, embodiments of this application provide a server, including at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is used to execute the computer program so that the server can implement the data retrieval method as described in any one of the first aspects, or the text database generation method as described in the second aspect.

[0048] In a seventh aspect, embodiments of this application provide a computer-readable storage medium carrying one or more computer programs, which are used by an electronic device such as the data retrieval method described in any of the first aspects, or the text database generation method described in the second aspect.

[0049] Eighthly, embodiments of this application provide a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the data retrieval method as described in any one of the first aspects, or the text database generation method as described in the second aspect. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of this embodiment, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 is a schematic diagram of a database generation or retrieval system provided in an embodiment of this application;

[0052] Figure 2 is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0053] Figure 3 is a flowchart illustrating the data retrieval method provided in an embodiment of this application;

[0054] Figure 4 is a schematic diagram of the storage process of a text block provided in an embodiment of this application;

[0055] Figure 5 is a flowchart of the text database generation method provided in an embodiment of this application;

[0056] Figure 6 is a flowchart illustrating a specific application of the data retrieval method provided in the embodiments of this application;

[0057] Figure 7 is a flowchart illustrating the restoration of related documents provided in the embodiments of this application;

[0058] Figure 8 is a structural diagram of the text database generation device provided in an embodiment of this application;

[0059] Figure 9 is a structural diagram of the data retrieval device provided in an embodiment of this application. Detailed Implementation

[0060] It should be noted that the embodiments described in this application are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] To make the following embodiments clear, the technical terms involved in the embodiments of this application will be introduced first.

[0062] A query is a statement used to retrieve data from a database. It allows users to select, filter, sort, and aggregate data from the database based on specific criteria.

[0063] Prompts are guiding texts provided when interacting with artificial intelligence (AI) dialogue systems.

[0064] Referring to Figure 1, Figure 1 illustrates a schematic diagram of a data indexing or retrieval system architecture. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers (Figure 1 illustrates this with an example of one server), and the server 200 may provide the methods provided in the embodiments of this application to one or more terminals.

[0065] The terminal 100 may be equipped with a database generation or data retrieval application. The application and webpage can provide an interface. The terminal 100 can receive relevant questions entered by the user on the interactive interface and send the questions to the server 200. The server 200 can obtain the processing result based on the received questions and return the processing result to the terminal 100.

[0066] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received question on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0067] The product form of terminal 100 in Figure 1 will be described next;

[0068] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0069] The product form of server 200 in Figure 1 will be described next;

[0070] Figure 2 provides a schematic diagram of the structure of a server 200. As shown in Figure 2, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0071] Bus 201 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in Figure 2, but this does not indicate that there is only one bus or one type of bus.

[0072] The processor 202 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0073] Memory 204 may include volatile memory, such as random access memory (RAM). Memory 204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0074] The memory 204 can be used to store software code related to data indexing or data retrieval methods, and the processor 202 can execute the steps of the chip's data indexing or data retrieval methods, and can also schedule other units to achieve corresponding functions.

[0075] It should be understood that the aforementioned terminal 100 and server 200 can be centralized or distributed devices. The processors (e.g., processor 170 and processor 202) in the aforementioned terminal 100 and server 200 can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the processor can be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0076] This application provides a data retrieval method. The method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0077] Referring to Embodiment 3 of this application, a data retrieval method is provided, applied to a text database. The text database includes documents and text blocks and metadata of the documents obtained according to the text database generation method described in the above embodiments. The data retrieval method specifically includes the following steps:

[0078] 301. Based on the first similarity between the problem text and the text blocks in the text database, extract the first number of text blocks from the text database. The text database includes multiple text blocks and metadata corresponding to each text block. The metadata includes the title path of the corresponding text block, the first order identifier of the text content in the first document, and the second order identifier of the text block in the text content. The text content is the text content in the first document that belongs to any final heading.

[0079] Specifically, the title path of a text block is the title path corresponding to the text content to which the text block belongs, that is, the path from the last-level heading of the text content to the root node title. Specifically, text blocks and their corresponding metadata can be stored in a pre-defined text database. A text block is the component obtained after segmenting the text content. To facilitate the reconstruction of text content and document content, each text block corresponds to metadata, where the title path of the text block is the path from the root node of the directory to the last-level heading. For example, if the last-level heading of a text block is Heading 3, Heading 1 is its root node, and Heading 2 is the direct parent node of Heading 3, then the title path of the text block is: Heading 1 / Heading 2 / Heading 3. A first order identifier needs to be added to the text content contained under Heading 3 to determine the order of the text content in the entire first document. Similarly, a second order identifier is added to each text block under each text content within that text content.

[0080] Based on the question text input by the user, the BM25 algorithm can be used to retrieve the K most similar text blocks from the text database.

[0081] 302. Based on the preset context range and the first metadata of each text block in the first number of text blocks, determine the associated text block of each text block, where the first metadata is the metadata corresponding to each text block in the first number of text blocks.

[0082] Specifically, considering that the length of text content that a large language model can input is limited, the length of related text used in the constructed prompts cannot be too long. Based on this preset context range, the maximum number of associated text blocks for each text block can be determined. Then, this maximum number is used as the length of the context window interval for each text block to determine the associated text blocks for each text block.

[0083] Taking the 4k-sized context length that a large language model can support as an example, the retrieved reference text, user question, the prompt content constructed by combining the reference text and user question, and the expected output answer must not exceed 4K.

[0084] The length of the reference text plus the length of the user question plus the length of the prompt plus the length of the answer must be less than 4KB. For example, if the length of the user question plus the prompt plus the answer is 1KB, then the maximum length of the reference text is 3KB. Since the text block is set to 512 bytes, a sliding window is needed when obtaining the context from the metadata, instead of retrieving the entire long text. For example, if there is only one relevant document, and the paragraph_id_l2 of the most similar text block is 5, then the sliding window can be set to [begin->5->end], where begin and end are the context window range for text block 5. The length of the concatenated text from begin to end cannot exceed 3KB, i.e., end – begin + 1 < 6. Text within this range is used as the reference text. This ensures that the length of the returned reference content meets the input requirements of the large language model.

[0085] 303. Based on the first metadata and the second metadata, sort and integrate each text block and its associated text blocks to obtain the target text content corresponding to each text block. The second metadata is the metadata of the associated text blocks of each text block.

[0086] Specifically, based on the aforementioned text blocks, they are concatenated according to the first and second sequence identifiers and the title path to obtain text content arranged in the order it appears in the document. Since the concatenated text content forms a complete component within the document, the integrity and coherence of the prompts are ensured.

[0087] 304. Based on the target text content corresponding to each text block, determine the response text corresponding to the question text.

[0088] Specifically, after identifying multiple target text contents, they can be input into a language model to generate response text. This is because the target text contents, which serve as prompts for the language model, are a complete component within the document, ensuring the completeness and coherence of the prompts and thus improving the accuracy of knowledge-based question answering.

[0089] It is understood that those skilled in the art can also use other retrieval algorithms to retrieve documents and text blocks, which will not be elaborated here.

[0090] In one possible implementation, to further improve the accuracy and speed of the response, the data retrieval method also includes the following steps:

[0091] Based on the first similarity, a second number (usually 1 to 2) of target text content is extracted from the target text content corresponding to each text block. Then, the second number of target text content is pushed to the large language model as prompt text and question text to obtain the response text to the question text. This makes the output of the user's question more accurate after the large language model has filtered the text.

[0092] Specifically, referring to Figure 6, after the processing in step 301 above, a total of k text blocks are obtained. These k text blocks are arranged in descending order of similarity, and then the first q text blocks are extracted. The text content is then restored based on the metadata of these q text blocks, and finally, all the obtained text content is sent to the large language model for use.

[0093] Specifically, after retrieving data using vectors and keywords, K text blocks are obtained. These blocks are then sorted based on similarity to determine the most relevant text block. Based on the metadata of this text block, such as paragraph_id_l1, paragraph_id_l2, paragraph_len, and the filename, multiple text blocks from the window context are used to compose the corresponding text for use by the large language model.

[0094] Considering that the above data retrieval method may also be applied to purposes other than large language models, such as only for retrieving relevant documents, as a specific application of the above data retrieval method, as shown in Figure 7, the BM25 algorithm is applied to retrieve the K most relevant text blocks from the knowledge base based on the user's question. Then, based on the preset context range and the first metadata of each text block in the K text blocks, the associated text blocks of each text block are determined. Based on the first metadata and the second metadata, each text block and the associated text blocks of each text block are sorted and integrated to obtain the target text content corresponding to each text block. Based on the target text content corresponding to each text block, the answer text corresponding to the question text is determined and returned as the relevant text segment for answering the question.

[0095] In one possible implementation, the process of obtaining text blocks and metadata from the aforementioned text database may specifically include:

[0096] The text content of the first document belonging to the lowest level heading is split into at least one text block, wherein the first document contains a table of contents, and the table of contents contains at least one level of headings;

[0097] Store text block index data to generate a text database. The text block index data includes text blocks and metadata.

[0098] Specifically, for first-level documents with table of contents information, such as HTML, CHM, and Word documents, the table of contents must contain at least one level of headings. The table of contents consists of headings arranged hierarchically. Generally, a document's table of contents will consist of multiple levels of headings. Each heading has a parent heading and its subheadings. The lowest-level headings only have a parent heading and no subheadings. The lowest-level headings can be called final-level headings. The text content belonging to a final-level heading can be the text content under that final-level heading (because final-level headings no longer contain headings, only the corresponding text content).

[0099] The first step is to determine the text content of each subheading under any given title based on the directory structure.

[0100] The second step, after determining the text content, is to segment each piece of text according to the actual retrieval needs. The length of the resulting text blocks should not be too large or too small. If the text blocks are too large, similarity searches will take longer, thus slowing down the retrieval process. If the text blocks are too small, they carry less content, making it difficult to find similar text blocks. Therefore, the length of the text blocks can be set to a length that is easy to segment and yields more reasonable text block lengths, such as 512 characters, according to the actual retrieval needs.

[0101] It's important to note that the title path of a text block is the title path of the text content to which the text block belongs; that is, the path from the last-level heading of the text content to the root node's title. Specifically, text blocks and their corresponding metadata can be stored in a pre-configured database. A text block is essentially the component obtained after segmenting the text content. To facilitate the reconstruction of the text content and document content, each text block corresponds to metadata, including the title path, which is the path from the root node of the directory to the last-level heading. For example, if the last-level heading of a text block is Heading 3, Heading 1 is its root node, and Heading 2 is the direct parent node of Heading 3, then the title path of the text block is: Heading 1 / Heading 2 / Heading 3. A first order identifier needs to be added to the text content contained under Heading 3 to determine the order of the text content within the entire first document. Similarly, a second order identifier is added to each text block under each text content within that text content.

[0102] In this way, when performing text block similarity retrieval, after obtaining the target text block, the complete text content under the corresponding target final-level heading can be reconstructed based on the first and second order identifiers of the target text block. Then, based on the first order identifier and the heading path, all the text content under the parent heading of the target final-level heading can be reconstructed, and finally, based on the heading path, all the text content under the root node can be directly reconstructed. This ensures the completeness and coherence of the returned content within the document, improving the accuracy of the response.

[0103] In one possible implementation, to improve the efficiency and accuracy of text block segmentation, the above-mentioned method of splitting text content belonging to the lowest-level heading into at least one text block may specifically include the following steps:

[0104] Step 1: Mark the text content to obtain text content with markers.

[0105] Specifically, referring to Figure 4, for documents in HTML, CHM, Word, and other formats with headings, the text content under each heading is first extracted based on the directory structure. Then, each text content is converted into lightweight markup language text, such as Markdown, using a formatting conversion library, and saved in JSON format. A portion of the resulting document content is shown below:

[0106] "id":1,

[0107] "title":" / FusionOne Center Alarms / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost",

[0108] "content":"\n####\nAlarm Explanation\nAlarm Description:\n```\nThe heartbeat signal between the device and the system is abnormal.\n```\nThis alarm is generated by FusionOne Center if heartbeat detection fails during device heartbeat detection.\n####\nAlarm Attributes\n####\nAlarm Parameters\n|\nAlarm ID\n|\nAlarm Level\n|\nAutomatically Clearable\n|\n|---|---|---|\n|\n0x1000000101000000\n|\nUrgent\n|\nYes\n|\n|\nParameter Name\n|\nParameter Meaning\n|\n|---|---|\n|\n arg1\n|\nDevice IP address.\n|\n####\nImpact on the system\nThis alarm indicates that the heartbeat between the FusionOne Center service and the device has been lost, preventing normal device management and affecting all user operations on the device.\n####\nPossible causes\n*Abnormal network connection between the device and FusionOne Center.\n*Device malfunction.\n####\nTroubleshooting steps\n1. Log in to the FusionOne Center WebUI.\n2. Check the device and FusionOne... Is the network communication between centers normal? \nCheck whether the network communication is normal, for example: whether the network interface is disconnected, whether the routing configuration is normal, whether the regional network is normal, etc. \n\t*If the network communication is normal, >execute\n\t[4]\n\t. \n\t*If the network communication is abnormal, please restore the network connection and check whether the alarm has been restored. If the network cannot be restored, execute\n\t[4]\n\t. \n\t\t+If the alarm has been restored, the alarm processing ends. \n\t\t+If the alarm is still >If it exists, execute\n\t\t[4]\n\t\t. \n3. Check if the device is running normally. \n\t*If the device is not running normally, please restore the device to normal operation and check if the alarm has been cleared. \n\t\t+If the alarm has been cleared, the alarm handling is finished. \n\t\t+If the alarm still exists, execute\n\t\t[4]\n\t\t. \n\t*If the device is running normally, execute\n\t[4]\n\t. \n4. Please contact the technical support engineer for help. \n"

[0109] Where "content": is the text content, "title": is the title path of the text block, and "id": is the sequence number of the text content (i.e., the first order identifier).

[0110] The symbols ###, |---|, and n\t are unique markers in Markdown format, such as headings and table tags.

[0111] Step 2: Based on the preset segmentation length and preset text block segmentation judgment rules, segment the text content with markers to obtain one or more text blocks.

[0112] Specifically, referring to Figure 4, the preset segmentation length can be set to 512 bytes when performing text block segmentation, meaning the length of the segmented text block is no greater than 512 bytes. During segmentation, the text content can be segmented according to the markers selected from the markers each time, and it is determined whether the length of each segmented text block is no greater than 512 bytes, and whether the combined length of two adjacent text blocks is no less than 512 bytes. If the resulting text blocks meet the above segmentation judgment conditions after the text content is segmented, no further segmentation is required.

[0113] To further improve the efficiency of text block segmentation, when segmenting the content using selected markers, the markers can be selected sequentially in descending order of priority, and then the text content can be segmented based on the selected markers. The priority of the currently selected marker is lower than that of the previously selected marker.

[0114] For example, the text content following "content": can be segmented according to the priority order of '\n\n', '\n', '。', '!', ',', using the selected markers from the previous segmentation to segment the text content. The default segmentation length is 512 bytes. If the text block length is less than 512 bytes and the combined length with adjacent text blocks exceeds the set length of 512 bytes, then no further segmentation will be performed.

[0115] For example, after using '\n' to split the data, the following results are generated: ['Alarm Explanation', 'Alarm Description:', ..., 'Execute if the device is running normally', ...]. Continuous text blocks are concatenated to form a text block with a length of just less than 512. After adding the file name and the title path of the text block, the result is placed into the result set to obtain a split result set.

[0116] The contents of the four text blocks obtained from the final segmentation are shown below:

[0117] 1.'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss\n#### Alarm Explanation\nAlarm Description:\n```\nThe heartbeat signal between the device (IP:\narg1\n) and the system is abnormal.\n```\nThis alarm is generated when FusionOne Center fails to detect the device's heartbeat.\n#### Alarm Attributes\n|Alarm ID|Alarm Level|Automatically Clearable|\n|---|---|---|\n|0x1000000101000000|Urgent|Yes|\n#### Alarm Parameters'.

[0118] 2. FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss\n|Parameter Name|Parameter Meaning|\n|---|---|\n|\narg1\n|Device IP Address. |\n####Impact on the System\nThis alarm indicates that the heartbeat between the FusionOne Center service and the device has been lost, making it impossible to manage the device normally and affecting all user operations on the device.\n####Possible Causes\nAbnormal network connection between the device and FusionOne Center.\nDevice malfunction.\n####Handling Steps\n1. Log in to the FusionOne Center WebUI.

[0119] 3.'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost\n2. Check if the network communication between the device and FusionOne Center is normal. Check if the network communication is normal, for example: whether the network interface is disconnected, whether the routing configuration is normal, whether the regional network is normal, etc.\n\t\nIf the network communication is normal, execute [4].\n\t\nIf the network communication is abnormal, please restore the network connection and check if the alarm is restored. If the network cannot be restored, execute [4].\n\t\t+If the alarm has been restored, the alarm handling ends.\n\t\t+If the alarm still exists, execute [4].\n3. Check if the device is running normally.\n\t\nIf the device is not running normally, please restore the device to the normal operating state and check if the alarm is restored.',

[0120] 4.'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost\n+If the alarm has been recovered, alarm handling ends.\n\t\t+If the alarm still exists, execute [4].\n\t\nIf the device is running normally, execute [4].\n4.]Please contact a technical support engineer for assistance.

[0121] The four text blocks numbered 1 to 4 are the result of the segmentation, and “FusionOne HCI 23 23.1.1 Alarm Processing 01” is the file name.

[0122] The title path of the text block is “FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost”.

[0123] Based on the above text blocks, corresponding metadata needs to be added to each text block. After adding the metadata, the four parts stored in the database are as follows:

[0124] 1. Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss\n#### Alarm Explanation\nAlarm Description:\n```\nThe heartbeat signal between the device (IP:\narg1\n) and the system is abnormal.\n```\nFusionOne This alarm is generated when the Center fails to detect the heartbeat of the device. \n#### Alarm Attributes\n|Alarm ID|Alarm Level|Automatically Clearable|\n|---|---|---|\n|0x1000000101000000|Urgent|Yes|\n#### Alarm Parameters',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 23 23.1.1 Alarm Handling 01','title':'FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost', 'content_embeded': array([0.00509322, -0.03263689, -0.02818436, ..., -0.01265258, 0.01112661, -0.01226701], dtype = float32), 'paragraph_id_l1': 2, 'paragraph_id_l2': 0, 'paragraph_len': 4}).

[0125] 2. Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss\n|Parameter Name|Parameter Meaning|\n|---|---|\n|\narg1\n|Device IP Address.|\n####Impact on the System\nThis alarm indicates that the heartbeat between the FusionOne Center service and the device is lost, making it impossible to manage the device normally, which will affect all user operations on the device.\n####Possible Causes\nAbnormal network connection between the device and FusionOne Center.\nDevice Failure.\n####Handling Steps\n1. Log in to FusionOne Center WebUI.',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 23 23.1.1 Alarm Handling 01','title':'FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss','content_embeded':array([-0.00316077,-0.02642941,-0.02343957,...,-0.01106526,0.0077391,-0.0079075],dtype=float32),'paragraph_id_l1':2,'paragraph_id_l2':1,'paragraph_len':4}).

[0126] 3. Document(page_content = 'FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss\n2. Check Device and FusionOne Is the network communication between centers normal? Check if the network communication is normal, for example: whether the network interface is disconnected, whether the routing configuration is normal, whether the regional network is normal, etc. If the network communication is normal, execute [4]. If the network communication is abnormal, restore the network connection and check if the alarm is restored. If the network cannot be restored, execute [4]. If the alarm has been restored, the alarm processing ends. If the alarm still exists, execute [4]. 3. Check if the device is running normally. If the device is not running normally, restore the device to normal operation and check if the alarm is restored. ',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 23 23.1.1 alarm processing 01','title':'FusionOne Center alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost', 'content_embeded': array([0.00960491, -0.03210478, -0.02278469, ..., -0.01252992, -0.02566933, -0.02070089], dtype = float32), 'paragraph_id_l1': 2, 'paragraph_id_l2': 2, 'paragraph_len': 4}).

[0127] 4.Document(page_content='FusionOne HCI 23 23.1.1 Alarm Handling 01\nFusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss\n+If the alarm has been recovered, the alarm handling ends.\n\t\t+If the alarm still exists, execute [4].\n\t\nIf the device is running normally, execute [4].\n4.Please contact the technical support engineer for help.',metadata={'source':'zh-cn_topic_0000001151233327.html','file_name':'FusionOne HCI 23 23.1.1 Alarm Handling 01','title':'FusionOne Center Alarm / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Lost', 'content_embeded': array([0.00483514, -0.02553011, -0.01561882, ..., -0.01515852, -0.01288328, -0.00948918], dtype = float32), 'paragraph_id_l1': 2, 'paragraph_id_l2': 3, 'paragraph_len': 4}).

[0128] The content before "metadata" in each section is the content of the aforementioned text block, and the content after "metadata" is metadata, including:

[0129] 'source':'zh-cn_topic_0000001151233327.html' indicates the source of this HTML file;

[0130] 'file_name':'FusionOne HCI 23 23.1.1 Alarm Handling 01', represents the filename;

[0131] 'title':'FusionOne Center Alarms / FusionOne Center Alarm List / 0x1000000101000000 Device Heartbeat Loss' indicates the title path of the text block;

[0132] 'paragraph_id_l1' indicates the order in which the text content is arranged in the document, i.e., the first order identifier;

[0133] 'paragraph_id_l2' indicates the order of the text blocks within the text content, i.e., the second order identifier;

[0134] To speed up text content reconstruction, 'paragraph_len' was added, representing the number of text blocks contained within the text content. This allows for faster text content retrieval and reconstruction by obtaining the second-order identifier of the text blocks and the number of text blocks within the text content.

[0135] Referring to Figure 4, the obtained text block index data is finally stored to generate a text database. In some implementations, considering that most retrieval algorithms calculate the similarity of vectors during retrieval, the obtained text blocks can be converted into word vectors through a word embedding model and stored together with the text blocks. This allows for direct use of the corresponding word vectors during retrieval, thereby improving retrieval speed.

[0136] The content following 'content_embeded' is an array that stores word vectors, containing the word vectors corresponding to the text blocks.

[0137] It is understood that those skilled in the art can select and adjust the content of the above metadata as needed, and no restrictions are imposed here.

[0138] Referring to Figure 5, which is a flowchart illustrating a text database generation method provided in an embodiment of this application, the method can be applied to a first document with directory information, such as a document in html, chm, or word format, to generate a database. The directory of the document contains at least one level of headings.

[0139] As shown in Figure 5, the text database generation method provided in this application embodiment may include steps 501 to 502, which will be described in detail below.

[0140] 501. Split the text content belonging to the lowest level heading in the first document into at least one text block.

[0141] The first document is a document with table of contents information, such as a document in html, chm, or word format.

[0142] The table of contents in the first document only includes one level of headings.

[0143] Specifically, a table of contents consists of headings arranged hierarchically. A typical document's table of contents comprises multiple levels of headings. Each heading has a parent heading and its subheadings. The lowest level heading only has a parent heading and no subheadings. The lowest level heading is called a final-level heading. The text content belonging to a final-level heading is the text content under that final-level heading (because final-level headings do not contain further headings, only their corresponding text content).

[0144] The first step is to determine the text content of each subheading under any given title based on the directory structure.

[0145] The second step, after determining the text content, is to segment each piece of text according to the actual retrieval needs. The length of the resulting text blocks should not be too large or too small. If the text blocks are too large, similarity searches will take longer, thus slowing down the retrieval process. If the text blocks are too small, they carry less content, making it difficult to find similar text blocks. Therefore, the length of the text blocks can be set to a length that is easy to segment and yields more reasonable text block lengths, such as 512 characters, according to the actual retrieval needs.

[0146] It should be noted that those skilled in the art can adjust and select the segmentation length of the text block according to the needs of actual applications, and no restrictions are imposed here.

[0147] 302. Store text block index data, the text block index data including: the text block and metadata, the metadata including: the title path of the text block, the first order identifier of the text content of the text block in the first document, and the second order identifier of the text block.

[0148] It's important to note that the title path of a text block is the title path of the text content to which the text block belongs; that is, the path from the last-level heading of the text content to the root node's title. Specifically, text blocks and their corresponding metadata can be stored in a pre-configured database. A text block is essentially the component obtained after segmenting the text content. To facilitate the reconstruction of the text content and document content, each text block corresponds to metadata, including the title path, which is the path from the root node of the directory to the last-level heading. For example, if the last-level heading of a text block is Heading 3, Heading 1 is its root node, and Heading 2 is the direct parent node of Heading 3, then the title path of the text block is: Heading 1 / Heading 2 / Heading 3. A first order identifier needs to be added to the text content contained under Heading 3 to determine the order of the text content within the entire first document. Similarly, a second order identifier is added to each text block under each text content within that text content.

[0149] In this way, when performing text block similarity retrieval, after obtaining the target text block, the complete text content under the corresponding target final-level heading can be reconstructed based on the first and second order identifiers of the target text block. Then, based on the first order identifier and the heading path, all the text content under the parent heading of the target final-level heading can be reconstructed, and finally, based on the heading path, all the text content under the root node can be directly reconstructed. This ensures the completeness and coherence of the returned content within the document, improving the accuracy of the response.

[0150] It should be noted that other specific implementations of the above-mentioned text database generation method can be found in the relevant part of the text database generation section of the above-mentioned data retrieval method, and will not be repeated here.

[0151] The above describes the text database generation method and data retrieval method provided by the embodiments of this application. The following will describe the apparatus for performing the above text database generation method and data retrieval method.

[0152] Please refer to Figure 8, which is a schematic diagram of the structure of a text database generation device provided in an embodiment of this application. As shown in Figure 8, the text database generation device includes:

[0153] Document segmentation module 801 is used to split the text content belonging to the lowest level heading in the first document into at least one text block; and

[0154] The data storage module 802 is used to store text block index data. The text block index data includes: text blocks and metadata. The metadata includes: the title path of the text block, the first order identifier of the text content in the first document, and the second order identifier of the text block.

[0155] Figure 9 shows a data retrieval device provided in this application, comprising:

[0156] The text block retrieval module 901 is used to extract a first number of text blocks from the text database based on the first similarity between the question text and the text blocks in the text database. The text database includes multiple text blocks and metadata corresponding to each text block. The metadata includes the title path of the corresponding text block, the first order identifier of the text content in the first document, and the second order identifier of the text block in the text content. The text content is the text content in the first document that belongs to any final heading.

[0157] The associated text determination module 902 is used to determine the associated text block of each text block based on a preset context range and the first metadata of each text block in a first number of text blocks. The first metadata is the metadata corresponding to each text block in the first number of text blocks.

[0158] The text block integration module 903 is used to sort and integrate each text block and its associated text blocks based on the first metadata and the second metadata to obtain the target text content corresponding to each text block; and

[0159] The response text generation module 904 is used to determine the response text corresponding to the question text based on the target text content corresponding to each text block.

[0160] In one possible implementation, the process by which the response text generation module 904 determines the response text corresponding to the question text based on the target text content corresponding to each text block includes:

[0161] Based on the first similarity, a second number of target text contents are extracted from the target text contents corresponding to each text block;

[0162] The second set of target text content is pushed to the large language model as prompt text and question text to obtain the response text to the question text.

[0163] In one possible implementation, the process by which the associated text determination module 902 determines the associated text blocks of each text block based on a preset context range and the first metadata of each text block in a first number of text blocks includes:

[0164] Based on a preset context range, determine the maximum number of associated text blocks for each text block;

[0165] The maximum number is used as the length of the context window range for each text block to determine the associated text blocks for each text block.

[0166] In one possible implementation, it also includes: a text database generation module, used for:

[0167] The text content of the first document belonging to the lowest level heading is split into at least one text block, wherein the first document contains a table of contents, and the table of contents contains at least one level of headings;

[0168] Store text block index data to generate a text database. The text block index data includes text blocks and metadata.

[0169] In one possible implementation, the text database generation module splits the text content of the first document belonging to the lowest-level heading into at least one text block, including:

[0170] The text content is marked to obtain text content with markers;

[0171] Based on the preset segmentation length and preset text block segmentation judgment rules, the text content with markers is segmented to obtain one or more text blocks.

[0172] In one possible implementation, the text database generation module segments the text content containing markers based on a preset segmentation length and preset text block segmentation judgment rules, including:

[0173] The text content is segmented according to the markers selected from the markers in a preset order, and it is determined whether the length of each text block obtained by each segment is not greater than the preset segmentation length, and whether the length of two adjacent text blocks is not less than the preset segmentation length.

[0174] In one possible implementation, the process by which the text database generation module segments the text content according to a preset order of selected tokens includes:

[0175] The tokens are selected sequentially in descending order of priority, with the priority of the currently selected token being lower than that of the previously selected token.

[0176] The text content obtained from the previous segmentation is segmented based on the selected markers each time.

[0177] In one possible implementation, the text database generation module performs tagging processing on the text content to obtain text content with tags, including:

[0178] The text content is converted into Markdown format using a preset conversion tool.

[0179] Input the Markdown-formatted text into a JSON converter to process the markup information, and obtain the text content with markup characters.

[0180] Embodiments of this application also provide a server, including at least one processor and a memory connected to the processor, wherein:

[0181] Memory is used to store computer programs;

[0182] The processor is used to execute computer programs to enable the server to implement the text database generation method or data retrieval method as described in the above embodiments.

[0183] Embodiments of this application also provide a computer-readable storage medium carrying one or more computer programs that, when executed by an electronic device, implement the text database generation method or data retrieval method as described in the above embodiments.

[0184] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the data retrieval method or text database generation method as described in the above embodiments.

[0185] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0186] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0187] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0188] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0189] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data retrieval method, characterized by, The method comprises the following steps: extracting a first number of text blocks from a text database based on a first similarity between a question text and a text block in the text database, the text database comprising a plurality of text blocks and metadata corresponding to each of the text blocks, the metadata comprising a title path of the corresponding text block, a first order identifier of a text content in a first document in which the text block is located, and a second order identifier of the text block in the text content, the text content being in the first document and belonging to any last-level title; determining an associated text block of each of the text blocks based on a preset context range and first metadata of each of the text blocks, the first metadata being the metadata corresponding to each of the text blocks; sorting and integrating each of the text blocks and the associated text block of each of the text blocks based on the first metadata and second metadata to obtain target text content corresponding to each of the text blocks, the second metadata being the metadata of the associated text block of each of the text blocks; determining a reply text corresponding to the question text based on the target text content corresponding to each of the text blocks.

2. The data retrieval method of claim 1, wherein, The method for determining the reply text corresponding to the question text based on the target text content corresponding to each of the text blocks comprises: extracting a second number of the target text content from the target text content corresponding to each of the text blocks based on the first similarity; pushing the second number of the target text content as prompt text and question text to a large language model to obtain the reply text of the question text.

3. The data retrieval method of claim 1 or 2, wherein, The method for determining the associated text block of each of the text blocks based on the preset context range and the first metadata of each of the text blocks comprises: determining a maximum number of the associated text block of each of the text blocks based on the preset context range; determining the associated text block of each of the text blocks as a length of a context window interval of each of the text blocks.

4. The data retrieval method according to any one of claims 1-3, characterized in that, The method further comprises: splitting text content belonging to a last-level title of a first document into at least one text block, wherein the first document comprises a directory, and the directory comprises at least one level of title; storing text block index data to generate the text database, the text block index data comprising the text block and the metadata.

5. The data retrieval method of claim 4, wherein, The method for splitting the text content belonging to the last-level title of the first document into at least one text block comprises: performing mark processing on the text content to obtain text content with marks; splitting the text content with marks into one or more text blocks based on a preset splitting length and a preset text block splitting judgment rule.

6. The data retrieval method of claim 5, wherein, The method for splitting the text content with marks based on the preset splitting length and the preset text block splitting judgment rule comprises: splitting the text content according to a mark selected from the marks in a preset order, and determining whether the length of each text block obtained by each splitting is not greater than the preset splitting length, and the length of adjacent two text blocks is not less than the preset splitting length.

7. The data retrieval method of claim 6, wherein, The mark selected from the mark symbol in the preset order is used to split the text content, including: The selection of the mark symbol is performed in the order of the priority of the mark symbol from high to low, wherein the priority of the mark symbol selected in the current time is lower than the priority of the mark symbol selected in the previous time; The text content obtained by splitting in the previous time is split based on the mark symbol selected each time.

8. The data retrieval method of claim 5, wherein, The text content is marked to obtain text content with mark symbols, including: The text content is converted into markdown format text based on a preset conversion tool; The markdown format text is input into a json converter for mark information processing to obtain the text content with mark symbols.

9. A text database generation method characterized by comprising: The method is applied to a first document containing a directory, and the directory contains at least one hierarchical title, and the method includes: Text content belonging to the last title in the first document is split into at least one text block; Text block index data is stored, and the text block index data includes the text block and metadata, and the metadata includes a title path of the text block, a first order identifier of the text block in the text content in the first document, and a second order identifier of the text block.

10. A server, characterized by The server includes at least one processor and a memory connected to the processor, wherein: The memory is used to store a computer program; The processor is used to execute the computer program to enable the server to implement the data retrieval method in any one of claims 1 to 8 or the text database generation method in claim 9.

Images