Knowledge processing method and device of question and answer knowledge base system, equipment and medium
Through dynamic chunking of text similarity algorithm, the search efficiency and accuracy of the Q&A knowledge base system are improved, and the problem of redundant information and insufficient information caused by improper chunking size in traditional methods is solved.
Patent Information
- Application Number
- CN202411052140.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The traditional Q&A knowledge base system does not answer comprehensively when the knowledge is split too much. When the split is too large, it contains redundant information, which affects the search efficiency and accuracy.
The preset text similarity algorithm is used to calculate the similarity between knowledge shards and dynamically divide it, so that the shards within each shard are highly correlated and the information is as complete as possible.
It improves the efficiency, accuracy and comprehensiveness of the search, and avoids the problem of redundant information and insufficient information caused by improper block size in traditional methods.
Smart Images

Figure CN119003722A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a knowledge processing method, device, equipment and medium for a question-and-answer knowledge base system. Background Art
[0002] The question-answering knowledge base system is a natural language processing technology based on a knowledge base. It can receive users' natural language questions and search for answers that meet users' needs from the knowledge base.
[0003] Existing search solutions are usually based on knowledge sharding, block division, and vectorization. Specifically, sharding is to split the original knowledge base into several independent, shorter knowledge points. Each knowledge point will be used as the minimum record of the question and answer, and will be matched with the question when answering the question; the scope of the block is larger. For example, the regulations on non-local medical insurance include several details, such as the non-local medical insurance reimbursement process, non-local medical insurance reimbursement ratio, non-local medical insurance reimbursement scope, and materials that need to be submitted for non-local medical insurance. Assuming that these details are the smallest knowledge points, non-local medical insurance is summarized into a large block, that is, a block, and is given together when the questioner asks questions about non-local medical insurance.
[0004] In traditional question-and-answer knowledge base systems, knowledge is divided into fixed blocks. If the blocks are divided too small, the answers may not be comprehensive. If the blocks are divided too large, they may contain redundant information, affecting search efficiency and accuracy. Summary of the invention
[0005] In view of this, the present invention provides a knowledge processing method, device, equipment and medium for a question-and-answer knowledge base system, which can effectively improve the efficiency, accuracy and comprehensiveness of the search.
[0006] In a first aspect, the present invention provides a knowledge processing method for a question-answering knowledge base system, the method comprising:
[0007] Acquire all the knowledge contained in the question-answer knowledge base system, and slice the acquired knowledge to obtain multiple knowledge slices;
[0008] Using a preset text similarity algorithm to calculate the similarity between each two of the knowledge fragments, and comparing each of the calculated similarities with a preset similarity threshold;
[0009] Any similarity greater than the similarity threshold is determined as a target similarity, and the two knowledge fragments based on which the target similarity is calculated are placed in the same knowledge block to implement block processing of the knowledge.
[0010] The knowledge processing method of a question-answering knowledge base system provided by the present invention has the following advantages:
[0011] The present invention is different from the traditional question-and-answer knowledge base system that splits knowledge into fixed blocks. Instead, it uses a text similarity algorithm to calculate the similarity between different blocks, and then dynamically divides the blocks according to the similarity between different blocks, so that the blocks contained in each block have a high correlation and the information in each block is as complete as possible, effectively improving the efficiency, accuracy and comprehensiveness of the search.
[0012] In an optional implementation, before placing the two knowledge fragments based on which the target similarity is calculated into the same knowledge block, the method further includes:
[0013] Determine in sequence that each of the two knowledge slices based on which the target similarity is calculated is a current knowledge slice;
[0014] If the similarity between the current knowledge slice and only one other knowledge slice is greater than the similarity threshold, then execute the step of placing the two knowledge slices based on which the target similarity is calculated into the same knowledge block; otherwise, determine the maximum value of all similarities calculated based on the current knowledge slice, and place the current knowledge slice and the other knowledge slice based on which the maximum value is calculated into the same knowledge block.
[0015] In an optional implementation, it also includes:
[0016] In the process of placing the knowledge slices into the knowledge blocks, if the number of the knowledge slices contained in any of the knowledge blocks reaches the corresponding capacity threshold, it is determined that the block of any of the knowledge blocks has been completed, and it is prohibited to place the knowledge slices into any of the knowledge blocks.
[0017] In an optional implementation, it also includes:
[0018] In the process of placing the knowledge slices into the knowledge blocks, if the two knowledge slices based on which any similarity greater than the similarity threshold is calculated cannot be placed in the same knowledge block, the placement of the corresponding knowledge slices is achieved based on the principle of preferentially placing the two knowledge slices based on which any similarity greater than the similarity threshold is calculated in adjacent positions.
[0019] In an optional implementation, the preset text similarity algorithm is specifically a cosine similarity algorithm of TF-IDF and word frequency vectors.
[0020] In an optional implementation, after implementing the block processing of the knowledge, the method further includes:
[0021] Determine that the search hit knowledge slice is a specified knowledge slice, and based on the sliding window, slide forward and backward through a specified number of knowledge slices starting from the specified knowledge slice, and feed back the specified knowledge slice and the knowledge slice that is slid to as search results.
[0022] In an optional implementation, before the sliding window starts from the designated knowledge slice and slides forward and backward through a designated number of knowledge slices respectively, the method further includes:
[0023] Receive a window adjustment instruction, and in response to the window adjustment instruction, adjust the sliding window to a corresponding value; wherein the value of the sliding window is the number of knowledge shards that the sliding window can slide over at a single time.
[0024] In a second aspect, the present invention provides a knowledge processing device for a question-answering knowledge base system, the device comprising:
[0025] A sharding module is used to: obtain all the knowledge contained in the question-answering knowledge base system, and perform sharding processing on the obtained knowledge to obtain multiple knowledge shards;
[0026] A calculation module, used to: calculate the similarity between each two of the knowledge fragments using a preset text similarity algorithm, and compare each of the calculated similarities with a preset similarity threshold;
[0027] The block module is used to: determine any similarity greater than the similarity threshold as the target similarity, and put the two knowledge fragments based on which the target similarity is calculated into the same knowledge block to realize block processing of the knowledge.
[0028] The knowledge processing device of a question-answering knowledge base system provided by the present invention has the following advantages:
[0029] The present invention is different from the traditional question-and-answer knowledge base system that splits knowledge into fixed blocks. Instead, it uses a text similarity algorithm to calculate the similarity between different blocks, and then dynamically divides the blocks according to the similarity between different blocks, so that the blocks contained in each block have a high correlation and the information in each block is as complete as possible, effectively improving the efficiency, accuracy and comprehensiveness of the search.
[0030] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the knowledge processing method of the question-and-answer knowledge base system of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the knowledge processing method of the question-and-answer knowledge base system of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 is a flow chart of a knowledge processing method of a question-answering knowledge base system provided according to an embodiment of the present invention;
[0034] Figure 2 This is a block flow diagram of a knowledge processing method of a question-and-answer knowledge base system provided in an embodiment of the present invention;
[0035] Figure 3 is a schematic diagram of block data obtained in a knowledge processing method of a question-answering knowledge base system provided in an embodiment of the present invention;
[0036] Figure 4 It is a schematic diagram of feeding back block data after a search hit in the prior art;
[0037] Figure 5 This is a schematic diagram of sliding a sliding window in a knowledge processing method of a question-and-answer knowledge base system provided in an embodiment of the present invention;
[0038] Figure 6 It is a schematic diagram of a problem that can be solved by a sliding window solution in a knowledge processing method of a question-answering knowledge base system provided according to an embodiment of the present invention;
[0039] Figure 7 is a structural block diagram of a knowledge processing device of a question-answering knowledge base system according to an embodiment of the present invention;
[0040] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0042] The question-and-answer knowledge base system has a wide range of applications, such as conversational robots and intelligent customer service. A vivid example is deploying a medical insurance question-and-answer knowledge base in a hospital. When a patient asks how to use non-local medical insurance in Beijing, the system searches the knowledge base, finds the corresponding clauses, and gives the correct answer.
[0043] The development of the question-answering knowledge base system has gone through the following stages:
[0044] 1. Rule-based question-answering knowledge base system. This is the most traditional question-answering knowledge base system. It judges the user's questions by writing a series of rules and selects the most suitable answer from the knowledge base. The advantage of this method is that it is highly controllable, but the disadvantage is that it takes a lot of time and effort to design rules, and it cannot handle complex problems.
[0045] 2. Statistical question-answering knowledge base system. It builds a statistical model by analyzing a large number of questions and answers, and then finds the most appropriate answer in the model based on the user's question. The advantage of this method is that it can handle complex problems, but the disadvantage is that it requires a large amount of labeled data, and the accuracy will be affected by the data quality.
[0046] 3. Question-answering knowledge base system based on deep learning. It trains a large number of questions and answers through deep learning algorithms, builds a deep neural network model, and then finds the most appropriate answer in the model based on the user's question. The advantage of this method is that it can handle complex problems and has a high accuracy rate. The disadvantage is that it requires a large amount of labeled data and computing resources.
[0047] 4. Question-answering knowledge base system based on large models. It uses large models to train questions and answers in the knowledge base, taking advantage of the excellent understanding and interpretability of large models to achieve more efficient and accurate question-answering functions. The disadvantage is that the time and computing resource costs of training large models are extremely high. At the same time, compared with the conservative traditional methods, the free play of large models often gives unexpected or difficult-to-control answers.
[0048] In addition to their own advantages and disadvantages, the above solutions also have a common problem. After a knowledge base is built, if you want to build another knowledge base, except for some deep learning network structures that can be reused, the rules, statistical models, deep learning models and large models all need to be retrained, which requires the corresponding personnel to have sufficient professional knowledge and a lot of annotation and design work. For example, if you build a medical insurance question and answer knowledge base and then build a stock question and answer knowledge base, the same work will almost have to be repeated.
[0049] In order to meet the needs of quickly building question-answering knowledge bases in different knowledge fields, a popular approach now is to convert the original knowledge text of the knowledge base into word vectors through word embedding to form a dictionary for storage. When asking questions, the questions are converted into word vectors in the same way, and then mature search algorithms are used to search for relevant knowledge points from the knowledge base dictionary to splice them into answers.
[0050] There is a problem here. Since the source of knowledge is controllable and processed, the content of the knowledge base is relatively standardized, but the questions asked by the questioner are relatively random, and the degree of match with the content in the knowledge base may not be ideal. For example: in the medical insurance question and answer knowledge base, there is content about medical insurance in other places and indexed as "medical insurance in other places"; if the patient asks "How to use medical insurance in other places?", the content can be accurately searched and the answer can be given, but if the patient asks "I am from Shanxi, how can I get reimbursed?", it is difficult to accurately identify the question and give an answer. Therefore, in traditional solutions, users are usually required to standardize questions (for example, only certain questions can be asked, or only questions in a fixed format can be asked), or the algorithm side continuously adds rules or trains models to adapt to various "unexpected" questions to identify Shanxi as a different place (relative to Beijing) and reimbursement as medical insurance.
[0051] After the emergence of the big model, a new solution was developed by using the big model's excellent understanding ability, that is, the random and complex questions raised by the user were processed into simple and standardized questions or designated questions corresponding to the question library through the big model. At the same time, in the answer generation stage, the big model also provides a new solution. After obtaining the content in the knowledge base, the big model's generation ability is used to generate answers that are easier to understand and smooth. The combination of the big model and the question-answering knowledge base is not explained here.
[0052] Regardless of which technology (rules, statistics, deep learning, large models) is combined with the search solution, it is based on knowledge sharding, chunking, and vectorized content.
[0053] Generate a tokenizer to divide the text into tokens, ensuring that each token has relatively complete and independent semantics for subsequent tasks such as embedding. There are multiple encoding methods to choose from, such as: r50k_base, p50k_base, cl100k_base, etc. If it is a model for OpenAI, the cl100k_base encoding method is usually used.
[0054] Sharding is to split the original knowledge base into several independent and shorter knowledge points. Each knowledge point will be used as the minimum record of the question and answer, and will be matched with the question when answering the question. The scope of the block is larger. For example, the regulations of non-local medical insurance include several details, such as the non-local medical insurance reimbursement process, non-local medical insurance reimbursement ratio, non-local medical insurance reimbursement scope, and materials required to be submitted for non-local medical insurance. Assuming that these details are the smallest knowledge points, then non-local medical insurance is summarized into a large block, that is, a block, and when the questioner asks questions about non-local medical insurance, they are given together. If the correlation between the knowledge points is relatively small, the step of block division will not be performed, and it can also be considered that one shard = one block. Most knowledge base systems have a sharding and block division process. A block consists of a specified number of shards. A principle that is often upheld is that it is better to give more answers than fewer.
[0055] In traditional question-and-answer knowledge base systems, knowledge is divided into fixed blocks. If the blocks are divided too small, the answers may not be comprehensive. If the blocks are divided too large, they may contain redundant information, affecting search efficiency and accuracy.
[0056] In this embodiment, a knowledge processing method of a question-answering knowledge base system is provided, which can be used in terminal devices such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a knowledge processing method of a question-answering knowledge base system according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0057] S11: Acquire all the knowledge contained in the question-answer knowledge base system, and segment the acquired knowledge to obtain multiple knowledge segments.
[0058] Among them, all the knowledge contained in the question-answer knowledge base system is the original knowledge base, which is usually text knowledge; these knowledge are segmented to prepare for subsequent segmentation, search, etc. Among them, when segmenting, the embodiment of the present invention can try to divide the segments into smaller ones to ensure the accuracy of the search.
[0059] S12: using a preset text similarity algorithm to calculate the similarity between every two of the knowledge fragments, and comparing each of the calculated similarities with a preset similarity threshold.
[0060] Among them, the similarity threshold can be pre-set according to actual needs. If the similarity between two knowledge fragments is greater than the similarity threshold, it is considered that the two knowledge fragments are highly correlated and are likely to be answers to the same question. Otherwise, it is considered that the two knowledge fragments are less correlated and are likely to be answers to different questions.
[0061] In addition, the text similarity algorithm can be selected according to actual needs, such as the deep learning algorithm BERT (Bidirectional Encoder Representations from Transformers), or TF-IDF (TermFrequency-Inverse DocumentFrequency) + cosine similarity of frequency vectors.
[0062] S13: Determine any similarity greater than the similarity threshold as the target similarity, and put the two knowledge fragments based on which the target similarity is calculated into the same knowledge block to implement block processing of the knowledge.
[0063] The prior art divides the data into blocks according to a set number after segmentation, for example, 3 segments are divided into 1 block; generally, the smaller the block, the higher the score given by the search algorithm during the search, and the more accurate the search; the larger the block, the more complete the answer. Under the set number of blocks, the smaller the segment, the smaller the block, and the larger the segment, the larger the block. To solve this problem, the embodiment of the present invention changes the fixed segmentation of the prior art into dynamic segmentation, specifically, after using a preset text similarity algorithm to calculate the similarity between two knowledge segments, if the obtained similarity is greater than a set threshold, the two knowledge segments are placed in the same knowledge block, otherwise, they are placed in different blocks, so that the knowledge segments within the same knowledge block are more correlated, such as Figure 2 shown.
[0064] The present invention is different from the traditional question-and-answer knowledge base system that splits knowledge into fixed blocks. Instead, it uses a text similarity algorithm to calculate the similarity between different blocks, and then dynamically divides the blocks according to the similarity between different blocks, so that the blocks contained in each block have a high correlation and the information in each block is as complete as possible, effectively improving the efficiency, accuracy and comprehensiveness of the search.
[0065] The knowledge processing method of the question-answering knowledge base system provided in this embodiment may further include, before placing the two knowledge fragments based on which the target similarity is calculated into the same knowledge block:
[0066] Determine in sequence that each of the two knowledge slices based on which the target similarity is calculated is a current knowledge slice;
[0067] If the similarity between the current knowledge slice and only one other knowledge slice is greater than the similarity threshold, then execute the step of placing the two knowledge slices based on which the target similarity is calculated into the same knowledge block; otherwise, determine the maximum value of all similarities calculated based on the current knowledge slice, and place the current knowledge slice and the other knowledge slice based on which the maximum value is calculated into the same knowledge block.
[0068] If the similarity between any knowledge slice and multiple other knowledge slices is greater than the similarity threshold, then in order to improve the comprehensiveness of the search, any knowledge slice can be copied into multiple copies, and each copy can be placed in the same knowledge block with each knowledge slice in multiple other knowledge slices corresponding to the similarity greater than the similarity threshold; for example: if the similarity between knowledge slice a and knowledge slices b, c, and d is greater than the similarity threshold, then a can be copied into three copies, and then a and b are placed in the same knowledge block, a and c are placed in the same knowledge block, and a and d are placed in the same knowledge block.
[0069] However, in order to improve the search accuracy, when the similarity between any knowledge slice and multiple other knowledge slices is greater than the similarity threshold, only the knowledge slice and the other knowledge slices corresponding to the maximum similarity are placed in the same knowledge block; for example: the similarity between knowledge slice a and knowledge slices b, c, and d is greater than the similarity threshold respectively, and the similarity between knowledge slice a and b is the greatest, then only a and b are placed in the same knowledge block.
[0070] Of course, other configurations may be performed according to actual needs, all of which are within the protection scope of the present invention and will not be described in detail here.
[0071] The knowledge processing method of a question-answering knowledge base system provided in this embodiment may further include:
[0072] In the process of placing the knowledge slices into the knowledge blocks, if the number of the knowledge slices contained in any of the knowledge blocks reaches the corresponding capacity threshold, it is determined that the block of any of the knowledge blocks has been completed, and it is prohibited to place the knowledge slices into any of the knowledge blocks.
[0073] To avoid too many shards in the same block, the embodiment of the present invention can set the maximum number of shards that a single knowledge block can contain, that is, the capacity threshold. Specifically, in order to improve the comprehensiveness of the search, the embodiment of the present invention can appropriately set the number of shards in each block to be more, for example, each block can contain up to 5 shards. After using the text similarity algorithm to calculate the similarity between shards, if the similarity is greater than the set threshold, the two shards are placed in the same block. When the number of shards in a block reaches the set number of shards per block (capacity threshold), it is prohibited to add more shards to the block, so that the corresponding shards are placed in one block according to the set number of shards.
[0074] The knowledge processing method of a question-answering knowledge base system provided in this embodiment may further include:
[0075] In the process of placing the knowledge slices into the knowledge blocks, if the two knowledge slices based on which any similarity greater than the similarity threshold is calculated cannot be placed in the same knowledge block, the placement of the corresponding knowledge slices is achieved based on the principle of preferentially placing the two knowledge slices based on which any similarity greater than the similarity threshold is calculated in adjacent positions.
[0076] When, due to factors such as the setting of the capacity threshold of the block, the two knowledge slices based on any similarity greater than the similarity threshold cannot be placed in the same knowledge block, the two knowledge slices can be placed in adjacent positions as much as possible, so that the adjacent knowledge slices have a higher correlation, and the comprehensiveness of the search can be further improved when supplementing the search.
[0077] After the process of the above embodiment is completed, the knowledge is divided into a series of fragments, and each fragment is attached with the index of the block in which it is located. Then the word embedding and storage are completed. Figure 3 shown.
[0078] The knowledge processing method of a question-answering knowledge base system provided in this embodiment, after implementing the block processing of the knowledge, may further include:
[0079] Determine that the search hit knowledge slice is a specified knowledge slice, and based on the sliding window, slide forward and backward through a specified number of knowledge slices starting from the specified knowledge slice, and feed back the specified knowledge slice and the knowledge slice that is slid to as search results.
[0080] Before the sliding window starts to slide forward and backward through a specified number of knowledge slices based on the specified knowledge slice, the method may further include:
[0081] Receive a window adjustment instruction, and in response to the window adjustment instruction, adjust the sliding window to a corresponding value; wherein the value of the sliding window is the number of knowledge shards that the sliding window can slide over at a single time.
[0082] After the knowledge is stored in the database, users of the question-answering knowledge base system ask questions. The system converts the questions into word vectors in the same way, and then uses mature search algorithms to search for relevant knowledge points from the knowledge base dictionary, splice them into answers, and then feedback. Figure 4 As shown, the search algorithm hits s0, s1, and s10, and then returns the chunks where these shards are located, namely c0 and c2, and splices them into the answer.
[0083] After the embodiment of the present invention adds a dynamic block solution, even if the search method of the prior art is used, the accuracy and comprehensiveness of the answer are relatively improved. And on the basis of the above-mentioned dynamic block solution, a sliding window adjustment solution is added; the implementation method is to set a sliding window, the window slides over a specified number of slices, and the window size is usually set to be the same as the size of the block (capacity threshold). For example Figure 5 As shown in the figure, assuming that the slice sn is hit by the search algorithm, the size of the sliding window is set to 5, sliding from {Sn-4,...,Sn} to {Sn,...,Sn+4}. The sliding window adjustment solution can solve the following problems: Figure 6 The questions shown further improve the completeness of the answers.
[0084] In actual operation, some parameters can be determined through experiments, such as the threshold setting of the dynamic block scheme, the size of the sliding window, etc., to obtain the best effect. In addition, users can also adjust the size of the sliding window through the window adjustment command to make it more in line with current needs.
[0085] The present invention solves the problem of poor accuracy and comprehensiveness of the existing question-and-answer knowledge base system when processing user questions through a dynamic block scheme and a sliding window adjustment scheme. The dynamic block scheme enables the system to dynamically block knowledge according to text similarity, thereby improving the accuracy of the search; the sliding window adjustment scheme further mines the information around the hit knowledge point, thereby improving the comprehensiveness of the answer. Specifically, in the traditional knowledge base system, knowledge is divided into fixed small blocks. If the block is too small, the answer may not be comprehensive. If the block is too large, it may contain redundant information, affecting the search efficiency and accuracy; the dynamic block scheme uses a text similarity calculation algorithm to dynamically block according to the similarity between different fragments, so that each block has a high correlation and the information is as complete as possible, thereby improving the accuracy and comprehensiveness of the search. After searching and matching, traditional knowledge base systems directly connect all relevant knowledge points to form answers, but this may ignore some relevant knowledge segments that are close to the target segment but not searched; the sliding window adjustment solution uses a fixed window to slide around the searched segment to bring in those ignored but actually important content, which ensures the quality of the answer while improving the comprehensiveness of the answer. The design of these two solutions is more in line with actual application scenarios and search needs, so this new technical solution can effectively solve existing problems.
[0086] In this embodiment, a knowledge processing device for a question-and-answer knowledge base system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0087] This embodiment provides a knowledge processing device for a question-answering knowledge base system, such as Figure 7 may include:
[0088] The sharding module 11 is used to obtain all the knowledge contained in the question-answering knowledge base system, and to perform sharding processing on the obtained knowledge to obtain a plurality of knowledge shards;
[0089] The calculation module 12 is used to: calculate the similarity between each two of the knowledge fragments using a preset text similarity algorithm, and compare each of the calculated similarities with a preset similarity threshold;
[0090] The block module 13 is used to: determine any similarity greater than the similarity threshold as the target similarity, and put the two knowledge fragments based on which the target similarity is calculated into the same knowledge block to realize block processing of the knowledge.
[0091] The knowledge processing device of the question-and-answer knowledge base system in this embodiment is presented in the form of a functional module, where the module refers to an application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0092] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0093] The embodiment of the present invention also provides a computer device having the above Figure 7 The knowledge processing device of the question-answering knowledge base system shown.
[0094] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade-sliced servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.
[0095] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware core slice. The hardware core slice may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.
[0096] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0097] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0098] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0099] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.
[0100] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0101] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0102] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A knowledge processing method for a question-answering knowledge base system, characterized in that: include: Acquire all the knowledge contained in the question-answer knowledge base system, and slice the acquired knowledge to obtain multiple knowledge slices; Using a preset text similarity algorithm to calculate the similarity between each two of the knowledge fragments, and comparing each of the calculated similarities with a preset similarity threshold; Any similarity greater than the similarity threshold is determined as a target similarity, and the two knowledge fragments based on which the target similarity is calculated are placed in the same knowledge block to implement block processing of the knowledge.
2. The method according to claim 1, characterized in that: Before placing the two knowledge fragments based on which the target similarity is calculated into the same knowledge block, the method further includes: Determine in sequence that each of the two knowledge slices based on which the target similarity is calculated is a current knowledge slice; If the similarity between the current knowledge slice and only one other knowledge slice is greater than the similarity threshold, then execute the step of placing the two knowledge slices based on which the target similarity is calculated into the same knowledge block; otherwise, determine the maximum value of all similarities calculated based on the current knowledge slice, and place the current knowledge slice and the other knowledge slice based on which the maximum value is calculated into the same knowledge block.
3. The method according to claim 2, characterized in that Also includes: In the process of placing the knowledge slices into the knowledge blocks, if the number of the knowledge slices contained in any of the knowledge blocks reaches the corresponding capacity threshold, it is determined that the block of any of the knowledge blocks has been completed, and it is prohibited to place the knowledge slices into any of the knowledge blocks.
4. The method according to claim 3, characterized in that Also includes: In the process of placing the knowledge slices into the knowledge blocks, if the two knowledge slices based on which any similarity greater than the similarity threshold is calculated cannot be placed in the same knowledge block, the placement of the corresponding knowledge slices is achieved based on the principle of preferentially placing the two knowledge slices based on which any similarity greater than the similarity threshold is calculated in adjacent positions.
5. The method according to claim 4, characterized in that The preset text similarity algorithm is specifically a cosine similarity algorithm of TF-IDF and word frequency vectors.
6. The method according to any one of claims 1 to 5, characterized in that After the block processing of the knowledge is realized, it also includes: Determine that the search hit knowledge slice is a specified knowledge slice, and based on the sliding window, slide forward and backward through a specified number of knowledge slices starting from the specified knowledge slice, and feed back the specified knowledge slice and the knowledge slice that is slid to as search results.
7. The method according to claim 6, characterized in that Before sliding forward and backward through a specified number of knowledge slices based on the sliding window starting from the specified knowledge slice, the method further includes: Receive a window adjustment instruction, and in response to the window adjustment instruction, adjust the sliding window to a corresponding value; wherein the value of the sliding window is the number of knowledge shards that the sliding window can slide over at a single time.
8. A knowledge processing device for a question-answering knowledge base system, characterized in that: include: A sharding module is used to: obtain all the knowledge contained in the question-answering knowledge base system, and perform sharding processing on the obtained knowledge to obtain multiple knowledge shards; A calculation module, used to: calculate the similarity between each two of the knowledge fragments using a preset text similarity algorithm, and compare each of the calculated similarities with a preset similarity threshold; The block module is used to: determine any similarity greater than the similarity threshold as the target similarity, and put the two knowledge fragments based on which the target similarity is calculated into the same knowledge block to realize block processing of the knowledge.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the knowledge processing method of the question-and-answer knowledge base system according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the knowledge processing method of the question-answering knowledge base system according to any one of claims 1 to 7.
Citation Information
Patent Citations
Hot knowledge generation method and device
CN111966733A
E-commerce customer service question answering method and device, equipment and medium
CN117131180A
Information retrieval method and device based on knowledge text library
CN117688131A
Knowledge question and answer processing method and device, electronic equipment and storage medium
CN118132715A
Question and answer method, device and equipment and computer readable storage medium
CN118394892A
