Multi-source and multi-mode fused knowledge reasoning method, system and device and medium

By adopting the fusion knowledge inference method of multi-source and multi-modal data in the field of legal services, using RAG search tool services and knowledge graph search library, the accuracy of multi-source and multi-modal data processing is solved, and the multi-source and multi-modal data processing is achieved in the field of legal services. Accurate retrieval and knowledge understanding of source data knowledge base.

CN119005340APending Publication Date: 2024-11-22SHANGHAI ZHIHE NETWORK TECH CO LTD +1
View PDF 0 Cites 32 Cited by

Patent Information

Application Number
CN202411224278.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-08-01
Filing Date
2024-09-03
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

It is difficult for the existing technology to effectively integrate and process diversified multi-source and multi-modal data in the field of legal services, resulting in inaccurate knowledge inference, inaccurate data retrieval and inaccurate semantic search.

Method used

A knowledge reasoning method that integrates multi-source and multi-modal data sets is proposed. By obtaining multi-modal data sets, it encapsulates and forms RAG retrieval tool services, uses large language models for intent recognition, problem rewriting, sub-problems and keyword extraction, and uses the It matches it to the corresponding RAG search tool service and knowledge graph search library, performs search calculation, de-overlapping and sorting, and generates knowledge inference results.

Benefits of technology

It realizes accurate retrieval and knowledge understanding of multi-source data in the legal field, improves the accuracy and interpretability of knowledge inference, and supports the effective retrieval of multi-source data knowledge base in the legal field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005340A_ABST
    Figure CN119005340A_ABST
Patent Text Reader

Abstract

The invention discloses a knowledge reasoning method, system and device fusing multiple sources and multiple modes and a medium. The method comprises the steps that obtained multi-mode data are packaged to form RAG retrieval tool services; performing intention recognition, question rewriting and sub-question and keyword extraction on a user query question to form a sub-question list and a keyword list; matching the sub-question list with an RAG retrieval tool service, determining the RAG retrieval tool service, and determining a knowledge graph retrieval library according to the keyword list; inputting the sub-questions into a matched RAG retrieval tool service for retrieval, and inputting the keywords into a knowledge graph retrieval library for retrieval; and the retrieval results are subjected to deduplication, combination and sorting by using the large language model, and the response result of the large language model is used as the knowledge reasoning result. The method can support a multi-source and multi-modal data scene applied in the legal field, realizes matching reasoning of knowledge base tool services and user query sub-problems and keywords, and effectively supports retrieval enhancement of a multi-source data knowledge base in the legal field.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a knowledge reasoning method, system, device and medium integrating multi-source and multi-modality. Background Art

[0002] Using conversational capabilities based on large language models (LLM) and retrieval-augmented generation (RAG) technology to provide knowledge question and answer, document writing, and legal case studies in legal service scenarios has become a hot spot for the application of artificial intelligence technology in the field of legal services. However, from a data perspective, the field of legal services usually involves a variety of different types and structures of data, such as text data such as legal clauses, judgments, and contracts, as well as legal audio and video records and legal judgment cases, where legal text data can be structured or semi-structured data. How to effectively integrate and process these diverse multi-source and multi-modal data and effectively support knowledge understanding and reasoning is a key issue in the application of artificial intelligence technology in the legal field.

[0003] At present, large language models will produce inaccurate information or hallucinations, slow knowledge updates, and poor interpretability of answers. The retrieval process of RAG usually sorts the embedding vectors of text blocks by vector similarity, relying on local retrieval of text blocks and lacking understanding of the overall content of the text. For legal knowledge reasoning with multi-source and multi-modal data such as legal texts, legal knowledge graphs, and legal audio and video records, traditional RAG retrieval and knowledge graph technologies have limitations to varying degrees, with problems such as inaccurate data retrieval, inaccurate semantic search, and low data quality.

[0004] Based on the above technical problems, the applicant proposed the technical solution of this application. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a knowledge reasoning method and system integrating multi-source and multi-modal, which can support multi-source and multi-modal data scenarios applied in the legal field, realize matching reasoning between knowledge base tool services and user query sub-questions and keywords, and effectively support retrieval enhancement of multi-source data knowledge base in the legal field.

[0006] In order to achieve the above object, the present invention discloses a knowledge reasoning method integrating multi-source and multi-modality, comprising the following steps:

[0007] Acquire a multimodal data set, encapsulate each multimodal data in the multimodal data set to form a RAG retrieval tool service, and generate a tool service list, wherein the tool service list includes a service name and a call address of at least one of the RAG retrieval tool services, and the tool service list also includes at least one preset knowledge graph retrieval library;

[0008] Based on the classification model, the obtained user query question is subjected to intent recognition to obtain an intent recognition result; based on the prompt word template and the large language model, the intent recognition result is subjected to question rewriting to obtain a question rewriting result; sub-questions and keywords are extracted from the question rewriting result to obtain multiple sub-questions and keywords corresponding to each of the sub-questions, and a sub-question list and a keyword list are formed;

[0009] Match the sub-question list with the RAG search tool service in the tool service list to determine the RAG search tool service used for the search, and determine the knowledge graph search library used for the search based on the keyword list;

[0010] Input each sub-question in the sub-question list into a matching RAG search tool service to perform RAG search calculation, use the first text blocks with the highest similarity in the search calculation results as first text context list data, input each keyword in the keyword list into a knowledge graph search library for search, and obtain the second text blocks with the highest similarity as second text context list data;

[0011] The first text context list data and the second text context list data are de-duplicated, merged and sorted by using a large language model, and a response result of the large language model is determined as a knowledge reasoning result.

[0012] Preferably, the step of respectively encapsulating each multimodal data in the multimodal data set to form a RAG retrieval tool service comprises the following steps:

[0013] Performing entity recognition, attribute extraction and content description on each multimodal data in the multimodal data set to obtain an entity recognition information set, an attribute set and a content description set corresponding to each multimodal data;

[0014] Generate a first prompt word based on a first prompt word template, input the first prompt word into a first large language model, generate a first output result answering the first prompt word, use the first output result as summary extraction information of the entity recognition information set, the attribute set and the content description set, and generate a text summary information knowledge base corresponding to each multimodal data according to the summary extraction information, wherein the first prompt word template includes a prompt instruction description and a matching pair list of entity recognition information sets, attribute sets and content description sets corresponding to multiple modalities;

[0015] The tool chain of text segmentation, text embedding vector and text vector library is used to encapsulate the text summary information knowledge base corresponding to each multimodal data, forming a RAG retrieval tool service corresponding to each multimodal data.

[0016] Preferably, the question rewriting is performed on the intention recognition result to obtain the question rewriting result, specifically:

[0017] Generate a second prompt word based on a second prompt word template, input the second prompt word into a second largest language model, generate a second output result answering the second prompt word, rewrite the second output result as the question of the intention recognition result, and obtain a question rewriting result, wherein the second prompt word template includes a prompt instruction description and a question rewriting rule.

[0018] Preferably, the extracting sub-questions and keywords from the question rewriting result is specifically:

[0019] A third prompt word is generated based on a third prompt word template, the third prompt word is input into a third language model, a third output result answering the third prompt word is generated, the third output result is used as a sub-question extraction and a keyword extraction of the question rewriting result, and a sub-question list and a keyword list are obtained, wherein the third prompt word template includes a prompt instruction description and sub-question and keyword extraction rules.

[0020] Preferably, the matching of the sub-question list with the RAG search tool service in the tool service list is specifically as follows:

[0021] The tool service list is added to the fourth prompt word template to generate a fourth prompt word, the fourth prompt word is input into the fourth language model, and a fourth output result answering the fourth prompt word is generated, wherein the fourth output result indicates that the fourth language model matches and registers at least one RAG retrieval tool service in the tool service list.

[0022] Preferably, the method further comprises:

[0023] The knowledge reasoning results are evaluated and assessed using the subsequent dialogue feedback of the sensitive word filtering and user query process, and the knowledge reasoning results are corrected according to the evaluation results.

[0024] The present invention also discloses a knowledge reasoning system integrating multiple sources and multiple modalities, comprising the following modules:

[0025] A packaging module, used to obtain a multimodal data set, respectively package each multimodal data in the multimodal data set to form a RAG retrieval tool service, and generate a tool service list, wherein the tool service list includes a service name and a call address of at least one of the RAG retrieval tool services, and the tool service list also includes at least one preset knowledge graph retrieval library;

[0026] A rewriting module is used to perform intent recognition on the acquired user query question based on the classification model to obtain an intent recognition result, rewrite the question on the intent recognition result based on the prompt word template and the large language model to obtain a question rewriting result, extract sub-questions and keywords from the question rewriting result to obtain multiple sub-questions and keywords corresponding to each of the sub-questions, and form a sub-question list and a keyword list;

[0027] A matching module, used for matching the sub-question list with the RAG search tool service in the tool service list, determining the RAG search tool service to be used for the search, and determining the knowledge graph search library to be used for the search according to the keyword list;

[0028] A retrieval module, used for inputting each sub-question in the sub-question list into a matching RAG retrieval tool service to perform RAG retrieval calculation, taking a plurality of first text blocks with the highest similarity in the retrieval calculation results as first text context list data, and inputting each keyword in the keyword list into a knowledge graph retrieval library for retrieval, to obtain a plurality of second text blocks with the highest similarity as second text context list data;

[0029] A merging module is used to use a large language model to remove duplications, merge and sort the first text context list data and the second text context list data, and determine the response result of the large language model as the knowledge reasoning result.

[0030] Preferably, the encapsulation module is also used for:

[0031] Performing entity recognition, attribute extraction and content description on each multimodal data in the multimodal data set to obtain an entity recognition information set, an attribute set and a content description set corresponding to each multimodal data;

[0032] Generate a first prompt word based on a first prompt word template, input the first prompt word into a first large language model, generate a first output result answering the first prompt word, use the first output result as summary extraction information of the entity recognition information set, the attribute set and the content description set, and generate a text summary information knowledge base corresponding to each multimodal data according to the summary extraction information, wherein the first prompt word template includes a prompt instruction description and a matching pair list of entity recognition information sets, attribute sets and content descriptions corresponding to multiple modalities;

[0033] By using a tool chain of text segmentation, text embedding vectors and text vector library, the text summary information knowledge base corresponding to each multimodal data is encapsulated to form a RAG retrieval tool service corresponding to each multimodal data.

[0034] The present invention also discloses a knowledge reasoning device that integrates multiple sources and multiple modalities. The knowledge reasoning device includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the computer program is executed by the processor, the steps of the knowledge reasoning method that integrates multiple sources and multiple modalities as described above are implemented.

[0035] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned knowledge reasoning method integrating multi-source and multi-modality are implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The multi-source and multi-modal knowledge reasoning method provided by the present invention converts the original multi-modal data into a text-modal knowledge base for processing by performing entity recognition, attribute extraction and content description on the multi-modal data, thereby unifying the diverse data processing in the legal field.

[0038] 2. The multi-source and multi-modal knowledge reasoning method provided by the present invention realizes the matching reasoning between the knowledge base tool service and the user query sub-questions and keywords through a large model based on a configurable prompt template, and effectively supports the retrieval enhancement of the multi-source data knowledge base in the legal field.

[0039] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the knowledge reasoning method integrating multiple sources and multiple modalities of the present invention.

[0041] Figure 2It is a flowchart of sub-steps in the knowledge reasoning method integrating multi-source and multi-modality of the present invention. DETAILED DESCRIPTION

[0042] In order to make the technical means, creative features, objectives and effects of the invention easier to understand, the invention is further described below with reference to specific diagrams. However, the invention is not limited to the following implementation cases.

[0043] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0044] "Large Language Model" refers to Large Language Model (LLM), which is a language model with a large parameter scale. It is designed to be trained with a large amount of text data to achieve the purpose of understanding and generating human language. It can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc. When using LLM to complete certain specific tasks, prompt words are needed to guide LLM to complete the tasks expected by the user.

[0045] like Figure 1 As shown, the first embodiment of the present invention provides a knowledge reasoning method integrating multiple sources and multiple modalities, comprising the following steps:

[0046] Step S1: Obtain a multimodal data set, encapsulate each multimodal data in the multimodal data set to form a RAG retrieval tool service, and generate a tool service list, wherein the tool service list includes a service name and a calling address of at least one of the RAG retrieval tool services, and the tool service list also includes at least one preset knowledge graph retrieval library.

[0047] Multimodal data sets in legal applications include image data, audio data, video data, etc. These image data, audio data, and video data are encapsulated separately. For example, entity recognition and attribute extraction are performed through computer vision and video analysis technology, and the image, audio and video content is described in natural language. The relevant entities, attributes, and natural language are summarized using a large language model, thereby obtaining a text description of the semantic content of the images, audio and video of the legal data.

[0048] like Figure 2 As shown, step S1 includes the following sub-steps:

[0049] Step S11: performing entity recognition, attribute extraction and content description on each multimodal data in the multimodal data set to obtain an entity recognition information set, an attribute set and a content description set corresponding to each multimodal data.

[0050] Image classifiers, such as different types of label classifiers, including people, scenes, landmarks, etc., are used to perform entity recognition and attribute extraction on the original image. Image classifier is an artificial intelligence algorithm that can recognize and classify objects or scenes in images. The implementation of image classifiers can use convolutional models and transformer models. Before the model processing of the image classifier, the image is usually preprocessed, including standardization and normalization of the image. Standardization, for example, scales the image resolution to a given resolution size, such as the width x height of the image is 224x 224 or 336x336, etc. Normalization, for example, performs normalization calculations on the pixel value range [0,255] of the original image to convert it into a floating point value range [0,1]. For three-channel color images, the pixel values ​​of each channel are normalized separately.

[0051] The image classifier uses the result of standardization and normalization of the original image as the model input. The image classifier is the backbone model of the entity recognition and attribute extraction model. Image entity recognition includes but is not limited to identifying objects in the image, such as people, animals, vehicles, etc., and scene recognition, such as indoors, outdoors, beaches, cities, etc. The goal of attribute extraction is to identify the attributes or features of objects in the image, such as color recognition, shape recognition, texture recognition, and sentiment analysis.

[0052] The original image is divided into blocks to form a sequence of image blocks, and the sequence-to-sequence model is used to describe the content of the image block sequence. The image is described and the sequence-to-sequence (Seq2Seq) model is used for reasoning. The sequence-to-sequence model is a type of deep learning model used to process sequence data, such as the Seq2Seq model (Sequence-to-Sequence). The sequence-to-sequence model receives an input sequence and maps the input sequence to a fixed-length vector representation through an encoder. The vector representation contains the semantic information of the input sequence, and then the fixed-length vector representation is remapped back to the output sequence through a decoder. The encoder uses a recurrent neural network RNN.

[0053] The input of the Seq2Seq model of an image is a sequence of image blocks. The image block division is the process of dividing the original image into several small blocks. The specific implementation can be to divide the image evenly into multiple small blocks of the same size in a uniform way. For example, a 256x 256 pixel image is divided into 16 8x8 pixel blocks, and the multiple small blocks of the image are flattened into a sequence of image blocks in a row-first manner. The image sequence-to-sequence model (Seq2Seq) can be implemented using a model structure that combines the encoder and decoder of the transformer, or using only the decoder as the main body of the model.

[0054] For audio data, the specific method is: extract Mel Frequency Cepstral Coefficient (MFCC) features of the audio, preprocess the original audio and generate an MFCC feature sequence.

[0055] The preprocessing of the original audio is to pre-emphasize, frame and window the speech signal to improve the signal quality of the original audio. Mel-frequency cepstral coefficient (MFCC) is a feature extraction technology used for sound signal processing. Based on the auditory characteristics of the human ear, the spectrum of the sound signal is transformed through nonlinear Mel scale to extract coefficients that can reflect the characteristics of the sound. First, the sound signal is converted from the time domain to the frequency domain through Fourier transform, and the spectrum is mapped to the Mel scale through Mel scale mapping. The triangular window function is used for division, and then the spectrum of each Mel scale is logarithmically transformed, and then the spectrum of each Mel scale is logarithmically transformed, and then the transformed spectrum is discrete cosine transformed to obtain the MFCC coefficient. After the audio is framed and windowed, the MFCC coefficient is extracted as the feature of the audio frame segment. The MFCC feature sequence is also the sequence of MFCC coefficients formed by the MFCC coefficients of each frame segment of the audio in time order.

[0056] The MFCC feature sequence of the audio is used as input, and the classifier model of the audio sequence is used to perform entity recognition and attribute extraction on the MFCC sequence of the audio. The classifier model of the audio sequence is used as the backbone model for audio entity recognition and attribute extraction, where the classifier model can adopt a convolutional model or a transformer model. Audio entity recognition refers to identifying specific entities or objects from audio data. These entities can be names, places, organizations, and events. Audio attribute extraction refers to extracting descriptive features or attributes from audio data. These attributes can be physical attributes or content attributes of the audio, such as identifying emotional attributes, speech speed, pitch, volume, rhythm, and language attributes expressed in the audio.

[0057] The MFCC feature sequence of the audio is used as input, and the sequence-to-sequence model is used to describe the content of the MFCC sequence. The audio sequence-to-sequence model (Seq2Seq) can be implemented using a model structure that combines the encoder and decoder of the transformer, or using only the decoder as the main body of the model. The input of the model is the MFCC feature sequence of the audio.

[0058] For video data, the specific method is: use uniform sampling to extract key frames from the video sequence and obtain the key frame sequence of the video.

[0059] In order to reduce the computational complexity of video processing, the video is sampled for video frames. The key frames of the video sequence are the video frames used for subsequent model processing. A method of uniformly adopting the time sequence can be adopted. For example, one video frame is extracted as the key frame in the video clip per second. The extracted video frames are arranged in time sequence to form a key frame sequence of the video.

[0060] The key frame sequence is used as input, and a classifier model, such as a convolutional model or a model based on a transformer, is used to extract features from each key frame to form a key frame feature sequence. Key frame feature extraction is to extract features from video key frames using an image classifier model and output a feature vector of the key frame. The output of the previous layer of the last layer of the image classifier model can be used as the feature vector of the key frame.

[0061] The key frame feature sequence is used as input, and the video classifier model is used to perform entity recognition and attribute extraction on the key frame feature sequence. The video classifier model is a sequence model that takes the time dimension into consideration, such as the transformer encoder.

[0062] Video entity recognition is the process of identifying people, objects, scenes, emotions, and events that appear in a video. Video attribute extraction refers to extracting descriptive features or attributes from video content. For example, visual attributes include color, shape, texture, and motion pattern, and sound attributes include speech rate, volume, and pitch.

[0063] The key frame feature sequence is used as input, and the video sequence-to-sequence model is used to describe the content of the key frame feature sequence. The video sequence-to-sequence model is a sequence model that takes the temporal dimension of the video into consideration, such as a model structure that combines the encoder and decoder of the transformer, or an implementation method that uses only the decoder as the main body of the model.

[0064] Step S12: Generate a first prompt word based on a first prompt word template, input the first prompt word into a first large language model, generate a first output result answering the first prompt word, use the first output result as summary extraction information of the entity recognition information set, the attribute set and the content description set, and generate a text summary information knowledge base corresponding to each multimodal data according to the summary extraction information, wherein the first prompt word template includes a prompt instruction description and entity recognition information sets, attribute sets and content description matching pair lists corresponding to multiple modalities.

[0065] Specifically, the implementation process is as follows:

[0066] First, set the first prompt word template, including prompt instruction Instruct, input and context. Prompt instruction Instruct is a description of the prompt instruction content and required rules. The general prompt word template includes prompt instruction description, input, and context. The prompt instruction description is used to define the task that the prompt word template is expected to complete for the large language model, and includes constraint rules for task execution. The prompt instruction description is a required part of the prompt word template, while input and context are optional in a given prompt word template.

[0067] The first prompt word template follows the specifications of the general prompt word template and is a prompt word for the content summary task based on the entity recognition information set, attributes and content description set matching of a single modality. For example, the summary prompt word of a single modality is as follows:

[0068] Instruction description: For example, give a summary of the following entity, attribute, and content description matching pairs;

[0069] Input: For example, the following is a list of entities, attributes, and content descriptions:

[0070] Entity 1, the associated attributes are attribute 1-1, attribute 1-2,

[0071] Entity 2, the associated attributes are attribute 2-1, attribute 2-2,

[0072] The associated content description is: The video shows characters doing outdoor sports.

[0073] Afterwards, generate the first prompt word template instance, that is, the first prompt word, and use the entity, attribute and content description matching pair list corresponding to each modality as the prompt input. If small sample reasoning ability is required, the entity, attribute and content description matching pairs of a single modality and the corresponding content summary description results can be supplemented in the context part. Small sample reasoning ability refers to the ability of a machine learning model to perform effective reasoning and generalization even with only a small number of samples. In the prompt word template, small sample reasoning is to input a small number of samples as the context in the prompt word template into the large language model. The large language model can use the sample examples given by this context to understand the entity, attribute and content description matching pairs of a single model. Taking the first prompt word as input, the first large language model performs knowledge reasoning based on the first prompt word to generate text summaries of each modality.

[0074] It should be noted that the image, audio and video modes may adopt a unified first prompt word template, and reference may be made to the first prompt word template example.

[0075] Step S13: adopting tool chains such as text segmentation, text embedding vectors and text vector libraries, the text summary information knowledge base corresponding to each multimodal data is encapsulated to form a RAG retrieval tool service corresponding to each multimodal data.

[0076] Specifically, the implementation process is as follows:

[0077] The text summary information of each multimodal data is divided into text blocks and a text block set is generated. The maximum text block length is set, for example, 512 words, and a certain length of overlap is maintained between adjacent text blocks, for example, the overlap interval length is 100 words.

[0078] After the text blocks are segmented, each text block is input into the embedding vector model to calculate the embedding vector of each text block. The embedding vector model is a method of converting high-dimensional discrete data, such as words and sentences, into low-dimensional continuous vector representations. The traditional BERT model can be used for processing, for example, the text block is tokenized using a tokenizer and used as the input of the embedding vector model, and the output of the last layer or another layer of the embedding vector model is used as the embedding vector of the text block.

[0079] The embedded vector set of the text block is stored in a text vector library, such as LlamaIndex, to provide vector similarity calculation support for the RAG retrieval tool service. With the help of LlamaIndex, data can be easily obtained and constructed, and data can be retrieved according to the LLM input prompt. LlamaIndex is used to construct a text vector library with the vector set of the text block. The vector library itself supports vector similarity calculation between the embedded vector of the text block and the embedded vector of the text vector library, such as using cosine similarity. The fastAPI library can be used to encapsulate the RAG retrieval service, and provide the caller with the link address of the RAG retrieval service and the functional description of the service.

[0080] Step S2: Based on the classification model, perform intent recognition on the acquired user query question to obtain an intent recognition result; based on the prompt word template and the large language model, perform question rewriting on the intent recognition result to obtain a question rewriting result; extract sub-questions and keywords from the question rewriting result to obtain multiple sub-questions and keywords corresponding to each of the sub-questions, and form a sub-question list and a keyword list.

[0081] It should be noted that the classification model used for intent recognition in this embodiment is the Bert model. Other classification models can also realize the function of determining the type of legal issues for the original question data in this embodiment. The classification models specifically include but are not limited to Bert, XLNet, ENRIE, Transformer-XL, T5, GPT, MoE and other models, all of which belong to the protection scope of this application. The Bert model is trained in a standard way, and the user query questions are input into the Berth model in the form of vectors for intent recognition prediction. The user query questions related to the law are identified through intent, and no subsequent processing is performed on the user query questions that are not related to the law.

[0082] A second prompt word is generated based on a second prompt word template, the second prompt word is input into a second largest language model, a second output result is generated to answer the second prompt word, and the second output result is used as a question rewrite of the intention recognition result to obtain a question rewrite result, wherein the second prompt word template includes a prompt instruction description and a question rewriting rule, such as synonym expansion, splicing error correction, etc.

[0083] Specifically, the implementation process is as follows:

[0084] First, set the second prompt word template to include prompt instruction Instruct, input and context. Prompt instruction Instruct is also the description of the prompt instruction content, including question rewriting rules, and the user query question is used as the input of the prompt word template. The second prompt word template follows the specifications of the general prompt word template and is oriented to the user query question rewriting task. For example, the example of the second prompt word template is as follows:

[0085] Instruction description: For example, rewrite the following question, pay attention to correcting typos and replacing synonyms, and make the question as written as possible.

[0086] Input: For example, in case A and case B, what are the similarities and differences in the court's judgment on the crime of intentional injury? Please extract relevant information from the judgments of the two cases and conduct a comparative analysis.

[0087] Afterwards, a second prompt word template instance, namely the second prompt word, is generated, and the user's original query question is used as the input of the second prompt word template. With the second prompt word as the input, the second language model performs knowledge reasoning based on the second prompt word to generate a question rewriting result of the user's query.

[0088] After obtaining the question rewriting result, a third prompt word is generated based on a third prompt word template, the third prompt word is input into a third language model, a third output result answering the third prompt word is generated, the third output result is used as a sub-question extraction and a keyword extraction of the question rewriting result, and a sub-question list and a keyword list are obtained, wherein the third prompt word template includes a prompt instruction description and sub-question and keyword extraction rules.

[0089] Specifically, the implementation process is as follows:

[0090] First, the third prompt word template is set to include prompt instruction Instruct, input and context. Prompt instruction Instruct is also the description of the prompt instruction content, including sub-questions and keyword extraction rules.

[0091] Then, a third prompt word template instance is generated, namely, the third prompt word, and the query question after the user's question is rewritten is used as the input of the third prompt word template. With the third prompt word as input, the third language model performs knowledge reasoning based on the third prompt word to generate a sub-question list and a keyword list.

[0092] The third prompt word template follows the specifications of the general prompt word template and is a task for splitting sub-questions and extracting keywords for user query questions. For example, an example of the third prompt word template is shown in Table 1:

[0093]

[0094]

[0095]

[0096] Table 1 User query sub-question extraction and keyword extraction samples

[0097] Step S3: Match the sub-question list with the RAG retrieval tool service in the tool service list, determine the RAG retrieval tool service used for retrieval, and determine the knowledge graph retrieval library used for retrieval based on the keyword list.

[0098] Among them, matching the sub-question list with the RAG retrieval tool service in the tool service list is to add the tool service list to the fourth prompt word template to generate a fourth prompt word, input the fourth prompt word into the fourth language model, and generate a fourth output result answering the fourth prompt word, and the fourth output result indicates that the fourth language model matches and registers at least one RAG retrieval tool service in the tool service list.

[0099] Specifically, the implementation process is as follows:

[0100] First, a fourth prompt word template is set. When designing the fourth prompt word template, the fourth prompt word template is used to instruct the fourth language model to match the sub-question and the text vector library to be queried, and the link address and function description of the tool service of the multi-source knowledge base are registered in the fourth prompt word template;

[0101] The fourth prompt word template follows the specifications of the general prompt word template and is a knowledge base tool service matching task oriented to sub-questions and queries. For example, an example of the fourth prompt word template is shown in Table 2:

[0102]

[0103]

[0104] Table 2 Example of call prompt word template for matching sub-questions with knowledge base services

[0105] After that, the fourth prompt word template instance is generated, that is, the fourth prompt word, which needs to include a list of sub-questions and a keyword list as inputs of the prompt word template. Among them, the keyword list is used to query the knowledge graph service of the multi-source data knowledge base. Taking the fourth prompt word as input, the fourth language model performs knowledge reasoning based on the fourth prompt word, and the output result defines the list of multi-source knowledge base services that need to be queried for each sub-question.

[0106] Step S4: Input each sub-question in the sub-question list into the matching RAG search tool service for RAG search calculation, use several first text blocks with the highest similarity in the search calculation results as the first text context list data, input each keyword in the keyword list into the knowledge graph search library for search, and obtain several second text blocks with the highest similarity as the second text context list data.

[0107] Specifically, the implementation process is as follows:

[0108] First, perform a RAG search on the text vector library, calculate the embedding vector based on each sub-problem input, and obtain the embedding vector of the sub-problem.

[0109] Afterwards, the embedding vector of the sub-problem is similarly calculated with the text vector library to obtain a list of top-K related text blocks.

[0110] RAG retrieval stands for Retrieval-augmented Generation, which combines language models and information retrieval technology. When the model needs to generate text or answer questions, it will first retrieve relevant information from a large document collection, and then use the retrieved information to guide text generation, thereby improving the quality and accuracy of the prediction.

[0111] The RAG retrieval here includes the calculation of the embedding vector for each sub-question text, and based on the text vector library, such as the vector similarity calculation function of LlamaIndex, for example, the vector similarity calculation is implemented using the cosine similarity calculation, and the top-K related text block list is retrieved from the text vector library.

[0112] For knowledge graph retrieval, we perform vector retrieval on the legal document knowledge graph based on the keywords corresponding to each sub-question to obtain the top-K related keywords, entities, etc. We assemble the related keywords, entities, and relationships in the acquired graph knowledge base into graph query prompts and convert them into the query language SQL supported by the graph knowledge base. We perform subgraph retrieval in the legal document knowledge graph library to obtain text blocks, subgraphs, etc. related to the user's query.

[0113] Step S5: using a large language model, remove duplications, merge and sort the first text context list data and the second text context list data, and determine the response result of the large language model as the knowledge reasoning result.

[0114] Specifically, the implementation process is as follows:

[0115] First, a context deduplication and sorting prompt word template is set, which includes the context sorting and merging process of the sub-question list of the user query. The context deduplication and sorting prompt word template follows the specifications of the general prompt word template. The prompt instruction description defines the context deduplication and sorting task description.

[0116] Then, a context-deduplicated and sorted prompt word template instance is generated, and the context list of the sub-question obtained in step S4 is used as the input of the prompt word template. The context-deduplicated and sorted prompt words are used as input, and the knowledge reasoning of the large language model is used to sort the context results of each sub-question in the multi-source knowledge base query and merge them to obtain the top-K context results.

[0117] In some examples, after step S5, the knowledge reasoning result is evaluated using sensitive word filtering and subsequent dialogue feedback from the user query process, and the knowledge reasoning result is modified according to the evaluation result.

[0118] Specifically, the implementation process is as follows:

[0119] First, use sensitive words to filter and process the knowledge reasoning results, and then feed them back to the user, who then uses subsequent dialogue to provide feedback.

[0120] Then, the model is used to correct the knowledge reasoning results based on the feedback from the user's subsequent conversation as the context history for knowledge reasoning. An optional way is to use the context history of the user's conversation as the context of the prompt engineering template for de-duplication, merging and sorting in step S5, and let the large language model reason with reference to the context history.

[0121] The second embodiment of the present invention provides a knowledge reasoning system that integrates multiple sources and multiple modalities, which is implemented using the method steps in the first embodiment. The knowledge reasoning system that integrates multiple sources and multiple modalities includes an encapsulation module, a rewriting module, a matching module, a retrieval module and a merging module.

[0122] Among them, the encapsulation module is used to obtain a multimodal data set, encapsulate each multimodal data in the multimodal data set to form a RAG retrieval tool service, and generate a tool service list. The tool service list includes a service name and a calling address of at least one of the RAG retrieval tool services, and the tool service list also includes at least one preset knowledge graph retrieval library.

[0123] A rewriting module is used to perform intent recognition on the acquired user query question based on the classification model to obtain an intent recognition result, rewrite the question on the intent recognition result based on the prompt word template and the large language model to obtain a question rewriting result, extract sub-questions and keywords from the question rewriting result to obtain multiple sub-questions and keywords corresponding to each of the sub-questions, and form a sub-question list and a keyword list;

[0124] A matching module is used to match the sub-question list with the RAG retrieval tool service in the tool service list, determine the RAG retrieval tool service used for retrieval, and determine the knowledge graph retrieval library used for retrieval based on the keyword list.

[0125] A retrieval module is used to input each sub-question in the sub-question list into a matching RAG retrieval tool service for RAG retrieval calculation, use several first text blocks with the highest similarity in the retrieval calculation results as first text context list data, input each keyword in the keyword list into a knowledge graph retrieval library for retrieval, and obtain several second text blocks with the highest similarity as second text context list data.

[0126] A merging module is used to use a large language model to remove duplications, merge and sort the first text context list data and the second text context list data, and determine the response result of the large language model as the knowledge reasoning result.

[0127] The third embodiment of the present invention provides a knowledge reasoning device that integrates multiple sources and multiple modalities. The knowledge reasoning device includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the computer program is executed by the processor, the steps of the knowledge reasoning method that integrates multiple sources and multiple modalities described in the first embodiment of the present invention are implemented.

[0128] The fourth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-source and multi-modal knowledge reasoning method described in the first embodiment of the present invention are implemented.

[0129] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A knowledge reasoning method integrating multi-source and multi-modal features, characterized in that: The following steps are involved: Acquire a multimodal data set, encapsulate each multimodal data in the multimodal data set to form a RAG retrieval tool service, and generate a tool service list, wherein the tool service list includes a service name and a call address of at least one of the RAG retrieval tool services, and the tool service list also includes at least one preset knowledge graph retrieval library; Based on the classification model, the obtained user query question is subjected to intent recognition to obtain an intent recognition result; based on the prompt word template and the large language model, the intent recognition result is subjected to question rewriting to obtain a question rewriting result; sub-questions and keywords are extracted from the question rewriting result to obtain multiple sub-questions and keywords corresponding to each of the sub-questions, and a sub-question list and a keyword list are formed; Match the sub-question list with the RAG search tool service in the tool service list to determine the RAG search tool service used for the search, and determine the knowledge graph search library used for the search based on the keyword list; Input each sub-question in the sub-question list into a matching RAG search tool service to perform RAG search calculation, use the first text blocks with the highest similarity in the search calculation results as first text context list data, input each keyword in the keyword list into a knowledge graph search library for search, and obtain the second text blocks with the highest similarity as second text context list data; The first text context list data and the second text context list data are de-duplicated, merged and sorted by using a large language model, and a response result of the large language model is determined as a knowledge reasoning result.

2. The knowledge reasoning method integrating multi-source and multi-modality according to claim 1 is characterized in that: The step of respectively encapsulating each multimodal data in the multimodal data set to form a RAG retrieval tool service comprises the following steps: Performing entity recognition, attribute extraction and content description on each multimodal data in the multimodal data set to obtain an entity recognition information set, an attribute set and a content description set corresponding to each multimodal data; Generate a first prompt word based on a first prompt word template, input the first prompt word into a first large language model, generate a first output result answering the first prompt word, use the first output result as summary extraction information of the entity recognition information set, the attribute set and the content description set, and generate a text summary information knowledge base corresponding to each multimodal data according to the summary extraction information, wherein the first prompt word template includes a prompt instruction description and a matching pair list of entity recognition information sets, attribute sets and content description sets corresponding to multiple modalities; The tool chain of text segmentation, text embedding vector and text vector library is used to encapsulate the text summary information knowledge base corresponding to each multimodal data, forming a RAG retrieval tool service corresponding to each multimodal data.

3. The knowledge reasoning method integrating multi-source and multi-modality according to claim 1 is characterized in that: The question is rewritten on the intention recognition result to obtain a question rewriting result, which is specifically: Generate a second prompt word based on a second prompt word template, input the second prompt word into a second largest language model, generate a second output result answering the second prompt word, rewrite the second output result as the question of the intention recognition result, and obtain a question rewriting result, wherein the second prompt word template includes a prompt instruction description and a question rewriting rule.

4. The knowledge reasoning method integrating multi-source and multi-modality according to claim 1 is characterized in that: The rewriting result of the question to extract sub-questions and keywords is specifically: A third prompt word is generated based on a third prompt word template, the third prompt word is input into a third language model, a third output result answering the third prompt word is generated, the third output result is used as a sub-question extraction and a keyword extraction of the question rewriting result, and a sub-question list and a keyword list are obtained, wherein the third prompt word template includes a prompt instruction description and sub-question and keyword extraction rules.

5. The knowledge reasoning method integrating multi-source and multi-modality according to claim 1 is characterized in that: The matching of the sub-question list with the RAG search tool service in the tool service list is specifically as follows: The tool service list is added to the fourth prompt word template to generate a fourth prompt word, the fourth prompt word is input into the fourth language model, and a fourth output result answering the fourth prompt word is generated, wherein the fourth output result indicates that the fourth language model matches and registers at least one RAG retrieval tool service in the tool service list.

6. The method for knowledge reasoning integrating multi-source and multi-modality according to claim 1 is characterized in that: The method further comprises: The knowledge reasoning results are evaluated and assessed using the subsequent dialogue feedback of the sensitive word filtering and user query process, and the knowledge reasoning results are corrected according to the evaluation results.

7. A knowledge reasoning system integrating multi-source and multi-modal features, characterized in that: Includes the following modules: A packaging module, used to obtain a multimodal data set, respectively package each multimodal data in the multimodal data set to form a RAG retrieval tool service, and generate a tool service list, wherein the tool service list includes a service name and a call address of at least one of the RAG retrieval tool services, and the tool service list also includes at least one preset knowledge graph retrieval library; A rewriting module is used to perform intent recognition on the acquired user query question based on the classification model to obtain an intent recognition result, rewrite the question on the intent recognition result based on the prompt word template and the large language model to obtain a question rewriting result, extract sub-questions and keywords from the question rewriting result to obtain multiple sub-questions and keywords corresponding to each of the sub-questions, and form a sub-question list and a keyword list; A matching module, used for matching the sub-question list with the RAG search tool service in the tool service list, determining the RAG search tool service to be used for the search, and determining the knowledge graph search library to be used for the search according to the keyword list; A retrieval module, used for inputting each sub-question in the sub-question list into a matching RAG retrieval tool service to perform RAG retrieval calculation, taking a plurality of first text blocks with the highest similarity in the retrieval calculation results as first text context list data, and inputting each keyword in the keyword list into a knowledge graph retrieval library for retrieval, to obtain a plurality of second text blocks with the highest similarity as second text context list data; A merging module is used to use a large language model to remove duplications, merge and sort the first text context list data and the second text context list data, and determine the response result of the large language model as the knowledge reasoning result.

8. The knowledge reasoning system integrating multi-source and multi-modality according to claim 7 is characterized in that: The packaging module is also used for: Performing entity recognition, attribute extraction and content description on each multimodal data in the multimodal data set to obtain an entity recognition information set, an attribute set and a content description set corresponding to each multimodal data; Generate a first prompt word based on a first prompt word template, input the first prompt word into a first large language model, generate a first output result answering the first prompt word, use the first output result as summary extraction information of the entity recognition information set, the attribute set and the content description set, and generate a text summary information knowledge base corresponding to each multimodal data according to the summary extraction information, wherein the first prompt word template includes a prompt instruction description and a matching pair list of entity recognition information sets, attribute sets and content descriptions corresponding to multiple modalities; By using a tool chain of text segmentation, text embedding vectors and text vector library, the text summary information knowledge base corresponding to each multimodal data is encapsulated to form a RAG retrieval tool service corresponding to each multimodal data.

9. A knowledge reasoning device integrating multi-source and multi-modal features, characterized in that: The knowledge reasoning device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the multi-source multi-modal knowledge reasoning method as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the multi-source multi-modal knowledge reasoning method according to any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Large model retrieval method, device and equipment based on prior atlas and storage medium

    CN119357366A

  • Large model retrieval method, device, equipment and storage medium based on prior graph

    CN119357366B

  • Intention recognition method for multiple rounds of dialogues in question and answer

    CN119441464A

  • A method for intent recognition in multi-turn conversations during question and answer

    CN119441464B

  • Media file querying and calling method and system based on AI dialogue scene

    CN119474419A