Knowledge reasoning method, system and equipment for knowledge-intensive tasks

Abstract

The invention provides a knowledge reasoning method, system and equipment for knowledge-intensive tasks, and belongs to the technical field of computers. The method comprises the following steps: inputting a multi-modal query text from a user terminal into a preset text recognition fusion model to determine a corresponding user query intention and a corresponding matched knowledge source type; and based on the matched knowledge source type and the multi-modal query text, matching a plurality of multi-modal query data from the corresponding knowledge base to construct a multi-modal query data set. Dividing the multi-modal query data according to preset modal type grouping, and inputting the multi-modal query data into a multi-modal data association degree measurement model to determine a first association score between the multi-modal query data; based on each first association score and a first preset threshold value, performing fusion processing on each piece of multi-modal query data to obtain multi-modal fusion feature information; and inputting the multi-modal fusion feature information into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multi-modal query text according to an engine output result.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a knowledge reasoning method, system and device for knowledge-intensive tasks. Background Art

[0002] In the field of knowledge-intensive task processing, knowledge reasoning has always been the key to improving task processing efficiency and accuracy. Traditional knowledge reasoning methods often rely on a single text input and a fixed knowledge base. This processing method is inadequate when faced with multimodal query texts. Multimodal query texts, that is, information containing multiple data types such as text, images, and audio, require the reasoning system to have more powerful data parsing and processing capabilities.

[0003] Although there have been some attempts to apply deep learning models to knowledge reasoning, these methods are mostly limited to the processing of text data, and there is still a lack of effective means for the fusion and reasoning of multimodal data. For example, some text processing methods based on models such as BERT or TextCNN have achieved remarkable results in the field of natural language processing, but they are stretched when processing multimodal data. These models often cannot effectively capture the characteristics of non-text data such as images and audio, and cannot establish effective associations between multimodal data.

[0004] Based on this, there is an urgent need for an efficient, accurate and flexible knowledge reasoning technology solution that can be used for multimodal data and knowledge-intensive tasks. Summary of the invention

[0005] To solve the above problems, the embodiments of the present application provide a knowledge reasoning method, system and device for knowledge-intensive tasks.

[0006] On the one hand, an embodiment of the present application provides a knowledge reasoning method for knowledge-intensive tasks, the method comprising: Inputting the multimodal query text from the user terminal into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type; wherein the text recognition fusion model is obtained by fusing the BERT model and the TextCNN model; Based on the matching knowledge source type and the multimodal query text, matching a plurality of multimodal query data from a corresponding knowledge base to construct a multimodal query data set; According to the preset modality type grouping, each of the multimodal query data in the multimodal query data set is divided and input into a multimodal data association measurement model to determine a first association score between each of the multimodal query data; wherein the multimodal data association measurement model includes a plurality of network structures, each of the network structures is used to process data corresponding to different preset modality type groups; Based on the comparison result of each of the first association scores and the first preset threshold, the corresponding multimodal query data are fused to obtain multimodal fusion feature information; The multimodal fusion feature information is input into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multimodal query text according to the engine output result.

[0007] In one implementation of the present application, a multimodal query text from a user terminal is input into a preset text recognition fusion model to determine the corresponding user query intention, specifically including: Determine, by means of the text recognition fusion model, first output information and second output information of the multimodal query text in the BERT model and the TextCNN model respectively; Determine a first value matrix and a first key matrix according to the first output information, and determine a second value matrix and a second key matrix according to the second output information; According to a preset attention mechanism, calculating a first fusion weight corresponding to the first output information and a second fusion weight corresponding to the second output information; The fusion output information is determined based on the sum of the first product value of the first fusion weight and the first value matrix and the second product value of the second fusion weight and the second value matrix to match the user query intent in the preset intent list according to the fusion output information.

[0008] In one implementation of the present application, a multimodal query text from a user terminal is input into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type, specifically including: Converting the multimodal query text into a structured query statement by using semantic parsing technology; The structured query statement is matched with each knowledge source of predefined classification in the knowledge base to determine the matching knowledge source type according to the matching result; wherein the predefined classification is based on data structure, source, format or content classification; the matching knowledge source type includes at least one or more of the following: structured data, semi-structured data, unstructured data.

[0009] In one implementation of the present application, based on the matching knowledge source type and the multimodal query text, a plurality of multimodal query data are matched from a corresponding knowledge base to construct a multimodal query data set, specifically including: Encoding the user query intent into an intent vector; Calculating the similarity between the intention vector and each knowledge source vector corresponding to the matching knowledge source type in the knowledge base to score the knowledge source relevance; According to the knowledge source relevance score and knowledge source weight formula , determine the knowledge source weights corresponding to each knowledge source; where, For the The weight of knowledge sources, For the The relevance score of each knowledge source, For the The relevance score of each knowledge source, The total number of knowledge sources corresponding to the matching knowledge source type; Taking the knowledge source whose knowledge source weight is greater than a second preset threshold as a query knowledge source; According to the structured query statement and each query knowledge source, the corresponding multimodal query data are queried, and each multimodal query data obtained by the query is added to the multimodal query data set.

[0010] In one implementation of the present application, according to the preset modality type grouping, each of the multimodal query data in the multimodal query data set is divided and input into a multimodal data association measurement model to determine a first association score between each of the multimodal query data, specifically including: Inputting the multimodal query data grouped by different preset modality types into the corresponding network structures for processing respectively to determine the corresponding query data feature set; wherein the number of the plurality of network structures is two; Inputting each of the query data feature sets into a preset attention score function, determining the attention score of each query data feature between each of the query data feature sets, and obtaining a corresponding number of query data feature groups; wherein the query data feature group includes a first query data feature and a second query data feature; Determine the first attention weight and the second attention weight corresponding to the query data feature group respectively according to each of the attention scores and the preset weight formula; Calculating a third product value of the first attention weight and the feature value of the first query data feature, calculating a fourth product value of the second attention weight and the feature value of the second query data feature, and adding the third product value and the fourth product value as a second association score; Each of the second association scores is input into a preset Sigmoid function to determine the first association score between the corresponding multimodal query data.

[0011] In one implementation of the present application, based on the comparison result of each of the first association scores with the first preset threshold, the corresponding multimodal query data are fused to obtain multimodal fusion feature information, which specifically includes: Comparing each of the first association scores with the first preset threshold respectively; When the first association score is greater than the first preset threshold, concatenating and fusing the multimodal query data to obtain the multimodal fusion feature information; Otherwise, update the user query intention and / or the matching knowledge source type.

[0012] In one implementation of the present application, before inputting the multimodal fusion feature information into a preset context-aware reasoning engine, the method further includes: Embed entities and relations in the preset knowledge graph into a low-dimensional vector space to align with the vector representation of the pre-trained large language model; The integrated preset knowledge graph and the pre-trained large language model are used as the preset context-aware reasoning engine.

[0013] In one implementation of the present application, the method further includes: Determining knowledge base update information according to the inference result and the inference result feedback information from the user terminal; The knowledge base is updated according to a preset learning rate and the knowledge base update information and the original knowledge base information; the preset learning rate indicates the degree of influence of the knowledge base update information on the original knowledge base information.

[0014] On the other hand, an embodiment of the present application further provides a knowledge reasoning system for knowledge-intensive tasks, the system comprising: A first input module is used to input the multimodal query text from the user terminal into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type; wherein the text recognition fusion model is obtained by fusing the BERT model and the TextCNN model; A matching module, configured to match a plurality of multimodal query data from a corresponding knowledge base based on the matching knowledge source type and the multimodal query text, so as to construct a multimodal query data set; A partitioning input module, used for partitioning each of the multimodal query data in the multimodal query data set according to the preset modal type grouping and inputting the multimodal data association measurement model to determine a first association score between each of the multimodal query data; wherein the multimodal data association measurement model includes a plurality of network structures, each of the network structures is used to process data corresponding to different preset modal type groups; a fusion processing module, configured to perform fusion processing on the corresponding multimodal query data based on the comparison result of each of the first association scores and the first preset threshold value to obtain multimodal fusion feature information; The second input module is used to input the multimodal fusion feature information into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multimodal query text according to the engine output result.

[0015] On the other hand, the embodiment of the present application further provides a knowledge reasoning device for knowledge-intensive tasks, the device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform a knowledge reasoning method for knowledge-intensive tasks as described above.

[0016] Compared with the prior art, the present invention has the following significant effects: By introducing a text recognition fusion model and a multimodal data association measurement model, this application can accurately process multimodal query texts, and effectively match and fuse multimodal data, thereby improving the accuracy and efficiency of knowledge reasoning. It can also process various types of data modalities and adapt to different types of knowledge sources, thereby enhancing the flexibility and adaptability of knowledge reasoning, enabling it to be widely used in various knowledge-intensive tasks. In addition, by accurately identifying user query intent and quickly giving reasoning results, this application can significantly improve user experience and meet user expectations and needs for knowledge reasoning systems. The technical solution of this application introduces new technical ideas and methods in the field of knowledge reasoning, promotes the development and innovation of knowledge reasoning technology, and can provide new ideas and directions for research and application in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a flow chart of a knowledge reasoning method for knowledge-intensive tasks in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a knowledge reasoning system for knowledge-intensive tasks in an embodiment of the present application; Figure 3 This is a schematic diagram of the structure of a knowledge reasoning device used for knowledge-intensive tasks in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0019] In the field of knowledge-intensive task processing, accurately fusing structured and unstructured data and performing effective reasoning has always been a technical challenge. With the continuous advancement of artificial intelligence technology, especially in natural language processing, large language models have become an important tool to achieve this goal. Although these models perform well in processing simple data and tasks, they still face challenges in understanding and generating accurate and effective reasoning results for complex tasks. How to intelligently perform knowledge fusion and effective reasoning, realize adaptive fusion of knowledge, perceive and reason about context, learn the association between different modal data, and dynamically update knowledge are still the key issues to solve the accurate fusion and effective reasoning of data in knowledge-intensive tasks.

[0020] Based on this, the embodiments of the present application provide a knowledge reasoning method, system and device for knowledge-intensive tasks, which are used to solve the current technical problem that it is difficult to perform knowledge reasoning on multimodal data efficiently, accurately and flexibly.

[0021] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.

[0022] The present application embodiment provides a knowledge reasoning method for knowledge-intensive tasks, such as Figure 1 As shown, the method may include steps S101-S105: S101, the server inputs the multimodal query text from the user terminal into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type.

[0023] Among them, the text recognition fusion model is obtained by fusing the BERT model and the TextCNN model.

[0024] It should be noted that the server, as the executor of the knowledge reasoning method for knowledge-intensive tasks, is only an example, and the executor is not limited to the server, and this application does not make any specific limitations on this.

[0025] That is to say, this application uses the fusion of the deep learning model BERT model and the TextCNN model to construct a text recognition fusion model, and uses the text recognition fusion model obtained by the fusion of the two to recognize and process the multimodal query text corresponding to the natural language input by the user through the user terminal. This application can perform fusion processing on the output of the multimodal query text according to the BERT model and the TextCNN model, for example, using an attention mechanism for fusion, or using ensemble learning for fusion.

[0026] Specifically, the present application uses the attention mechanism to implement the above output fusion and obtain the user query intention. The multimodal query text from the user terminal is input into the preset text recognition fusion model to determine the corresponding user query intention, specifically including: Through the text recognition fusion model, determine the first output information and the second output information of the multimodal query text in the BERT model and the TextCNN model respectively. Determine the first value matrix and the first key matrix according to the first output information, and determine the second value matrix and the second key matrix according to the second output information. According to the preset attention mechanism, calculate the first fusion weight corresponding to the first output information and the second fusion weight corresponding to the second output information. Determine the fusion output information based on the sum of the first product value of the first fusion weight and the first value matrix and the second product value of the second fusion weight and the second value matrix, so as to match the user query intent in the preset intent list according to the fusion output information.

[0027] In other words, the multimodal query text is input into the BERT model, the BERT model extracts the text features of the multimodal query text, and then encodes it into the first output information, which contains the multimodal query text such as vocabulary, grammar or preliminary semantic information; then, the first output information is input into the TextCNN model, the TextCNN model will further extract the features in the semantic information, and generate the second output information, such as deep semantics and contextual relationships. After obtaining the output information of the BERT model and the TextCNN model respectively, the present application constructs two sets of value matrices and key matrices corresponding to the two output information respectively. The first output information and the second output information are generally in the form of vectors or matrices. Specifically, after obtaining the first output information, the server can generate an output matrix by performing various operations such as linear transformation, nonlinear transformation, and dimensional transformation, using a preset transformation weight matrix and a preset transformation bias item of a linear transformation. The server can pre-store different preset transformation weight matrices and different preset transformation bias items to convert the first output information into a first value matrix and a first key matrix, respectively; similarly, after obtaining the second output information, the server can convert the second output information into a second value matrix and a second key matrix through the above-mentioned conversion method. Converting the output information into a value matrix and a key matrix is ​​not limited to the above-mentioned linear transformation implementation method, and can also be implemented by methods such as convolutional neural networks, attention splicing, feature splicing, etc., and this application does not make specific limitations on this.

[0028] Among them, the value (V) matrix and the key (K) matrix corresponding to the same output information are the same matrix. Subsequently, the present application adopts an attention mechanism, based on the preset query (Q) matrix, to calculate the similarity between the query matrix and each key matrix, and converts it into a fusion weight through a softmax function. Specifically, the present application calculates the dot product or cosine similarity between Q and the first key matrix K1 and the second key matrix K2, respectively, and obtains two fusion weights through a softmax function. Subsequently, the fusion output information w1*V1+w2*V2 is fused and calculated by calculating the first fusion weight w1 and the second fusion weight w2, where V1 is the first value matrix and V2 is the second value matrix. In addition, after obtaining the fused output information, the server can input it into a pre-set classifier or decoder to complete the recognition of the user's query intention.

[0029] The output information of the BERT model and the TextCNN model, combined with the attention mechanism, can more comprehensively capture the key information in the multimodal query text and improve the accuracy of user query intent recognition. The use of fusion weights enables the information output by different models to be weighted according to its importance, further improving the accuracy of intent recognition.

[0030] In another embodiment of the present application, the multimodal query text from the user terminal is input into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type, specifically including: The multimodal query text is converted into a structured query statement through semantic parsing technology. The structured query statement is matched with each knowledge source of predefined classification in the knowledge base to determine the matching knowledge source type according to the matching result. The predefined classification is based on data structure, source, format or content classification. The matching knowledge source type includes at least one or more of the following: structured data, semi-structured data, and unstructured data.

[0031] That is to say, the present application can quickly determine the type of matching knowledge source based on the multimodal query text, and convert the multimodal query text into a structured query statement through semantic parsing technology, thereby simplifying the query process and improving the query efficiency. Matching with the predefined classified knowledge sources in the knowledge base can quickly and accurately determine the type of matching knowledge source, providing a basis for subsequent data matching and reasoning.

[0032] S102, the server matches a plurality of multimodal query data from a corresponding knowledge base based on matching knowledge source types and multimodal query texts to construct a multimodal query data set.

[0033] In the embodiment of the present application, the above-mentioned matching of the knowledge source type and the multimodal query text, matching multiple multimodal query data from the corresponding knowledge base to construct a multimodal query data set specifically includes: The user query intention is encoded into an intention vector. The similarity between the intention vector and each knowledge source vector corresponding to the matching knowledge source type in the knowledge base is calculated to score the knowledge source relevance. According to the relevance score of each knowledge source and the knowledge source weight formula , determine the knowledge source weights corresponding to each knowledge source. Among them, For the The weight of knowledge sources, For the The relevance score of each knowledge source, For the The relevance score of each knowledge source, The total number of knowledge sources corresponding to the matching knowledge source type. The knowledge source whose knowledge source weight is greater than the second preset threshold is used as the query knowledge source. According to the structured query statement and each query knowledge source, the corresponding multimodal query data is queried, and the multimodal query data obtained by the query is added to the multimodal query data set.

[0034] That is to say, after determining the type of knowledge source required by the user, the present application can locate the knowledge source and query the relevant knowledge. The queried knowledge includes unstructured knowledge, semi-structured knowledge, and structured knowledge. The present application needs to efficiently match multimodal data related to the query intent from a huge knowledge base. Therefore, by calculating the similarity between the intention vector and the knowledge source vector, such as cosine similarity, inverse Euclidean distance, etc., the knowledge source correlation score between the user's query intent and the knowledge source is obtained. The knowledge source weight is calculated using the above-mentioned knowledge source weight formula. Subsequently, the query knowledge source is obtained by comparing the knowledge source weight with the preset second preset threshold. The second preset threshold can be set by the user during actual use, and the present application does not make specific restrictions on this. Subsequently, the server queries various multimodal query data, such as images and texts, from the query knowledge source through the structured query statement obtained above.

[0035] By calculating the similarity between the intent vector and the knowledge source vector and combining the knowledge source weight formula, the knowledge source that is highly relevant to the query intent can be screened out, thereby improving the accuracy and efficiency of data matching. Setting the second preset threshold further ensures the quality of the query knowledge source and reduces the interference of irrelevant data.

[0036] Among them, the above-mentioned application queries the knowledge source through structured query statements. When obtaining relevant knowledge, semantic matching is performed for unstructured knowledge; for semi-structured data, the corresponding query technology is used to match relevant knowledge. At the same time, for the queried knowledge, text fusion technology is used to integrate information from different knowledge sources into a coherent and consistent knowledge representation. For structured data, semantic web technology (RDF) is used to align and fuse concepts, entities, and relationships in different texts to form a unified semantic representation; then a text summary algorithm (TextRank) is used to extract key information from multiple texts and generate a concise summary. Finally, different modal data are combined to construct a multimodal query dataset.

[0037] S103: The server divides each multimodal query data in the multimodal query data set according to a preset modality type grouping and inputs the divided data into a multimodal data association measurement model to determine a first association score between each multimodal query data.

[0038] Among them, the multimodal data association measurement model includes multiple network structures, and each network structure is used to process data corresponding to different preset modal type groups.

[0039] In an embodiment of the present application, according to the preset modality type grouping, each multimodal query data in the multimodal query data set is divided and input into a multimodal data association measurement model to determine a first association score between each multimodal query data, specifically including: The multimodal query data grouped by different preset modal types are respectively input into the corresponding network structure for processing to determine the corresponding query data feature set. Among them, the number of multiple network structures is two. Each query data feature set is respectively input into the preset attention score function, the attention score of each query data feature between each query data feature set is determined, and a corresponding number of query data feature groups are obtained. Among them, the query data feature group includes a first query data feature and a second query data feature. According to each attention score and the preset weight formula, the first attention weight and the second attention weight corresponding to the query data feature group are determined. The third product value of the first attention weight and the eigenvalue of the first query data feature is calculated, and the fourth product value of the second attention weight and the eigenvalue of the second query data feature is calculated, and the sum of the third product value and the fourth product value is used as the second association score. Each second association score is input into the preset Sigmoid function to determine the first association score between the corresponding multimodal query data.

[0040] That is to say, the multimodal data association measurement model of the present application adopts a dual-stream network structure, in which one stream is a convolutional neural network (CNN) for extracting image features, and the other stream is a Transformer architecture for processing text data. The preset modal type groupings of the present application include image groupings and text groupings, and each multimodal query data is processed by a multimodal data association measurement model with a dual-stream network structure. The multimodal data association measurement model of the present application can calculate the attention score between two multimodal query data of different modal types in the query data feature set through a preset attention score function. The preset attention score function is as follows:

[0041]

[0042] in, , Respectively represent Group query data feature group text and images For the attention score of feature fusion representation, and is the learnable weight matrix, represents element-wise multiplication, , are the preset bias items respectively; Represents a function that converts text to the same dimensions as an image, such as a fully connected layer.

[0043] The first attention weight and the second attention weight are calculated by using the preset weight formula and the above attention score. The preset weight formula is as follows:

[0044]

[0045] in, , are the first attention weight and the second attention weight respectively.

[0046] Server Computing , get the second correlation score The server then enters , obtaining a first relevance score corresponding to each multimodal query data in the query data feature set.

[0047] This application can extract the respective feature sets by inputting the multimodal query data into the corresponding network structure for processing; using the attention score function and weight formula, it can more accurately evaluate the correlation between different query data features and improve the accuracy of the correlation score. The use of the Sigmoid function makes the correlation score smoother, which is convenient for subsequent data fusion processing.

[0048] S104: The server performs fusion processing on the corresponding multimodal query data based on the comparison result of each first association score with the first preset threshold to obtain multimodal fusion feature information.

[0049] In the embodiment of the present application, based on the comparison result of each first association score with the first preset threshold, the corresponding multimodal query data is fused to obtain multimodal fusion feature information, which specifically includes: Each first correlation score is compared with a first preset threshold. If the first correlation score is greater than the first preset threshold, each multimodal query data is spliced ​​and fused to obtain multimodal fusion feature information. Otherwise, the user query intent is updated and / or the knowledge source type is matched.

[0050] Among them, the first preset threshold can be set by the user according to the actual usage scenario. By setting the first preset threshold, multimodal query data with strong correlation can be screened out for fusion, avoiding the interference of irrelevant data. The splicing and fusion method is simple and effective, which can preserve the integrity of the original data and improve the quality of the fused data. When the correlation score does not meet the threshold requirements, the accuracy of data matching can be further improved by updating the user query intent or matching the knowledge source type.

[0051] For example, the multimodal query text is: the number and amount of sales orders in August, arranged in descending order; through the above steps, the present application uses the deep learning models BERT and TextCNN to identify the user input intent and determine what type of knowledge source the user needs, such as sales data, order information, etc. Through semantic parsing technology, the natural language query input by the user is converted into a structured query statement to more accurately locate the required knowledge source. For structured data sources (such as databases), SQL query technology is used to retrieve sales order data in August. For unstructured or semi-structured data sources (such as text reports or web pages), semantic matching and relevant information extraction are performed. Subsequently, the text and the image are spliced ​​in the same information. In the present application, when the first association score is less than or equal to the first preset threshold, it means that the multimodal query data obtained by matching is not accurate enough. The server can re-execute the above steps of user query intent and / or matching knowledge source type to update the multimodal query data. After repeating the predetermined number of times, an alarm message can be generated so that relevant personnel can intervene to repair the error.

[0052] S105, the server inputs the multimodal fusion feature information into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multimodal query text according to the engine output result.

[0053] In the embodiment of the present application, before inputting the multimodal fusion feature information into the preset context-aware reasoning engine, the method further includes: Embed entities and relations in the preset knowledge graph into a low-dimensional vector space to align with the vector representation of the pre-trained large language model. Use the integrated preset knowledge graph and the pre-trained large language model as the preset context-aware reasoning engine. The pre-trained large language model is such as Qwen2.

[0054] This application embeds entities and relationships in the knowledge graph into a low-dimensional vector space and aligns them with the vector representation of the pre-trained large language model, which can fully utilize the structured information in the knowledge graph and the semantic understanding ability of the pre-trained large language model, and improve the reasoning ability of the reasoning engine. The integrated reasoning engine can understand the context information more comprehensively, thereby giving more accurate reasoning results.

[0055] In another embodiment of the present application, in order to maintain the timeliness and accuracy of the knowledge base, the method further includes: According to the inference result and the inference result feedback information from the user terminal, the knowledge base update information is determined. According to the preset learning rate and the knowledge base update information and the original knowledge base information, the knowledge base is updated. The preset learning rate indicates the degree of influence of the knowledge base update information on the original knowledge base information.

[0056] This application can timely discover errors or deficiencies in the knowledge base by collecting the inference result feedback information of the user terminal. Combined with the preset learning rate and knowledge base update information, the knowledge base can be updated smoothly to avoid system instability or performance degradation caused by the update. Real-time updating of the knowledge base can ensure the accuracy and timeliness of the inference results and improve the overall performance of the system.

[0057] Through the above technical solutions, this application can accurately process multimodal query texts and effectively match and fuse multimodal data by introducing text recognition fusion models and multimodal data association measurement models, thereby improving the accuracy and efficiency of knowledge reasoning. It can also process various types of data modalities and adapt to different types of knowledge sources, thereby enhancing the flexibility and adaptability of knowledge reasoning, enabling it to be widely used in various knowledge-intensive tasks. In addition, by accurately identifying user query intentions and quickly giving reasoning results, this application can significantly improve user experience and meet user expectations and needs for knowledge reasoning systems. The technical solution of this application introduces new technical ideas and methods in the field of knowledge reasoning, promotes the development and innovation of knowledge reasoning technology, and can provide new ideas and directions for research and application in related fields.

[0058] Figure 2 A schematic diagram of the structure of a knowledge reasoning system for knowledge-intensive tasks provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the knowledge reasoning system 200 for knowledge-intensive tasks includes: The first input module 201 is used to input the multimodal query text from the user terminal into the preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type. The text recognition fusion model is obtained by fusion of the BERT model and the TextCNN model. The matching module 202 is used to match multiple multimodal query data from the corresponding knowledge base based on the matching knowledge source type and the multimodal query text to construct a multimodal query data set. The partition input module 203 is used to group the multimodal query data in the multimodal query data set according to the preset modal type, and input the multimodal data association measurement model to determine the first association score between each multimodal query data. The multimodal data association measurement model includes multiple network structures, each of which is used to process data corresponding to different preset modal type groups. The fusion processing module 204 is used to fuse the corresponding multimodal query data based on the comparison result of each first association score with the first preset threshold value to obtain multimodal fusion feature information. The second input module 205 is used to input the multimodal fusion feature information into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multimodal query text according to the engine output result.

[0059] Figure 3A schematic diagram of the structure of a knowledge reasoning device for knowledge-intensive tasks provided in an embodiment of the present application, such as Figure 3 As shown, the device includes: At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute: The multimodal query text from the user terminal is input into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type. Among them, the text recognition fusion model is obtained by fusion of the BERT model and the TextCNN model. Based on the matching knowledge source type and the multimodal query text, multiple multimodal query data are matched from the corresponding knowledge base to construct a multimodal query data set. According to the preset modal type grouping, each multimodal query data in the multimodal query data set is divided and input into the multimodal data association measurement model to determine the first association score between each multimodal query data. Among them, the multimodal data association measurement model includes multiple network structures, each network structure is used to process data corresponding to different preset modal type groups. Based on the comparison result of each first association score with the first preset threshold, the corresponding multimodal query data is fused to obtain multimodal fusion feature information. The multimodal fusion feature information is input into a preset context-aware reasoning engine to obtain the reasoning result corresponding to the multimodal query text according to the engine output result.

[0060] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0061] The system and equipment provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and equipment also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and equipment will not be repeated here.

[0062] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0063] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A knowledge reasoning method for knowledge-intensive tasks, characterized in that: The method comprises: Inputting the multimodal query text from the user terminal into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type; wherein the text recognition fusion model is obtained by fusing the BERT model and the TextCNN model; Based on the matching knowledge source type and the multimodal query text, matching a plurality of multimodal query data from a corresponding knowledge base to construct a multimodal query data set; According to the preset modality type grouping, each of the multimodal query data in the multimodal query data set is divided and input into a multimodal data association measurement model to determine a first association score between each of the multimodal query data; wherein the multimodal data association measurement model includes a plurality of network structures, each of the network structures is used to process data corresponding to different preset modality type groups; Based on the comparison result of each of the first association scores and the first preset threshold, the corresponding multimodal query data are fused to obtain multimodal fusion feature information; The multimodal fusion feature information is input into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multimodal query text according to the engine output result.

2. A knowledge reasoning method for knowledge-intensive tasks according to claim 1, characterized in that: The multimodal query text from the user terminal is input into the preset text recognition fusion model to determine the corresponding user query intention, including: Determine, by means of the text recognition fusion model, first output information and second output information of the multimodal query text in the BERT model and the TextCNN model respectively; Determine a first value matrix and a first key matrix according to the first output information, and determine a second value matrix and a second key matrix according to the second output information; According to a preset attention mechanism, calculating a first fusion weight corresponding to the first output information and a second fusion weight corresponding to the second output information; The fusion output information is determined based on the sum of the first product value of the first fusion weight and the first value matrix and the second product value of the second fusion weight and the second value matrix to match the user query intent in the preset intent list according to the fusion output information.

3. A knowledge reasoning method for knowledge-intensive tasks according to claim 2, characterized in that: The multimodal query text from the user terminal is input into the preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type, specifically including: Converting the multimodal query text into a structured query statement by using semantic parsing technology; The structured query statement is matched with each knowledge source of predefined classification in the knowledge base to determine the matching knowledge source type according to the matching result; wherein the predefined classification is based on data structure, source, format or content classification; the matching knowledge source type includes at least one or more of the following: structured data, semi-structured data, unstructured data.

4. A knowledge reasoning method for knowledge-intensive tasks according to claim 3, characterized in that: Based on the matching knowledge source type and the multimodal query text, a plurality of multimodal query data are matched from a corresponding knowledge base to construct a multimodal query data set, specifically including: Encoding the user query intent into an intent vector; Calculating the similarity between the intention vector and each knowledge source vector corresponding to the matching knowledge source type in the knowledge base to score the knowledge source relevance; According to the knowledge source relevance score and knowledge source weight formula , determine the knowledge source weights corresponding to each knowledge source; where, For the The weight of knowledge sources, For the The knowledge source relevance score, For the The knowledge source relevance score, The total number of knowledge sources corresponding to the matching knowledge source type; Taking the knowledge source whose knowledge source weight is greater than a second preset threshold as a query knowledge source; According to the structured query statement and each query knowledge source, the corresponding multimodal query data are queried, and each multimodal query data obtained by the query is added to the multimodal query data set.

5. The knowledge reasoning method for knowledge-intensive tasks according to claim 1, characterized in that: According to the preset modality type grouping, each of the multimodal query data in the multimodal query data set is divided and input into a multimodal data association measurement model to determine a first association score between each of the multimodal query data, specifically including: Inputting the multimodal query data grouped by different preset modality types into the corresponding network structures for processing respectively to determine the corresponding query data feature set; wherein the number of the plurality of network structures is two; Inputting each of the query data feature sets into a preset attention score function, determining the attention score of each query data feature between each of the query data feature sets, and obtaining a corresponding number of query data feature groups; wherein the query data feature group includes a first query data feature and a second query data feature; Determine the first attention weight and the second attention weight corresponding to the query data feature group respectively according to each of the attention scores and the preset weight formula; Calculating a third product value of the first attention weight and the feature value of the first query data feature, calculating a fourth product value of the second attention weight and the feature value of the second query data feature, and adding the third product value and the fourth product value as a second association score; Each of the second association scores is input into a preset Sigmoid function to determine the first association score between the corresponding multimodal query data.

6. A knowledge reasoning method for knowledge-intensive tasks according to claim 1, characterized in that: Based on the comparison result of each of the first association scores and the first preset threshold, the corresponding multimodal query data are fused to obtain multimodal fusion feature information, which specifically includes: Comparing each of the first association scores with the first preset threshold respectively; When the first association score is greater than the first preset threshold, concatenating and fusing the multimodal query data to obtain the multimodal fusion feature information; Otherwise, update the user query intention and / or the matching knowledge source type.

7. A knowledge reasoning method for knowledge-intensive tasks according to claim 1, characterized in that: Before inputting the multimodal fusion feature information into a preset context-aware reasoning engine, the method further includes: Embed entities and relations in the preset knowledge graph into a low-dimensional vector space to align with the vector representation of the pre-trained large language model; The integrated preset knowledge graph and the pre-trained large language model are used as the preset context-aware reasoning engine.

8. The knowledge reasoning method for knowledge-intensive tasks according to claim 1, characterized in that: The method further comprises: Determining knowledge base update information according to the inference result and the inference result feedback information from the user terminal; The knowledge base is updated according to a preset learning rate and the knowledge base update information and the original knowledge base information; the preset learning rate indicates the degree of influence of the knowledge base update information on the original knowledge base information.

9. A knowledge reasoning system for knowledge-intensive tasks, characterized in that: The system comprises: A first input module is used to input the multimodal query text from the user terminal into a preset text recognition fusion model to determine the corresponding user query intention and its corresponding matching knowledge source type; wherein the text recognition fusion model is obtained by fusing the BERT model and the TextCNN model; A matching module, configured to match a plurality of multimodal query data from a corresponding knowledge base based on the matching knowledge source type and the multimodal query text, so as to construct a multimodal query data set; A partitioning input module, used for partitioning each of the multimodal query data in the multimodal query data set according to the preset modal type grouping and inputting the multimodal data association measurement model to determine a first association score between each of the multimodal query data; wherein the multimodal data association measurement model includes a plurality of network structures, each of the network structures is used to process data corresponding to different preset modal type groups; A fusion processing module, configured to perform fusion processing on the corresponding multimodal query data based on the comparison result of each of the first association scores and a first preset threshold, so as to obtain multimodal fusion feature information; The second input module is used to input the multimodal fusion feature information into a preset context-aware reasoning engine to obtain a reasoning result corresponding to the multimodal query text according to the engine output result.

10. A knowledge reasoning device for knowledge-intensive tasks, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a knowledge reasoning method for knowledge-intensive tasks as described in any one of claims 1 to 8.

Images