Multi-modal data fusion processing method and system based on artificial intelligence

Abstract

The invention discloses a multi-modal data fusion processing method and system based on artificial intelligence, and relates to the technical field of data processing, and the method comprises the steps: taking each entity # imgabs0 # as a starting point, carrying out the extension according to a target extension depth # imgabs1 #, and obtaining a candidate subgraph # imgabs3 # of each entity # imgabs2 #; obtaining a sub-graph semantic vector # imgabs5 # of each # imgabs4 #, and extracting a # imgabs8 # according to the # imgabs6 # and the # imgabs7 #; the path vector # imgabs10 # and the path coverage degree # imgabs11 # of each path in the # imgabs9 # are obtained, and the # imgabs14 # is extracted according to the # imgabs12 # and the # imgabs13 #; structuring # imgabs15 # and an original question into Prompt, and inputting the Prompt into a pre-training large language model to obtain a question and answer result; according to the method, while the accuracy of knowledge questions and answers in the specific field is ensured, the balance of semantic coverage, computing resource control and reasoning efficiency is considered.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for multimodal data fusion processing based on artificial intelligence. Background Art

[0002] With the rapid development of artificial intelligence (AI), pre-trained large language models have made significant progress in natural language understanding and generation tasks. Intelligent question-answering systems based on language models have been widely used in various fields, such as software engineering. These models possess powerful semantic modeling and context generation capabilities, enabling them to generate relatively natural and complete responses to natural language questions.

[0003] However, in scenarios with high professional knowledge density and strong semantic logic requirements, language models still suffer from typical problems such as knowledge illusions, missing reasoning chains, and unexplainable results. Especially when questions involve structured semantic requirements such as causal chain tracing, entity relationship identification, and multi-hop reasoning, the corpus memory in the model parameters alone often cannot provide reliable support, seriously affecting the accuracy and credibility of the question and answer.

[0004] To enhance the knowledge capabilities of large language models, attempts are underway to incorporate structured knowledge graphs into the question-answering process. By constructing networks of entities and their semantic relationships, these networks provide logical guidance and knowledge constraints for question answering. This integration approach has demonstrated promising results in various professional scenarios. However, in practice, key challenges remain in efficiently extracting knowledge content highly relevant to the question semantics from large-scale knowledge graphs and balancing the scope of knowledge expansion with the logical structure of the answers.

[0005] Taking the field of software engineering as an example, developers often ask questions such as "How to fix a system crash caused by a null pointer exception?" The relevant knowledge may be distributed in multimodal data sources such as API documentation, code repositories, dependency graphs, error stacks, etc. In the process of graph expansion, if there is a lack of effective strategic guidance, it is very easy to introduce a large number of redundant nodes, affecting the efficiency and focus of answers; and if the expansion scope is insufficient, key entity paths may be missed, resulting in a knowledge chain break. In the process of subgraph screening and path judgment, existing methods are often based on fixed numbers or shallow similarity sorting, lacking dynamic semantic judgment and path value evaluation mechanisms, making it difficult to ensure the professionalism, logic and interpretability of the question and answer content.

[0006] Therefore, the existing question-answering system that integrates language models and knowledge graphs has not yet effectively solved the problem of how to balance semantic coverage, reasoning efficiency and computing resource control while ensuring the accuracy of professional questions and answers. Summary of the Invention

[0007] The purpose of the present invention is to provide a method and system for multimodal data fusion processing based on artificial intelligence to solve the problems in the above-mentioned background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solutions: In a first aspect, the present invention provides a method for multimodal data fusion processing based on artificial intelligence, comprising: Get a set of semantically related entities from the original question , and use the sentence vector model to encode the original question to obtain the question semantic vector , n is a positive integer; In the preset professional field knowledge graph In Each entity in As a starting point, expand the depth according to the target Expand and get each entity Candidate subgraph of , , i and m are positive integers; Get each Subgraph semantic vector of ,according to and From multiple Extract the first knowledge subgraph ; Get The path vector for each path in and path coverage ,according to and from Extract the second knowledge subgraph ; Will After the original question is structured into a prompt, it is input into the pre-trained large language model to obtain the question-answering result.

[0009] Furthermore, the target expansion depth The method to obtain is as follows: For each entity As a starting point, in the professional field knowledge map Perform breadth-first search layer by layer to get the cumulative number of nodes searched at the current layer d , d is a positive integer; Record the cumulative number of nodes searched at the current level d , and the cumulative number of nodes compared to the d-1 layer ; according to and Calculate the information gain rate under the current layer number d , and and the preset information gain rate threshold Compare; Among them, the information gain rate The expression is: ; like , then stop executing the layer-by-layer breadth-first search and use the current layer d as the target expansion depth ; like , then continue to perform breadth-first search layer by layer until When the target expansion depth is obtained .

[0010] Furthermore, the acquisition of each Subgraph semantic vector of ,include: extract The knowledge triple (h, r, t) corresponding to each path in , where h represents the head entity, r represents the relationship, and t represents the tail entity; Use the fine-tuned T5 / ChatGLM to convert each knowledge triple (h, r, t) into a natural sentence, and directly concatenate the natural sentences corresponding to all knowledge triples to obtain Semantic description of Use sentence vector model to The semantic description is encoded as a subgraph semantic vector ; Repeat the above steps until each The encoding of the semantic description of each Subgraph semantic vector of .

[0011] Furthermore, the basis and From multiple Extract the first knowledge subgraph ,include: For each Subgraph semantic vector of , calculate its semantic vector with the question The similarity of each First similarity to the original question ; Among them, the first similarity The calculation formula is as follows: ; For multiple first similarities Sort descending, from multiple Select the top-K Merge to form the first knowledge subgraph ; Where K is a positive integer.

[0012] Furthermore, the method for obtaining the K value in the Top-K is as follows: Each , by first similarity Arrange in descending order, calculate adjacent The decrease in similarity between , and its specific calculation formula is: Where: is the first similarity of the i-th , is the first similarity of the i+1th ; Calculate the amount of descent Average value and standard deviation ; According to the average and standard deviation , set the drop threshold ; Traverse all the descending amounts in turn , until satisfied Stop when i+1=K and get the K value in Top-K.

[0013] Furthermore, the path coverage The method to obtain is as follows: extract The set of all entities for each path in , r is a positive integer; according to and Calculating path coverage , and its calculation formula is: ; Repeat the above steps until the path coverage calculation of all paths is completed and multiple path coverages are obtained. .

[0014] Furthermore, the basis and from Extract the second knowledge subgraph ,include: Calculate and extract separately The path vector for each path in and question semantic vector The similarity between each path and the original problem is obtained. ; Among them, the first similarity The calculation formula is as follows: ; According to the path coverage and the second similarity Perform weighted calculation to obtain the comprehensive score of each path , and its calculation formula is , where: is a weight factor greater than zero; Will Compared with the preset path comprehensive score threshold Compare; culling The corresponding path of The corresponding path is the second knowledge subgraph Output.

[0015] In a second aspect, the present invention provides an artificial intelligence-based multimodal data fusion processing system, which is implemented based on the above-mentioned artificial intelligence-based multimodal data fusion processing method, including: The acquisition module is used to obtain a set of semantically related entities from the original question , and use the sentence vector model to encode the original question to obtain the question semantic vector , n is a positive integer; Extension module, used to build knowledge graphs in preset professional fields In Each entity in As a starting point, expand the depth according to the target Expand and get each entity Candidate subgraph of , , i and m are positive integers; The first extraction module is used to obtain each Subgraph semantic vector of ,according to and From multiple Extract the first knowledge subgraph ; The second extraction module is used to obtain The path vector for each path in and path coverage ,according to and from Extract the second knowledge subgraph ; Question and answer module, used to After the original question is structured into a prompt, it is input into the pre-trained large language model to obtain the question-answering result.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements any of the above-mentioned artificial intelligence-based multimodal data fusion processing methods.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements any of the above-mentioned artificial intelligence-based multimodal data fusion processing methods.

[0018] In the above technical solution, the technical effects and advantages provided by the present invention are: The present invention dynamically determines the expansion depth of the knowledge graph by setting an information gain rate threshold, avoiding the introduction of redundant content due to excessive expansion, and preventing the omission of key nodes due to insufficient expansion, thereby reducing computing resource consumption. This significantly improves the efficiency and pertinence of graph retrieval, and provides a structural optimization basis for subsequent semantic modeling and path screening.

[0019] After obtaining the candidate subgraphs, the present invention constructs subgraph semantic vectors and adaptively selects Top-K subgraphs based on the cliff-like similarity drop judgment, thereby ensuring that the selected subgraphs are highly semantically relevant to the original question, reducing redundant knowledge interference, and improving the focus and accuracy of the overall response of the question-answering system.

[0020] Based on the first knowledge subgraph, we further combine the path vector and path coverage for comprehensive scoring to screen out the key paths that constitute the final answer reasoning chain, thereby enhancing the logical coherence and interpretability of the question-and-answer results and improving the professional credibility of the system in high-demand scenarios.

[0021] In summary, the present invention is beneficial for ensuring the accuracy of knowledge questions and answers in specific fields while taking into account the balance between semantic coverage, computing resource control and reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0023] Figure 1 This is a flow chart of an artificial intelligence-based multimodal data fusion processing method of the present invention; Figure 2 This is a framework diagram of an artificial intelligence-based multimodal data fusion processing system of the present invention; Figure 3 This is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of this disclosure will be more comprehensive and complete, and will fully convey the concepts of the example embodiments to those skilled in the art. The accompanying drawings are merely schematic illustrations of the disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures indicate identical or similar parts, and thus any repetitive description thereof will be omitted.

[0025] In addition, the described features, structures or characteristics can be combined in one or more example embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the example embodiments disclosed in this application. However, those skilled in the art will appreciate that the technical solutions disclosed in this application can be practiced while omitting one or more of the specific details, or other methods, components, steps, etc. can be adopted. In other cases, well-known structures, methods, implementations or operations are not shown or described in detail to avoid obscuring the various aspects disclosed in this application.

[0026] Example 1 like Figure 1 As shown, this embodiment discloses a method for multimodal data fusion processing based on artificial intelligence, including: S101: Obtain a set of semantically related entities from the original question , and use the sentence vector model to encode the original question to obtain the question semantic vector , n is a positive integer; It should be noted that: the semantically related entity set obtained from the original question , implemented through existing natural language processing methods, including but not limited to performing operations such as word segmentation, named entity recognition or dependency parsing; wherein the semantic vector of the original question The acquisition of is achieved using a pre-trained sentence vector encoding model (such as SimCSE, BERT, etc.); For example, taking the field of software engineering as an example, assuming that the user gives the original question "How to fix the system crash caused by the null pointer exception?", the system uses word segmentation tools (such as jieba, HanLP or LTP) to split it into meaningful word units and obtain {"null pointer exception", "cause", "system crash", "how", "repair"}. Then, based on the pre-built software engineering dictionary (from the atlas or manual compilation) and with the help of named entity recognition tools (such as HanLP, LTP, BERT-NER, etc.), entities with semantic value are identified from the above words as {"null pointer exception", "system crash", "repair"}. Finally, the original question is encoded as a whole by using the Chinese pre-trained sentence vector model SimCSE to obtain the semantic vector for subsequent knowledge graph similarity screening. , is a set of fixed-length vectors (such as 128-dimensional or 768-dimensional floating-point arrays), The field of real numbers.

[0027] S102: In the preset professional field knowledge map In Each entity in As a starting point, expand the depth according to the target Expand and get each entity Candidate subgraph of , , i and m are positive integers; It should be noted that: the preset professional field knowledge graph , which is constructed based on predefined knowledge such as semantic rules, relationship type definitions, node attribute specifications, and graph hierarchy constraints in the target domain; taking the professional domain knowledge graph in the direction of software engineering as an example, the professional domain knowledge graph , can be built based on open source code libraries (such as GitHub, Gitee), technical documents, fault case databases, API documents, development manuals, etc., extracting and organizing core concepts such as exception types, error stacks, repair methods, code patterns, design principles, version dependencies, etc., and combining the call relationships, dependencies, causal relationships, adaptation relationships, etc. between them to build a multi-level structured map; In implementation, the target expansion depth The method to obtain is as follows: For each entity As a starting point, in the professional field knowledge map Perform breadth-first search layer by layer to get the cumulative number of nodes searched at the current layer d , d is a positive integer; It should be understood that breadth-first search is a search method that expands nodes layer by layer in a graph. Its characteristic is that it first visits the nodes closest to the starting point (i.e., the "first-layer neighbors"); then visits the "second-layer neighbors", then the third layer, and so on; when all the nodes in each layer are visited, it enters the next layer. Therefore, for each entity ,from By expanding neighbor nodes layer by layer, we can get the cumulative number of nodes in each layer. It is worth noting that the cumulative number of nodes includes the number of all neighbor nodes in the previous layer. Record the cumulative number of nodes searched at the current level d , and the cumulative number of nodes compared to the d-1 layer ; according to and Calculate the information gain rate under the current layer number d , and and the preset information gain rate threshold Compare; Among them, the information gain rate The expression is: ; like , then stop executing the layer-by-layer breadth-first search and use the current layer d as the target expansion depth ; like , then continue to perform breadth-first search layer by layer until When the target expansion depth is obtained ; For example, following the above assumptions, if we start with the entity "Null Pointer Exception" and expand layer by layer, we find that at the first layer, , at the second level, At this time, taking the second layer as the current layer number d, we find that , assuming ,because , then output d=2 as the target expansion depth ; S103: Get each Subgraph semantic vector of ,according to and From multiple Extract the first knowledge subgraph ; In implementation, the acquisition of each Subgraph semantic vector of ,include: extract The knowledge triple (h, r, t) corresponding to each path in , where h represents the head entity, r represents the relationship, and t represents the tail entity; Use the fine-tuned T5 / ChatGLM to convert each knowledge triple (h, r, t) into a natural sentence, and directly concatenate the natural sentences corresponding to all knowledge triples to obtain Semantic description of It should be noted that pre-trained models such as T5 and ChatGLM are essentially text generation models based on the Transformer architecture. They have strong contextual understanding and natural language generation capabilities. To make them suitable for specific fields (such as software engineering), they need to be "fine-tuned" (which can be implemented based on public frameworks (such as HuggingFace Transformers) and constructed in combination with training corpora for specific fields). That is, using a large number of triple-natural language pairs in the field (for example, assuming that software engineering triples (such as {Null pointer exception, causes, system crash}) and natural sentences ("Null pointer exception may cause system crash") are used as training pairs) as supervised training samples, secondary training is performed on the original model. Specifically, the training data containing triple-natural sentence pairs is obtained, and T5 / ChatGLM is used as the base model. The triples are used as the input of the base model and the natural sentences are used as the output. The base model is trained to obtain a fine-tuned T5 / ChatGLM. Use sentence vector model to The semantic description is encoded as a subgraph semantic vector ; Repeat the above steps until each The encoding of the semantic description of each Subgraph semantic vector of ; In implementation, the and From multiple Extract the first knowledge subgraph ,include: For each Subgraph semantic vector of , calculate its semantic vector with the question The similarity of each First similarity to the original question ; Among them, the first similarity The calculation formula is as follows: ; For multiple first similarities Sort descending, from multiple Select the top-K Merge to form the first knowledge subgraph ; Wherein, K is a positive integer; Specifically, the method for obtaining the K value in the Top-K is as follows: Each , by the first similarity Arrange in descending order, calculate adjacent The decrease in similarity between , and its specific calculation formula is: Where: is the first similarity of the i-th , is the first similarity of the i+1th ; Calculate the amount of descent Average value and standard deviation ; in, , , where: for the number of According to the average and standard deviation , set the drop threshold ; Traverse all the descending amounts in turn , until satisfied Stop when i+1=K, and get the K value in Top-K; For example, assuming that the candidate subgraph and question semantic vector The first similarity descending sorting result is as follows, S1 = [0.92, 0.91, 0.89, 0.88, 0.63, 0.60, 0.59]. After calculating the decrease between adjacent similarities in sequence, it is found that: Δ1 = |0.92- 0.91| = 0.01, Δ2 = |0.91- 0.89| = 0.02, Δ3 = |0.89- 0.88| = 0.01, Δ4 = |0.88 - 0.63| = 0.25, Δ5 = |0.63- 0.60| = 0.03, Δ6 = |0.60 - 0.59| = 0.01, then: average = (0.01 + 0.02 + 0.01 + 0.25 + 0.03 + 0.01) / 6=0.07 (round to the nearest two decimal places). (rounded to the nearest two decimal places), drop threshold , since Δ4 = 0.25 > τ, it means that there is a cliff-like drop in similarity from the 4th item to the 5th item. At this time, K is set to: K = 4 + 1 = 5. Therefore, the first 5 Selected into Top-K.

[0028] S104: Acquisition The path vector for each path in and path coverage ,according to and from Extract the second knowledge subgraph ; It is understandable that: although This is knowledge content related to the original question, but there are still too many paths, and we need to further filter out the few paths that are truly useful for answering the user's question; It should be noted that: The path vector for each path in The acquisition logic is the same as the above subgraph semantic vector The acquisition logic is consistent with that of , which is detailed in the relevant section above. Both are based on converting the knowledge triples (h, r, t) in the graph structure into natural language sentences and further encoding them into vectors; however, the differences are: The entire candidate subgraph As a unit, the triple descriptions of all paths are concatenated and encoded as a whole to obtain a set of global semantic representations; and the path vector It takes a single path as a unit, extracts only the triples on the path and converts them into sentences as semantic encoding input to obtain an independent representation vector for the path; In implementation, the path coverage The method to obtain is as follows: extract The set of all entities for each path in , r is a positive integer; according to and Calculating path coverage , and its calculation formula is: ; Repeat the above steps until the path coverage calculation of all paths is completed and multiple path coverages are obtained. ; In implementation, the and from Extract the second knowledge subgraph ,include: Calculate and extract separately The path vector for each path in and question semantic vector The similarity between each path and the original problem is obtained. ; Among them, the first similarity The calculation formula is as follows: ; According to the path coverage and the second similarity Perform weighted calculation to obtain the comprehensive score of each path , and its calculation formula is , where: is a weight factor greater than zero; Will Compared with the preset path comprehensive score threshold Compare; culling The corresponding path of The corresponding path is the second knowledge subgraph Output; S105: After the original question is structured into a prompt, it is input into the pre-trained large language model to obtain the question-answering result; The Prompt structure consists of three parts: question context guidance, subgraph knowledge embedding, and answer target instructions, aiming to stimulate the large language model to reason and answer along a specified knowledge path. For example, assume that the user's original question is "How to fix the system crash caused by Null Pointer Exception?", and The background knowledge includes: 1. Null pointer exceptions are usually caused by calling methods on uninitialized objects; 2. Uncaught exceptions may cause system crashes; 3. Common repair methods include null value judgment and using Optional to wrap objects in try-catch blocks to catch exceptions; through problem background guidance + sub-graph knowledge embedding + answering target instructions, the following is formed: "You are an intelligent question-and-answer assistant in the field of software engineering; Problem background: Null pointer exceptions cause system crashes, how to fix it? Sub-graph knowledge embedding: 1. Null pointer exceptions are usually caused by calling methods on uninitialized objects; 2. Uncaught exceptions may cause system crashes; 3. Common repair methods include null value judgment, using Optional to wrap objects, and trying-catch blocks to catch exceptions; Task instructions: Based on the above knowledge, please provide professional and accurate repair suggestions."

[0029] It should be noted that the pre-trained large language model is an existing large language model, such as OpenAI's GPT series, Google's BERT, T5, and Meta's LLaMA. This type of pre-trained model has billions to hundreds of billions of parameters, so it is called a "large language model". However, although these large language models can provide professional questions and answers in various fields, the pre-trained large language model relies on static training corpus and lacks the ability to update knowledge in specific professional fields in real time. Once the professional knowledge system changes, the model's answer results may still be based on outdated knowledge, affecting professional accuracy, and in the absence of external knowledge constraints, it is easy to "fabricate" knowledge content that seems reasonable but is actually wrong. This hallucinatory output will directly affect the reliability of the results; and the knowledge graph itself lacks natural language processing and semantic understanding capabilities. It is difficult to directly respond to questions freely expressed by users, and it is impossible to generate natural language answers based on the context of the user's questions. It is especially difficult to organize smooth and logically complete answer content; The present invention extracts multi-granularity semantic subgraphs from the domain knowledge graph and constructs structured prompts to guide the semantic reasoning path in the pre-trained language model, thereby achieving an intelligent question-answering effect with precise positioning, knowledge-driven, and strong interpretability.

[0030] Example 2 like Figure 2 As shown, the parts not described in detail in this embodiment are as shown in Example 1. This embodiment discloses a multimodal data fusion processing system based on artificial intelligence, including: Acquisition module 201 is used to obtain a set of semantically related entities from the original question , and use the sentence vector model to encode the original question to obtain the question semantic vector , n is a positive integer; Extension module 202 is used to expand the knowledge graph in the preset professional field In Each entity in As a starting point, expand the depth according to the target Expand and get each entity Candidate subgraph of , , i and m are positive integers; The first extraction module 203 is used to obtain each Subgraph semantic vector of ,according to and From multiple Extract the first knowledge subgraph ; The second extraction module 204 is used to obtain The path vector for each path in and path coverage ,according to and from Extract the second knowledge subgraph ; Question and answer module 205 is used to After the original question is structured into a prompt, it is input into the pre-trained large language model to obtain the question-answering result.

[0031] Example 3 See also Figure 3 As shown, this embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements any one of the artificial intelligence-based multimodal data fusion processing methods provided by the above methods.

[0032] Since the electronic device described in this embodiment is an electronic device used to implement the artificial intelligence-based multimodal data fusion processing method in the embodiment of this application, based on the artificial intelligence-based multimodal data fusion processing method described in the embodiment of this application, those skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as those skilled in the art implement the electronic device used in the artificial intelligence-based multimodal data fusion processing method in the embodiment of this application, it falls within the scope of protection of this application.

[0033] Example 4 This embodiment discloses a computer-readable storage medium, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements any one of the artificial intelligence-based multimodal data fusion processing methods provided by the above methods.

[0034] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters, weights and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0035] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0036] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A multimodal data fusion processing method based on artificial intelligence, characterized in that: include: Get a set of semantically related entities from the original question , and use the sentence vector model to encode the original question to obtain the question semantic vector , n is a positive integer; In the preset professional field knowledge graph In Each entity in As a starting point, expand the depth according to the target Expand and get each entity Candidate subgraph of , , i and m are positive integers; Get each Subgraph semantic vector of ,according to and From multiple Extract the first knowledge subgraph ; Get The path vector for each path in and path coverage ,according to and from Extract the second knowledge subgraph ; Will After the original question is structured into a prompt, it is input into the pre-trained large language model to obtain the question-answering result.

2. The method for multimodal data fusion processing based on artificial intelligence according to claim 1, characterized in that: The target extension depth The method to obtain is as follows: For each entity As a starting point, in the professional field knowledge map Perform breadth-first search layer by layer to get the cumulative number of nodes searched at the current layer d , d is a positive integer; Record the cumulative number of nodes searched at the current level d , and the cumulative number of nodes compared to the d-1 layer ; according to and Calculate the information gain rate under the current layer number d , and and the preset information gain rate threshold Compare; Among them, the information gain rate The expression is: ; like , then stop executing the layer-by-layer breadth-first search and use the current layer d as the target expansion depth ; like , then continue to perform breadth-first search layer by layer until When the target expansion depth is obtained .

3. The method for multimodal data fusion processing based on artificial intelligence according to claim 1, characterized in that: The acquisition of each Subgraph semantic vector of ,include: extract The knowledge triple (h, r, t) corresponding to each path in , where h represents the head entity, r represents the relationship, and t represents the tail entity; Use the fine-tuned T5 / ChatGLM to convert each knowledge triple (h, r, t) into a natural sentence, and directly concatenate the natural sentences corresponding to all knowledge triples to obtain Semantic description of Use sentence vector model to The semantic description is encoded as a subgraph semantic vector ; Repeat the above steps until each The encoding of the semantic description of each Subgraph semantic vector of .

4. The method for multimodal data fusion processing based on artificial intelligence according to claim 1, characterized in that: The basis and From multiple Extract the first knowledge subgraph ,include: For each Subgraph semantic vector of , calculate its semantic vector with the question The similarity of each First similarity to the original question ; Among them, the first similarity The calculation formula is as follows: ； For multiple first similarities Sort descending, from multiple Select the top-K Merge to form the first knowledge subgraph ; Where K is a positive integer.

5. The artificial intelligence-based multimodal data fusion processing method according to claim 4, characterized in that: The method for obtaining the K value in Top-K is as follows: Each , by first similarity Arrange in descending order, calculate adjacent The decrease in similarity between , and its specific calculation formula is: Where: is the first similarity of the i-th , is the first similarity of the i+1th ; Calculate the amount of descent Average value and standard deviation ; According to the average and standard deviation , set the drop threshold ; Traverse all the descending amounts in turn , until satisfied Stop when i+1=K and get the K value in Top-K.

6. The artificial intelligence-based multimodal data fusion processing method according to claim 1, characterized in that: The path coverage The method to obtain is as follows: extract The set of all entities for each path in , r is a positive integer; according to and Calculating path coverage , and its calculation formula is: ; Repeat the above steps until the path coverage calculation of all paths is completed and multiple path coverages are obtained. .

7. The artificial intelligence-based multimodal data fusion processing method according to claim 6, characterized in that: The basis and from Extract the second knowledge subgraph ,include: Calculate and extract separately The path vector for each path in and question semantic vector The similarity between each path and the original problem is obtained. ; Among them, the first similarity The calculation formula is as follows: ； According to the path coverage and the second similarity Perform weighted calculation to obtain the comprehensive score of each path , and its calculation formula is , where: is a weight factor greater than zero; Will Compared with the preset path comprehensive score threshold Compare; culling The corresponding path of The corresponding path is the second knowledge subgraph Output.

8. An artificial intelligence-based multimodal data fusion processing system, implemented based on the artificial intelligence-based multimodal data fusion processing method according to any one of claims 1 to 7, characterized in that: include: The acquisition module is used to obtain a set of semantically related entities from the original question , and use the sentence vector model to encode the original question to obtain the question semantic vector , n is a positive integer; Extension module, used to build knowledge graphs in preset professional fields In Each entity in As a starting point, expand the depth according to the target Expand and get each entity Candidate subgraph of , , i and m are positive integers; The first extraction module is used to obtain each Subgraph semantic vector of ,according to and From multiple Extract the first knowledge subgraph ; The second extraction module is used to obtain The path vector for each path in and path coverage ,according to and from Extract the second knowledge subgraph ; Question and answer module, used to After the original question is structured into a prompt, it is input into the pre-trained large language model to obtain the question-answering result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the artificial intelligence-based multimodal data fusion processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the artificial intelligence-based multimodal data fusion processing method according to any one of claims 1 to 7.

Images