An artificial intelligence-based multi-modal data fusion processing method and system

Abstract

The application discloses a kind of based on artificial intelligence multimodal data fusion processing method and system, it is related to data processing technical field, including: respectively with each entity as starting point, according to target expansion depth is expanded, obtains the candidate subgraph of each entity;The semantic vector of each subgraph is obtained, according to and extraction;The path vector and path coverage of each path in are obtained, according to and extraction;With original question structured as Prompt after input pre-training large language model, obtain question and answer result;The application is favorable in guaranteeing the accuracy of specific field knowledge question and answer while, give consideration to semantic coverage, the balance of computing resource control and reasoning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal data fusion processing method and system based on artificial intelligence. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, pre-training large language models have made significant progress in natural language understanding and generation tasks. Intelligent question and answer systems based on language models have been widely applied in various professional fields, such as software engineering. Such models have strong semantic modeling and context generation capabilities, and can generate natural and complete answer texts according to natural language questions.

[0003] However, in scenarios with high density of professional knowledge and strong semantic logic requirements, language models still have typical problems such as knowledge illusion, missing reasoning chain, and uninterpretable results. Especially when the question involves structured semantic requirements such as causal chain tracing, entity relationship identification, and multi-hop reasoning, relying solely on corpus memory in model parameters often cannot provide reliable support, severely affecting the accuracy and credibility of question and answer.

[0004] To enhance the knowledge ability of large language models, structured knowledge graphs are introduced into the question and answer process to provide logical guidance and knowledge constraints by constructing entity and semantic relationship networks. This fusion approach has shown potential for improvement in multiple professional scenarios. However, in the actual fusion process, how to efficiently extract knowledge content highly related to the semantic of the question from large-scale knowledge graphs, and control the balance between knowledge expansion range and answer logic structure, remains a key challenge.

[0005] Taking the software engineering field as an example, developers often ask questions such as "How to fix system crash caused by null pointer exception". The relevant knowledge may be distributed in multiple modal data sources such as API documentation, code repository, dependency graph, and error stack. Without effective strategy guidance, a large number of redundant nodes may be introduced during graph expansion, affecting the efficiency and focus of the answer. If the expansion range is insufficient, key entity paths may be missed, causing knowledge chain breakage. In the subgraph selection and path judgment process, existing methods often rely on a fixed number or shallow similarity sorting, lacking dynamic semantic judgment and path value evaluation mechanism, making it difficult to ensure the professionalism, logic, and interpretability of the question and answer content.

[0006] Therefore, the existing question and answer system that integrates language models and knowledge graphs has not effectively solved the problem of balancing the accuracy of professional question and answer while considering the balance of semantic coverage, reasoning efficiency, and control of computing resources. SUMMARY

[0007] The application aims to provide an artificial intelligence-based multi-modal data fusion processing method and system to solve the problems in the background art.

[0008] To achieve the above-mentioned purpose, the application provides the following technical solutions.

[0009] In a first aspect, the application provides an artificial intelligence-based multi-modal data fusion processing method, comprising:

[0010] Obtaining a set of semantically related entities from the original question , and encoding the original question using a sentence vector model to obtain a question semantic vector , n is a positive integer;

[0011] In a preset professional knowledge graph , each entity in is taken as a starting point, and expansion is performed according to a target expansion depth to obtain a candidate subgraph of each entity , , i and m are positive integers;

[0012] Obtaining a subgraph semantic vector of each , and extracting a first knowledge subgraph from multiple according to and ;

[0013] Obtaining a path vector and a path coverage of each path in , and extracting a second knowledge subgraph from according to and ;

[0014] After structuring and the original question into a prompt, inputting them into a pre-trained large language model to obtain a question and answer result.

[0015] Further, the target expansion depth is obtained by the following method:

[0016] Taking each entity as a starting point, performing layer-by-layer breadth-first search in the professional knowledge graph to obtain a cumulative node number that has been searched under the current layer d, d is a positive integer;

[0017] Record the cumulative number of nodes found at the current level d. And the cumulative number of nodes compared to the d-1 level. ;

[0018] according to and Calculate the information gain ratio at the current layer number d. and will Compared with the preset information gain rate threshold Compare;

[0019] Wherein, the information gain rate The expression is: ;

[0020] like If the layer-by-layer breadth-first search is stopped, the current layer d is used as the target expansion depth. ;

[0021] like Then continue with a layer-by-layer breadth-first search until... At that time, the target expansion depth is obtained. .

[0022] Furthermore, the acquisition of each Subgraph semantic vector ,include:

[0023] extract Each path in the knowledge triple (h, r, t) corresponds to a knowledge triple, where h represents the head entity, r represents the relation, and t represents the tail entity.

[0024] The fine-tuned T5 / ChatGLM is used to convert each knowledge triple (h, r, t) into a natural language statement, and then the natural language statements corresponding to all knowledge triples are directly concatenated to obtain... Semantic description;

[0025] Using sentence vector models The semantic description is encoded as a subgraph semantic vector. ;

[0026] Repeat the above steps until each one is completed. The semantic description is encoded to obtain each Subgraph semantic vector .

[0027] Furthermore, the aforementioned according to and From multiple Extract the first knowledge subgraph ,include:

[0028] For each Subgraph semantic vector Calculate its relationship with the question semantic vector The similarity is obtained for each First similarity to the original problem ;

[0029] Wherein, the first similarity The calculation formula is as follows:

[0030] ;

[0031] For multiple first similarity Sort in descending order from multiple Select the Top-K Merge to form the first knowledge subgraph Where K is a positive integer.

[0032] Furthermore, the method for obtaining the K value in the Top-K is as follows:

[0033] Each According to the first similarity Sort in descending order and calculate adjacent items in turn. The decrease in similarity between The specific calculation formula is as follows: In the formula: For the i-th first similarity , For the (i+1)th first similarity ;

[0034] Calculate the amount of descent average and standard deviation ;

[0035] According to the average and standard deviation Set a threshold for the amount of descent. ;

[0036] Iterate through all the decreases in sequence until satisfied Stop when i+1=K, and obtain the value of K in Top-K.

[0037] Furthermore, the path coverage The method to obtain it is as follows:

[0038] extract The set of all entities along each path r is a positive integer;

[0039] According to and Calculate the path coverage , the formula is: ;

[0040] Repeat the above steps until the path coverage calculation of all paths is completed, and a plurality of path coverages .

[0041] Further, the according to and Extract the second knowledge subgraph from , including:

[0042] Calculate the path vector of each path in the extracted and the similarity between the problem semantic vector , to get the second similarity of each path and the original question ;

[0043] Wherein, the first similarity The formula is as follows:

[0044] ;

[0045] According to the path coverage and the second similarity Weighted calculation, get the path comprehensive score of each path , the formula is , wherein: is a weight factor greater than zero;

[0046] Compare with the preset path comprehensive score threshold ;

[0047] Eliminate the corresponding path of , and take the corresponding path of as the second knowledge subgraph output.

[0048] Secondly, the present application provides a kind of multi-modal data fusion processing system based on artificial intelligence based on the multi-modal data fusion processing method based on artificial intelligence described above is realized, including:

[0049] Acquisition module, for obtaining semantic related entity set from original question , and encode original question using sentence vector model, to obtain problem semantic vector , n is positive integer; ​

[0050] An expansion module is configured to expand each entity in the preset professional knowledge graph according to a target expansion depth i and m are positive integers;

[0051] A first extraction module is configured to obtain a subgraph semantic vector of each entity from a plurality of knowledge subgraphs ;

[0052] A second extraction module is configured to obtain a path vector and a path coverage of each path in the knowledge subgraph from the knowledge subgraph ;

[0053] A question and answer module is configured to input the Prompt obtained by structuring the original question and the original question into a pre-trained large language model to obtain a question and answer result.

[0054] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the above-mentioned artificial intelligence-based multi-modal data fusion processing method when executing the computer program.

[0055] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed to implement the above-mentioned artificial intelligence-based multi-modal data fusion processing method.

[0056] In the above technical solution, the present application provides the following technical effects and advantages:

[0057] The present application dynamically determines the expansion depth of the knowledge graph by setting the information gain rate threshold, avoids introducing redundant content due to excessive expansion, prevents missing key nodes due to insufficient expansion, reduces the consumption of computing resources, and thus significantly improves the efficiency and pertinence of graph retrieval, and provides a structural optimization basis for subsequent semantic modeling and path selection.

[0058] ​​​​​​​​​​​​​​​​After obtaining the candidate sub-graphs, the application constructs a sub-graph semantic vector and adaptively selects Top-K sub-graphs based on a cliff-like similarity drop, so as to ensure that the selected sub-graphs have high relevance in semantics with the original problem, reduce the interference of redundant knowledge, and improve the focusing and accuracy of the overall response of the question and answer system.

[0059] On the basis of the first knowledge sub-graph, the path vector and the path coverage are further combined for comprehensive scoring, so as to screen out the key path constituting the final answer reasoning chain, and thus the logical coherence and interpretability of the question and answer result are enhanced, and the professional credibility of the system in a high requirement scene is improved.

[0060] In summary, the application is beneficial to balancing the semantic coverage, the control of computing resources and the reasoning efficiency while ensuring the accuracy of knowledge question and answer in a specific field. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0062] Figure 1 A flow chart of the artificial intelligence multi-modal data fusion processing method of the present application;

[0063] Figure 2 A framework diagram of the artificial intelligence multi-modal data fusion processing system of the present application;

[0064] Figure 3 A structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0065] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these example implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The accompanying drawings are included to provide a further understanding of example implementations and are incorporated into and constitute a part of this application. The drawings are not intended to be restrictive in any way. Like reference numerals in the figures indicate like components, where appropriate.

[0066] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more example embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the example embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail in order to avoid obscuring aspects of the application.

[0067] Embodiment 1

[0068] As Figure 1 shown, the embodiment disclosed provides an artificial intelligence-based multi-modal data fusion processing method, comprising:

[0069] S101: obtaining a set of semantically related entities from an original question , and encoding the original question using a sentence vector model to obtain a question semantic vector , n is a positive integer;

[0070] It should be noted that the set of semantically related entities obtained from the original question is achieved by existing natural language processing methods, including but not limited to performing word segmentation, named entity recognition, or dependency syntax analysis, etc.; wherein the semantic vector of the original question is obtained using a pre-trained sentence vector encoding model (such as SimCSE, BERT, etc.);

[0071] Exemplarily, taking the field of software engineering as an example, assuming that the user gives the original question as "How to fix the system crash caused by the null pointer exception?", the system cuts it into meaningful word units by a word segmentation tool (such as jieba, HanLP, or LTP), obtaining { "null pointer exception", "cause", "system crash", "how", "fix"}. Then, based on a pre-built software engineering dictionary (from a graph or manually sorted) and with the help of a named entity recognition tool (such as HanLP, LTP, BERT-NER, etc.), the semantically valuable entities are identified from the above words as { "null pointer exception", "system crash", "fix"}. Finally, the original question is encoded as a whole by using a Chinese pre-trained sentence vector model SimCSE, obtaining a semantic vector , which is a set of fixed-length vectors (such as 128-dimensional or 768-dimensional floating-point arrays), is a real number field.

[0072] S102: screening the obtained semantic vector in a pre-set professional domain knowledge graph In, respectively Each entity in Starting from the target, expand the depth accordingly. Expand to obtain each entity candidate subgraph , i and m are positive integers;

[0073] It should be noted that the preset professional domain knowledge graph It is constructed based on predefined knowledge such as semantic rules of the target domain, relation type definitions, node attribute specifications, and graph hierarchy constraints; taking the professional domain knowledge graph in software engineering as an example, the professional domain knowledge graph... It can be built on open source code repositories (such as GitHub and Gitee), technical documents, fault case databases, API documents, development manuals, etc., extracting and organizing core concepts such as exception types, error stacks, repair methods, code patterns, design principles, and version dependencies, and combining them with call relationships, dependency relationships, causal relationships, and adaptation relationships to build a multi-level structured graph;

[0074] In implementation, the target expansion depth The method to obtain it is as follows:

[0075] With each entity Starting with professional domain knowledge graphs Perform a layer-by-layer breadth-first search to obtain the cumulative number of nodes searched at the current layer d. d is a positive integer;

[0076] It should be understood that breadth-first search is a method of searching nodes in a graph by expanding the search layer by layer. Its characteristics are: first, it visits the nodes closest to the starting point (i.e., "first-level neighbors"); then it visits the "second-level neighbors," then the third-level neighbors, and so on; only after all nodes in each level have been visited does it move to the next level. Therefore, for each entity... ,from By expanding the neighboring nodes layer by layer, the cumulative number of nodes in each layer can be obtained. It is worth noting that the cumulative number of nodes includes the number of all neighboring nodes in the previous layer.

[0077] Record the cumulative number of nodes found at the current level d. And the cumulative number of nodes compared to the d-1 level. ;

[0078] according to and Calculate the information gain ratio at the current layer number d. and will a preset information gain rate threshold comparing;

[0079] wherein the information gain rate is expressed as: ;

[0080] If , stop performing the layer-by-layer breadth-first search, and take the current layer d as the target expansion depth ;

[0081] If , continue to perform the layer-by-layer breadth-first search until , and obtain the target expansion depth ;

[0082] For example, based on the above assumptions, if the entity "null pointer exception" is taken as the starting point, and the layer-by-layer expansion is performed, it is found that at the first layer, , at the second layer, , at this time, the second layer is taken as the current layer d, and it is found after calculation that , assuming , since , the output d=2 is taken as the target expansion depth ;

[0083] S103: Obtain a subgraph semantic vector of each , and extract a first knowledge subgraph from a plurality of according to the subgraph semantic vector and the knowledge graph ; ; ;

[0084] In implementation, the obtaining of the subgraph semantic vector of each includes: extracting a knowledge triple (h, r, t) corresponding to each path in the

[0085] , wherein h represents a head entity, r represents a relationship, and t represents a tail entity; using the fine-tuned T5 / ChatGLM to convert each knowledge triple (h, r, t) into a natural language sentence, and directly splicing the natural language sentences corresponding to all knowledge triples to obtain a semantic description of the

[0086] ;

[0087] ​​It should be noted that pre-trained models such as T5 and ChatGLM are essentially text generation models based on the Transformer architecture. They have powerful context understanding and natural language generation capabilities. In order to make them suitable for specific domains (such as software engineering), they need to be trained through "fine-tuning" (which can be implemented based on public frameworks (such as HuggingFace Transformers) and combined with training corpora for specific domains). That is, a large number of triple-natural language pairs in the domain (for example, assuming that software engineering triples (such as {null pointer exception, leading to system crash}) and natural sentences ("null pointer exception may lead to system crash") are used as training pairs) as supervised training samples, and secondary training is performed on the basis of the original model. Specifically, this includes: obtaining training data containing triple-natural sentence pairs, using T5 / ChatGLM as the base model, taking triples as the input of the base model and natural sentences as the output, training the base model to obtain the fine-tuned T5 / ChatGLM;

[0088] Using sentence vector models The semantic description is encoded as a subgraph semantic vector. ;

[0089] Repeat the above steps until each one is completed. The semantic description is encoded to obtain each Subgraph semantic vector ;

[0090] In practice, the aforementioned... and From multiple Extract the first knowledge subgraph ,include:

[0091] For each Subgraph semantic vector Calculate its relationship with the question semantic vector The similarity is obtained for each First similarity to the original problem ;

[0092] Wherein, the first similarity The calculation formula is as follows:

[0093] ;

[0094] For multiple first similarity Sort in descending order from multiple Select the Top-K Merge to form the first knowledge subgraph Where K is a positive integer;

[0095] Specifically, the method for obtaining the K value in the Top-K is as follows:

[0096] Each According to the first similarity Sort in descending order and calculate adjacent items in turn. The decrease in similarity between The specific calculation formula is as follows: In the formula: For the i-th first similarity , For the (i+1)th first similarity ;

[0097] Calculate the amount of descent average and standard deviation ;

[0098] in, , In the formula: for Quantity;

[0099] According to the average and standard deviation Set a threshold for the amount of descent. ;

[0100] Iterate through all the decreases in sequence until satisfied Stop at the specified time, let i+1=K, and obtain the value of K in Top-K;

[0101] For example, suppose a candidate subgraph With question semantic vector The first similarity ranking results in descending order are as follows: S1 = [0.92, 0.91, 0.89, 0.88, 0.63, 0.60, 0.59]. After calculating the decrease in similarity between adjacent similarities, we find that: Δ1 = |0.92 - 0.91| = 0.01, Δ2 = |0.91 - 0.89| = 0.02, Δ3 = |0.89 - 0.88| = 0.01, Δ4 = |0.88 - 0.63| = 0.25, Δ5 = |0.63 - 0.60| = 0.03, Δ6 = |0.60 - 0.59| = 0.01. Therefore, the average value... = (0.01 + 0.02 + 0.01 + 0.25 + 0.03 + 0.01) / 6=0.07 (rounded to two decimal places). (Rounded to two decimal places), threshold for descent Since Δ4 = 0.25 > τ, it indicates that there is a similarity cliff between the 4th term and the 5th term, at this time, K is set as: K = 4 + 1 = 5, thus, the first 5 are selected into Top-K.

[0102] S104: Obtain the path vector of each path in and the path coverage , according to and extract the second knowledge subgraph from ; ;

[0103] It can be understood that: although is the relevant knowledge content of the original question, there are still too many paths in it, which need to be further screened out to find a few paths that are really useful to answer the user's question;

[0104] It should be noted that: the path vector of each path in is obtained according to the same logic as the path vector of each path in the subgraph semantic vector above, please refer to the relevant part above for details, which is based on converting the knowledge triple (h, r, t) in the graph structure into a natural language statement and further encoding it into a vector; but the difference is: is the unit of the entire candidate subgraph , all the triplets of the paths in it are spliced and encoded as a whole to obtain a set of global semantic representations; while the path vector is a single path unit, only the triplets on the path are extracted and converted into a statement as a semantic encoding input to obtain an independent representation vector of the path;

[0105] In implementation, the path coverage is obtained by the following method:

[0106] Extract all entity sets of each path in , r is a positive integer; Calculate the path coverage

[0107] according to and , and the calculation formula is: ; Repeat the above steps until the path coverage of all paths is calculated to obtain multiple path coverages

[0108] ;

[0109] ​In practice, the aforementioned... and from Extract the second knowledge subgraph ,include:

[0110] Calculate the extraction separately Path vector of each path With question semantic vector The similarity is used to obtain the second similarity between each path and the original problem. ;

[0111] Wherein, the first similarity The calculation formula is as follows:

[0112] ;

[0113] Based on path coverage Second similarity A weighted calculation is performed to obtain the overall path score for each path. Its calculation formula is In the formula: A weighting factor that is greater than zero;

[0114] Will Combined with the preset path comprehensive score threshold Compare;

[0115] Eliminate The corresponding path, and The corresponding path is used as the second knowledge subgraph Output;

[0116] S105: Will After the original question is structured into a Prompt, it is input into a pre-trained large language model to obtain the question-answering results.

[0117] The structure of the Prompt includes three parts: question background guidance, subgraph knowledge embedding, and answer target instruction, which aims to stimulate the large language model to reason and answer along the specified knowledge path;

[0118] For example, suppose the user's original question is "How to fix a system crash caused by a null pointer exception?", and The background knowledge in the question includes: 1. The null pointer exception is usually caused by a method call to an uninitialized object; 2. Uncaught exceptions can cause system crashes; 3. Common fixes include null value checks, using Optional wrapper objects, and try-catch blocks to catch exceptions; guided by the problem background + subgraph knowledge embedding + answer target instruction, you are an intelligent question and answer assistant in the field of software engineering; problem background: null pointer exception causes system crash, how to fix? Subgraph knowledge embedding: 1. The null pointer exception is usually caused by a method call to an uninitialized object; 2. Uncaught exceptions can cause system crashes; 3. Common fixes include null value checks, using Optional wrapper objects, and try-catch blocks to catch exceptions; task instruction: please provide professional and accurate repair suggestions based on the above knowledge.

[0119] It should be noted that the pre-trained large language model is an existing large language model, such as OpenAI's GPT series, Google's BERT, T5, and Meta's LLaMA, etc. Such pre-trained models have tens of billions to hundreds of billions of parameters, so they are called "large language models". However, these large language models can provide professional question and answer in various fields, but the pre-trained large language models rely on static training corpus and lack real-time updating ability for specific professional knowledge. Once the professional knowledge system changes, the model's answer result may still be based on outdated knowledge, affecting professional accuracy, and in the absence of external knowledge constraints, it is easy to "invent" seemingly reasonable but actually incorrect knowledge content. This illusionary output will directly affect the reliability of the results; and the knowledge graph itself lacks natural language processing and semantic understanding ability, making it difficult to directly respond to user free expression questions, and unable to generate natural language answers based on user problem backgrounds, especially difficult to organize smooth and logically complete answer content;

[0120] The present application realizes the intelligent question and answer effect with precise positioning, knowledge driving and strong explainability by extracting multi-granularity semantic subgraphs from the domain knowledge graph and constructing structured prompt words Prompt to guide the semantic reasoning path in the pre-trained language model.

[0121] Embodiment 2

[0122] As shown in Figure 2 , the part not detailed in the present embodiment is shown in embodiment 1, and the present embodiment discloses a kind of multi-modal data fusion processing systems based on artificial intelligence, comprising:

[0123] The acquisition module 201 is used to acquire a set of semantic related entities from the original question , and encode the original question using a sentence vector model to obtain a question semantic vector , n is a positive integer;

[0124] Extension module 202 is used to build a knowledge graph in a predefined professional field. In, respectively Each entity in Starting from the target, expand the depth accordingly. Expand to obtain each entity candidate subgraph , where i and m are positive integers;

[0125] The first extraction module 203 is used to obtain each Subgraph semantic vector ,according to and From multiple Extract the first knowledge subgraph ;

[0126] The second extraction module 204 is used to obtain... Path vector of each path and path coverage ,according to and from Extract the second knowledge subgraph ;

[0127] Question and answer module 205 is used to... After the original question is structured into a Prompt, it is input into a pre-trained large language model to obtain the question-answering results.

[0128] Example 3

[0129] Please see Figure 3 As shown, this embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements any of the artificial intelligence-based multimodal data fusion processing methods described above.

[0130] Since the electronic device described in this embodiment is the electronic device used to implement the AI-based multimodal data fusion processing method in the embodiments of this application, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the AI-based multimodal data fusion processing method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the AI-based multimodal data fusion processing method in the embodiments of this application falls within the scope of protection of this application.

[0131] Embodiment 4

[0132] The embodiment discloses a computer readable storage medium, comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the artificial intelligence based multi-modal data fusion processing method provided by any one of the above methods when executing the computer program.

[0133] The above formulas are all dimensionless numerical calculations, the formulas are obtained by collecting a large amount of data to simulate the latest real situation, and the preset parameters, weights and threshold values in the formulas are set by a person skilled in the art according to actual conditions.

[0134] The above embodiments can be realized wholly or partially by software, hardware, firmware or any other combination. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the above-mentioned processes or functions are wholly or partially generated according to the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired network or a wireless network. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center and the like containing one or more available medium sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD) or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0135] The above only describes some exemplary embodiments of the present application by way of illustration, and it is needless to say that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above drawings and descriptions are illustrative in nature and should not be understood as limiting the scope of protection of the claims of the present application.

Claims

1. An artificial intelligence-based multi-modal data fusion processing method, characterized in that, The application relates to an artificial intelligence multi-modal data fusion processing method. Obtaining a set of semantically relevant entities from a raw question , and encoding the raw question by using a sentence vector model to obtain a question semantic vector , n is a positive integer; In the pre-defined professional domain knowledge graph In, respectively Each entity in Starting from the target, expand the depth accordingly. Expand to obtain each entity candidate subgraph , where i and m are positive integers; Wherein, the target extension depth The acquisition method is as follows: With each entity Starting with professional domain knowledge graphs Perform a layer-by-layer breadth-first search to obtain the cumulative number of nodes searched at the current layer d. d is a positive integer; recorded at the current layer d the cumulative number of nodes searched and compared to the cumulative number of nodes of layer d-1 ; According to And Calculate the information gain rate under the current layer d And Compare with the preset information gain rate threshold Compare The information gain ratio is expressed as: The expression of the information gain ratio is: ; If , stop performing the layer-by-layer breadth-first search, and take the current layer d as the target expansion depth ; If , then continue to perform the breadth-first search layer by layer until , the target expansion depth is obtained. acquire each subgraph semantic vector of the subgraph , according to and extract a first knowledge subgraph from a plurality of ;​ acquiring path vector of each path and path coverage , according to and extracting a second knowledge sub-graph from ;​ Will After the original question is structured into a Prompt, it is input into a pre-trained large language model to obtain the question-answering results. 2.The artificial intelligence-based multi-modal data fusion processing method of claim 1, wherein, The acquisition of each subgraph semantic vector , comprising: Extract Each path corresponds to a knowledge triple (h, r, t), where h represents the head entity, r represents the relationship, and t represents the tail entity. The fine-tuned T5 / ChatGLM is used to convert each knowledge triple (h, r, t) into a natural language sentence, and the natural language sentences corresponding to all knowledge triples are directly spliced to obtain a semantic description of ; encoding the semantic description of the sentence into a subgraph semantic vector using a sentence vector model ;​ The above steps are repeated until the encoding of the semantic description of each is completed, resulting in a subgraph semantic vector for each . ​ 3.The artificial intelligence-based multi-modal data fusion processing method of claim 1, wherein, The according And Extracting a first knowledge sub-graph from a plurality of , comprising:​ For each subgraph semantic vector , the similarity between it and the question semantic vector is calculated, obtaining the first similarity of each subgraph semantic vector to the original question. The first similarity degree The calculation formula is as follows: ； a plurality of first similarities descending order, from the plurality of selecting Top-K from the plurality of merge to form a first knowledge sub-graph ; wherein K is a positive integer. 4.The artificial intelligence-based multi-modal data fusion processing method of claim 3, wherein, The K values in the Top-K are obtained as follows: Each of the first similarities is arranged in descending order, and the similarity drop between adjacent first similarities is calculated in turn, and the specific calculation formula is: ; in the formula: is the i-th first similarity , is the i+1-th first similarity ;​​ The average of the amount of descent The average of the amount of descent and the standard deviation ; According to the average value and the standard deviation , a drop threshold value is set; Iterate all descending quantities in turn , until the condition is met, stop, let i+1=K, and get the K values in Top-K. 5.The artificial intelligence-based multi-modal data fusion processing method of claim 1, wherein, The path coverage The acquisition method is as follows: extracting all entity sets of each path r is a positive integer; According to and Computing path coverage The formula is: ; The above steps are repeated until the path coverage calculation for all paths is completed, obtaining a plurality of path coverages . 6.The artificial intelligence-based multi-modal data fusion processing method of claim 5, wherein, The according And Extracting a second knowledge sub-graph from , comprising:​ Calculate the path vector of each path in the extracted paths respectively Calculate the similarity between the path vector of each path and the problem semantic vector , and obtain the second similarity between each path and the original problem ; The second similarity The calculation formula is as follows: ； According to path coverage and the second similarity Weighted calculation is performed to obtain a path comprehensive score of each path The calculation formula is In the formula: is a weight factor greater than zero; will be described below. comparing the predetermined path synthesis score threshold value comparison; culling the corresponding path of the corresponding path of output.

7. An artificial intelligence-based multi-modal data fusion processing system based on the artificial intelligence-based multi-modal data fusion processing method of any one of claims 1-6, characterized in that, The processor executes the computer program to realize the artificial intelligence multi-modal data fusion processing method in any one of claims 1-6. An acquisition module is configured to acquire a set of semantically related entities from the original question , and encode the original question by using a sentence vector model to obtain a question semantic vector , n is a positive integer; Extension modules are used to build knowledge graphs in predefined professional fields. In, respectively Each entity in Starting from the target, expand the depth accordingly. Expand to obtain each entity candidate subgraph , where i and m are positive integers; The first extraction module is configured to acquire each subgraph semantic vector from a plurality of and extract a first knowledge subgraph ;​ a second extraction module, configured to acquire a path vector of each path and a path coverage , according to and extract a second knowledge sub-graph from ;​ A question and answer module is configured to input the original question and the structured question into a pre-trained large language model to obtain a question and answer result. After the original question is structured as Prompt, the original question and the structured question are input into the pre-trained large language model to obtain the question and answer result.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed to realize the artificial intelligence multi-modal data fusion processing method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, ​

Images