Multi-mode intelligent question answering method, system and device

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a multi-modal intelligent question answering method, system and device, and the method comprises the steps: obtaining multi-modal data; preprocessing the multi-modal data, extracting corresponding feature representations from the preprocessed structured multi-modal data, and fusing the feature representations to obtain fused feature representations; finding entities and relationships related to the fused feature representation in a pre-constructed knowledge graph, and forming an entity set and a relationship set; adjusting the weight to obtain an entity set and a relation set after the weight is adjusted; screening all possible alternative answers from a pre-constructed knowledge graph, and constructing a guide path for each alternative answer by using an adaptive path exploration method; all the guiding paths are evaluated, optimized and distributed with weights, the guiding path with the highest weight is selected, and the alternative answer pointed by the guiding path with the highest weight serves as the retrieval answer corresponding to the user question.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multimodal intelligent question-answering method, system and device. Background Art

[0002] Human-computer interaction is a key technology for communication between humans and computers in the information age and has received widespread attention. Among them, the question-answering system is a typical interaction method that enables computers to communicate with humans through natural language and has become an important research direction in the field of artificial intelligence.

[0003] Research on traditional question-answering systems mostly focuses on a single modality such as text or images. However, this approach is difficult to integrate multiple sources of information and is prone to deviations in question understanding and natural language generation. In addition, it is difficult to accurately analyze the context in which the user inputs the question. In contrast, a multimodal question-answering system is closer to reality. By integrating information from multiple modalities, it can more comprehensively analyze user questions and contextual states, thereby providing more accurate and satisfactory answers. To this end, a multimodal intelligent question-answering system is proposed. Summary of the invention

[0004] In order to solve the problems raised in the above background technology, the present invention provides the following technical solutions:

[0005] A multimodal intelligent question answering method is provided, the method comprising:

[0006] Step S1, obtaining multimodal data; the multimodal data is question data input by the user;

[0007] Wherein, the multimodal data includes: text data, image data, audio data and video data;

[0008] Step S2, preprocessing the multimodal data to obtain preprocessed structured multimodal data;

[0009] Step S3, extracting corresponding feature representations from the preprocessed structured multimodal data and fusing them to obtain fused feature representations;

[0010] Step S4: in the pre-constructed knowledge graph, find entities and relationships related to the fused feature representation, and form an entity set and a relationship set;

[0011] Step S5: adjusting the weights of the elements in the entity set and the relationship set according to the context to obtain the entity set and the relationship set after the weights are adjusted;

[0012] Step S6: According to the entity set and relationship set after adjusting the weights, all possible alternative answers are screened from the pre-built knowledge graph, and a guiding path is constructed for each alternative answer using an adaptive path exploration method;

[0013] Step S7: Evaluate and optimize all guidance paths and assign weights to them, select the guidance path with the highest weight, and use the alternative answer pointed to by the guidance path with the highest weight as the search answer corresponding to the user's question.

[0014] Preferably, the specific steps of preprocessing the multimodal data in step S2 include:

[0015] Step S21, preprocessing the text data by word segmentation, noise removal, part-of-speech tagging, syntactic analysis, and embedding representation to obtain preprocessed structured text data;

[0016] Step S22, performing preprocessing of resizing, normalization, and data enhancement on the image data to obtain preprocessed structured image data;

[0017] Step S23, preprocessing the audio data by noise reduction and slicing to obtain preprocessed structured audio data;

[0018] Step S24, preprocessing the video data by frame extraction, motion detection, and time segmentation to obtain preprocessed structured video data;

[0019] Preferably, the specific steps of extracting corresponding feature representations from the preprocessed structured multimodal data and fusing them in step S3 include:

[0020] Step S31, using natural language processing technology to extract text feature representation from the pre-processed structured text data;

[0021] Step S32: using convolutional neural network technology to extract image feature representation from the preprocessed structured image data;

[0022] Step S33, extracting audio feature representation, including using a deep learning model to extract frequency domain and time domain feature representations from the preprocessed structured audio data;

[0023] Step S34, extracting video feature representation, including using a 3D convolutional neural network to extract spatiotemporal feature representation from the preprocessed structured video data, using a long short-term memory network modeling to extract time series feature representation, and using an optical flow method to analyze and extract motion trajectory feature representation;

[0024] Step S35: Use weighted fusion technology to map feature representations of different modalities to the same feature space, and use the attention mechanism to align the time axis of video data and audio data to ensure that all feature representations can match each other at the same time point.

[0025] Step S36: weighted fusion is adopted to assign different weights according to the reliability of each modal model to obtain the fused feature representation.

[0026] Preferably, the pre-constructed knowledge graph of step S4 is constructed based on an existing TCM literature library, and the construction method thereof is:

[0027] Step S41, collecting data from an existing TCM literature database;

[0028] Step S42, preprocessing the collected data, including: removing noise, standardizing terminology, and unifying data format;

[0029] Step S43: Use graph database technology to construct a TCM knowledge graph to associate TCM concepts, symptoms, drugs, prescriptions, therapies, etc.

[0030] Preferably, the specific steps of adjusting the weights of elements in the entity set and the relationship set according to the context in step S5 include:

[0031] Step S51: Perform dialogue management and design dialogue flow to enable multiple rounds of dialogue between the system and the user;

[0032] Step S52: extract key information from multiple rounds of historical conversations, and perform semantic analysis based on the key information to generate contextual semantic representation;

[0033] Step S53: Based on the contextual semantic representation, evaluate the relevance of the entity to the current question, and adjust the weight value of each entity in the entity set through a weight distribution formula or an attention mechanism;

[0034] Preferably, the specific steps of step S6 include:

[0035] Step S61: Filter out all possible candidate answers from the pre-built knowledge graph according to the entity set and relationship set after adjusting the weights;

[0036] Step S62: formulating a path search strategy according to the number of all possible candidate answers screened out;

[0037] Step S63: dynamically adjust the path search strategy using an adaptive path exploration method to obtain an adjusted path search strategy. The path search strategy is dynamically adjusted according to the context information of the current question and the relevance of the alternative answers. If some paths are significantly better than other paths, these paths are preferentially explored through weighted adjustment.

[0038] Step S64: Using the entity set and relationship set with adjusted weights as the search starting point, in the pre-built knowledge graph, execute the adjusted path search strategy to find all possible paths leading to the final answer.

[0039] Preferably, a multimodal intelligent question-answering system is provided, comprising:

[0040] An input module, used to obtain multimodal data input by users;

[0041] A preprocessing module, used for preprocessing multimodal data to obtain structured multimodal data;

[0042] A data fusion module is used to extract corresponding feature representations from structured multimodal data and fuse them to obtain fused feature representations;

[0043] A graph query module, used to find entities and relationships related to the fused feature representation in a pre-built knowledge graph, and to form an entity set and a relationship set;

[0044] A weight adjustment module is used to adjust the weights of elements in the entity set and the relationship set according to the context to obtain the entity set and the relationship set after the weight adjustment;

[0045] The path building module selects all possible candidate answers from the pre-built knowledge graph based on the entity set and relationship set after adjusting the weights, and uses the adaptive path exploration method to build a guiding path for each candidate answer;

[0046] The path evaluation module evaluates and optimizes all guiding paths and assigns weights to them, selects the guiding path with the highest weight, and uses the alternative answer pointed to by the guiding path with the highest weight as the retrieval answer corresponding to the user's question.

[0047] Preferably, a multimodal intelligent question-answering device is provided, comprising: a storage medium for storing programs and pre-created knowledge graphs, wherein the programs are used to implement all the above methods; and a processor for executing corresponding programs according to user input.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. Construct a TCM knowledge graph covering etiology, pathogenesis, syndromes, prescriptions and medicines, and use deep learning technology to understand, reason and apply knowledge.

[0050] 2. Integrate natural language processing technology to achieve natural human-computer interaction, provide consultation, and dynamically adjust treatment plans based on patient feedback and disease progression data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0052] Figure 1 Schematic diagram of the process of the multimodal intelligent question answering method of the present invention;

[0053] Figure 2 It is a schematic diagram of the structure of one of the single chip microcomputers of the present invention;

[0054] Figure 3 It is a schematic diagram of the structure of one of the single chip microcomputers of the present invention; DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] Depend on Figure 1 It is given that an embodiment of the present invention provides a multimodal intelligent question answering method, the method comprising:

[0057] Step S1, obtaining multimodal data; the multimodal data is question data input by the user;

[0058] Among them, multimodal data includes: text data, image data, audio data and video data;

[0059] Step S2: preprocessing the multimodal data to obtain preprocessed structured multimodal data;

[0060] Step S3, extracting corresponding feature representations from the preprocessed structured multimodal data and fusing them to obtain fused feature representations;

[0061] Step S4: In the pre-constructed knowledge graph, find entities and relationships related to the fused feature representation, and form an entity set and a relationship set;

[0062] Step S5: adjusting the weights of the elements in the entity set and the relationship set according to the context to obtain the entity set and the relationship set after the weights are adjusted;

[0063] Step S6: According to the entity set and relationship set after adjusting the weights, all possible alternative answers are screened from the pre-built knowledge graph, and a guiding path is constructed for each alternative answer using an adaptive path exploration method;

[0064] Step S7: Evaluate and optimize all guidance paths and assign weights to them, select the guidance path with the highest weight, and use the alternative answer pointed to by the guidance path with the highest weight as the search answer corresponding to the user's question.

[0065] 2. The method according to claim 1, characterized in that: the specific steps of preprocessing the multimodal data in step S2 include:

[0066] Step S21, preprocessing the text data by word segmentation, noise removal, part-of-speech tagging, syntactic analysis, and embedding representation to obtain preprocessed structured text data;

[0067] Among them, the tools used for embedding representation are Word2Vec, GloVe, and FastText;

[0068] Step S22, performing preprocessing of resizing, normalization, and data enhancement on the image data to obtain preprocessed structured image data;

[0069] Among them, the tools used for data enhancement are OpenCV and PyTorch;

[0070] Step S23, preprocessing the audio data by noise reduction and slicing to obtain preprocessed structured audio data;

[0071] Among them, the slicing tools include PyDub;

[0072] Step S24, preprocessing the video data by frame extraction, motion detection, and time segmentation to obtain preprocessed structured video data;

[0073] Among them, the tools used for frame extraction and motion detection are OpenCV;

[0074] 3. The method according to claim 1 is characterized in that: the specific steps of extracting corresponding feature representations from the pre-processed structured multimodal data and fusing them in step S3 include:

[0075] Step S31, using natural language processing technology to extract text feature representation from the pre-processed structured text data;

[0076] Step S32: using convolutional neural network technology to extract image feature representation from the preprocessed structured image data;

[0077] Step S33, extracting audio feature representation, including using a deep learning model to extract frequency domain and time domain feature representations from the preprocessed structured audio data;

[0078] Step S34, extracting video feature representation, including using a 3D convolutional neural network to extract spatiotemporal feature representation from the preprocessed structured video data, using a long short-term memory network modeling to extract time series feature representation, and using an optical flow method to analyze and extract motion trajectory feature representation;

[0079] Step S35: Use weighted fusion technology to map feature representations of different modalities to the same feature space, and use attention mechanism to align the time axis of video data and audio data to ensure that all feature representations can match each other at the same time point;

[0080] Suppose the video feature is: V = [v1, v2, ..., vT]

[0081] The audio feature is: A = [a1, a2, ..., aT]

[0082] The aligned features are expressed as:

[0083]

[0084] Where: α t,i and β t,j are the weights of the attention mechanism on video and audio features at time step t, respectively;

[0085] The weights are calculated using the attention mechanism:

[0086]

[0087] Step S36: weighted fusion is adopted to assign different weights according to the reliability of each modal model to obtain the fused feature representation.

[0088] The pre-constructed knowledge graph of step S4 is constructed based on the existing TCM literature library, and its construction method is as follows:

[0089] Step S41, collecting data from an existing TCM literature database;

[0090] Among them: the Chinese medicine literature database includes ancient documents, modern research papers, clinical cases, pharmacopoeia, and teaching materials.

[0091] Step S42, preprocessing the collected data, including: removing noise, standardizing terminology, and unifying data format;

[0092] Step S43: Use graph database technology to construct a TCM knowledge graph to associate TCM concepts, symptoms, drugs, prescriptions, therapies, etc.

[0093] The specific steps of adjusting the weights of elements in the entity set and the relationship set according to the context in step S5 include:

[0094] Step S51: Perform dialogue management and design dialogue flow to enable multiple rounds of dialogue between the system and the user;

[0095] Step S52: extract key information from multiple rounds of historical conversations, and perform semantic analysis based on the key information to generate contextual semantic representation;

[0096] Step S53: Based on the contextual semantic representation, evaluate the relevance of the entity to the current question, and adjust the weight value of each entity in the entity set through a weight distribution formula or an attention mechanism;

[0097] The specific steps of step S6 include:

[0098] Step S61: Filter out all possible candidate answers from the pre-built knowledge graph according to the entity set and relationship set after adjusting the weights;

[0099] Step S62: formulating a path search strategy according to the number of all possible candidate answers screened out;

[0100] Step S63: dynamically adjust the path search strategy using an adaptive path exploration method to obtain an adjusted path search strategy. The path search strategy is dynamically adjusted according to the context information of the current question and the relevance of the alternative answers. If some paths are significantly better than other paths, these paths are preferentially explored through weighted adjustment.

[0101] Given a collection of paths:

[0102] P={p1,p2,…,p n}, path weight ω i By the path scoring function Score(p i )Decide:

[0103]

[0104] The path scoring function can be defined based on the context information and the relevance of the alternative answers:

[0105] Score(p i )=λ1·Rel(p i ,q)+λ2·Sim(p i ,C)

[0106] Among them: Rel(p i ,q): path p i Relevance to the current question q; Sim(p i ,C): path p i Similarity with context C; λ1, λ2: weight hyperparameters used to balance the contribution of the two parts.

[0107] Weighted path exploration formula: Assuming that the adjusted path search strategy determines the path exploration priority by weighted random sampling, the path p i The probability of being explored is:

[0108]

[0109] For the entity set E and relationship set R involved in the path, the dynamically adjusted weights are:

[0110]

[0111] in: Represents the previous weight of an entity or relationship; Rel(e k ,q),Rel(r m ,q) represents the relevance of the entity or relationship to the current problem; γ represents the smoothing factor that controls the dynamic update.

[0112] Step S64: Using the entity set and relationship set with adjusted weights as the search starting point, in the pre-built knowledge graph, execute the adjusted path search strategy to find all possible paths leading to the final answer.

[0113] A multimodal intelligent question answering system, comprising:

[0114] An input module, used to obtain multimodal data input by users;

[0115] A preprocessing module, used for preprocessing multimodal data to obtain structured multimodal data;

[0116] A data fusion module is used to extract corresponding feature representations from structured multimodal data and fuse them to obtain fused feature representations;

[0117] The graph query module is used to find entities and relationships related to the fused feature representation in the pre-built knowledge graph and form entity sets and relationship sets;

[0118] A weight adjustment module is used to adjust the weights of elements in the entity set and the relationship set according to the context to obtain the entity set and the relationship set after the weight adjustment;

[0119] The path building module selects all possible candidate answers from the pre-built knowledge graph based on the entity set and relationship set after adjusting the weights, and uses the adaptive path exploration method to build a guiding path for each candidate answer;

[0120] The path evaluation module evaluates and optimizes all guiding paths and assigns weights to them, selects the guiding path with the highest weight, and uses the alternative answer pointed to by the guiding path with the highest weight as the retrieval answer corresponding to the user's question.

[0121] A multimodal intelligent question-answering device, comprising:

[0122] A storage medium for storing a program and a pre-created knowledge graph, wherein the program is used to implement the multimodal intelligent question-answering method; and

[0123] The processor is used to execute corresponding programs according to user input.

[0124] Among them, the processor can be selected as ESP32-S3-WROOM-1 (such as Figure 2 ) and ESP32-S3-WROOM-1U (as Figure 3 The two general-purpose Wi-Fi + low-power Bluetooth MCU modules are equipped with ESP32-S3 series chips. In addition to a rich set of peripheral interfaces, the modules also have powerful neural network computing and signal processing capabilities, which are suitable for a variety of application scenarios in the AIoT field, such as wake-up word detection and voice command recognition, face detection and recognition, smart home, smart appliances, smart control panels, smart speakers, etc. ESP32-S3-WROOM-1 uses a PCB onboard antenna, and ESP32-S3-WROOM-1U uses a connector to connect an external antenna. Both modules are available in a variety of models.

[0125] The operating temperature of ESP32-S3-WROOM-1-H4 and ESP32-S3-WROOM-1U-H4 is –40 to 105°C, the operating temperature of modules with built-in ESP32-S3R8 and ESP32-S3R16V is –40 to 65°C, and the operating temperature of other models is –40 to 85°C. Please note that for R8 and R16V series modules (built-in OctalSPIPSRAM), if the PSRAM ECC function is enabled, the maximum module temperature can be increased to 85°C, but the available capacity of PSRAM will be reduced by 1 / 16.

[0126] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0127] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal intelligent question answering method, characterized in that: The method comprises: Step S1, obtaining multimodal data; the multimodal data is question data input by the user; Wherein, the multimodal data includes: text data, image data, audio data and video data; Step S2, preprocessing the multimodal data to obtain preprocessed structured multimodal data; Step S3, extracting corresponding feature representations from the preprocessed structured multimodal data and fusing them to obtain fused feature representations; Step S4: in the pre-constructed knowledge graph, find entities and relationships related to the fused feature representation, and form an entity set and a relationship set; Step S5: adjusting the weights of the elements in the entity set and the relationship set according to the context to obtain the entity set and the relationship set after the weights are adjusted; Step S6: According to the entity set and relationship set after adjusting the weights, all possible alternative answers are screened from the pre-built knowledge graph, and a guiding path is constructed for each alternative answer using an adaptive path exploration method; Step S7: Evaluate and optimize all guidance paths and assign weights to them, select the guidance path with the highest weight, and use the alternative answer pointed to by the guidance path with the highest weight as the search answer corresponding to the user's question.

2. The method according to claim 1, characterized in that: The specific steps of preprocessing the multimodal data in step S2 include: Step S21, preprocessing the text data by word segmentation, noise removal, part-of-speech tagging, syntactic analysis, and embedding representation to obtain preprocessed structured text data; Step S22, performing preprocessing of resizing, normalization, and data enhancement on the image data to obtain preprocessed structured image data; Step S23, preprocessing the audio data by noise reduction and slicing to obtain preprocessed structured audio data; Step S24: pre-process the video data by frame extraction, motion detection, and time segmentation to obtain pre-processed structured video data.

3. The method according to claim 1, characterized in that: The specific steps of extracting corresponding feature representations from the preprocessed structured multimodal data and fusing them in step S3 include: Step S31, using natural language processing technology to extract text feature representation from the pre-processed structured text data; Step S32: using convolutional neural network technology to extract image feature representation from the preprocessed structured image data; Step S33, extracting audio feature representation, including using a deep learning model to extract frequency domain and time domain feature representations from the preprocessed structured audio data; Step S34, extracting video feature representation, including using a 3D convolutional neural network to extract spatiotemporal feature representation from the preprocessed structured video data, using a long short-term memory network modeling to extract time series feature representation, and using an optical flow method to analyze and extract motion trajectory feature representation; Step S35: Use weighted fusion technology to map feature representations of different modalities to the same feature space, and use the attention mechanism to align the time axis of video data and audio data to ensure that all feature representations can match each other at the same time point. Step S36: weighted fusion is adopted to assign different weights according to the reliability of each modal model to obtain the fused feature representation.

4. The method according to claim 1, characterized in that: The pre-constructed knowledge graph of step S4 is constructed based on the existing TCM literature library, and its construction method is as follows: Step S41, collecting data from an existing TCM literature database; Step S42, preprocessing the collected data, including: removing noise, standardizing terminology, and unifying data format; Step S43: Use graph database technology to construct a TCM knowledge graph to associate TCM concepts, symptoms, drugs, prescriptions, therapies, etc.

5. The method according to claim 1, characterized in that The specific steps of adjusting the weights of elements in the entity set and the relationship set according to the context in step S5 include: Step S51: Perform dialogue management and design dialogue flow to enable multiple rounds of dialogue between the system and the user; Step S52: extract key information from multiple rounds of historical conversations, and perform semantic analysis based on the key information to generate contextual semantic representation; Step S53: Based on the contextual semantic representation, evaluate the relevance of the entity to the current question, and adjust the weight value of each entity in the entity set through a weight allocation formula or an attention mechanism.

6. The method according to claim 1, characterized in that: The specific steps of step S6 include: Step S61: Filter out all possible candidate answers from the pre-built knowledge graph according to the entity set and relationship set after adjusting the weights; Step S62: formulating a path search strategy according to the number of all possible candidate answers screened out; Step S63: dynamically adjust the path search strategy using an adaptive path exploration method to obtain an adjusted path search strategy. The path search strategy is dynamically adjusted according to the context information of the current question and the relevance of the alternative answers. If some paths are significantly better than other paths, these paths are preferentially explored through weighted adjustment. Step S64: Using the entity set and relationship set with adjusted weights as the search starting point, in the pre-built knowledge graph, execute the adjusted path search strategy to find all possible paths leading to the final answer.

7. A multimodal intelligent question-answering system, characterized in that: The system comprises: An input module, used to obtain multimodal data input by users; A preprocessing module, used for preprocessing multimodal data to obtain structured multimodal data; A data fusion module is used to extract corresponding feature representations from structured multimodal data and fuse them to obtain fused feature representations; A graph query module, used to find entities and relationships related to the fused feature representation in a pre-built knowledge graph, and to form an entity set and a relationship set; A weight adjustment module is used to adjust the weights of elements in the entity set and the relationship set according to the context to obtain the entity set and the relationship set after the weight adjustment; The path building module selects all possible candidate answers from the pre-built knowledge graph based on the entity set and relationship set after adjusting the weights, and uses the adaptive path exploration method to build a guiding path for each candidate answer; The path evaluation module evaluates and optimizes all guiding paths and assigns weights to them, selects the guiding path with the highest weight, and uses the alternative answer pointed to by the guiding path with the highest weight as the retrieval answer corresponding to the user's question.

8. A multimodal intelligent question-answering device, characterized in that: The device comprises: A storage medium for storing a program and a pre-created knowledge graph, wherein the program is used to implement the multimodal intelligent question-answering method described in any one of claims 1-6. The processor is used to execute corresponding programs according to user input.

Images