Expressway maintenance knowledge question-answering method and system based on multi-modal information

Abstract

The invention discloses a highway maintenance knowledge question and answer method and system based on multi-modal information, and belongs to the technical field of knowledge question and answer. Comprising the steps of obtaining user question data and converting the user question data into a plurality of modal feature vectors; processing the plurality of modal feature vectors through a preset multi-modal decoupling network to obtain complementarity features and consistency semantic features; on the basis of the complementarity features and the consistency semantic features, independent retrieval of all modes is carried out in a preset multi-mode knowledge base with the consistency semantic features as guidance, cross-mode association retrieval is carried out through the association degree between different modes of knowledge, and candidate knowledge of all modes is obtained; and according to the consistent semantic features, processing each modal candidate knowledge through an answer generation network to obtain a multi-modal answer. The accuracy and flexibility of answering the expressway maintenance knowledge can be improved, and the problem that the processing accuracy of the existing question and answer task containing the multi-modal expressway maintenance knowledge is not high is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge question answering, and in particular to a highway maintenance knowledge question answering method and system based on multimodal information. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Highway maintenance knowledge quiz plays a vital role in maintaining the value of public assets and improving the efficiency of road administration and operation. On the one hand, highways are important public infrastructure, and their maintenance is not only related to the smoothness and safety of traffic, but also to the preservation and appreciation of public investment. Knowledge quiz can integrate and apply a large amount of maintenance data and historical experience to help decision makers formulate scientific maintenance plans and strategies; on the other hand, based on advanced data analysis technology, the knowledge quiz system can predict the degradation trend and maintenance needs of road facilities, help managers to carry out preventive maintenance in a timely manner, reduce the frequency and maintenance time of road failures, avoid expensive overhauls and sudden accidents, and improve the efficiency and reliability of road operation.

[0004] Building an effective and flexible highway maintenance knowledge question-answering system is the premise and basis for realizing the above functions. However, due to the complexity and diversity of sensor forms and data modalities in the highway environment, the design of the maintenance knowledge question-answering system has always been unresolved.

[0005] Existing question-answering strategies can be divided into rule-based, learning-based, and retrieval-enhanced methods. Rule-based methods usually target small-scale fixed scenarios and implement simple question-answering through a few sets of manually preset answers, which cannot cope with complex and diverse highway scenario question-answering. Learning-based methods usually rely on large language models trained by massive data. Although they achieve more flexible question-answering, they perform poorly in scenarios with strong domain knowledge. Retrieval-enhanced methods combine local domain knowledge bases with large language models to achieve flexible and professional question-answering interactions. However, affected by the model architecture and data format, existing methods still cannot handle question-answering tasks involving multimodal highway maintenance knowledge well. Summary of the invention

[0006] In order to address the deficiencies of the prior art, the present invention provides a highway maintenance knowledge question and answer method, system, electronic device, computer-readable storage medium and computer program product based on multimodal information, which achieves compatibility with multimodal data formats in the three stages of questioning, retrieval and answering, thereby improving the accuracy and flexibility of highway maintenance knowledge question and answer.

[0007] In a first aspect, the present invention provides a highway maintenance knowledge question-answering method based on multimodal information;

[0008] A highway maintenance knowledge question answering method based on multimodal information, comprising:

[0009] Obtain user question data and convert it into multiple modal feature vectors;

[0010] The multi-modal feature vectors are processed through a preset multi-modal decoupling network to obtain complementary features and consistent semantic features;

[0011] Based on complementary features and consistent semantic features, each modality is independently searched in the preset multimodal knowledge base with the consistent semantic features as the guide, and cross-modal association search is performed using the correlation degree between different modal knowledge to obtain candidate knowledge of each modality;

[0012] According to the consistent semantic features, the candidate knowledge of each modality is processed through the answer generation network to obtain multimodal answers.

[0013] In some implementations, before obtaining the user question data, the method further includes:

[0014] Expressway maintenance knowledge is acquired, the expressway maintenance knowledge is stored in a single mode according to the modality type, and cross-modal associations between different modalities are established to form a multimodal knowledge base.

[0015] In some embodiments, the multimodal decoupling network includes a plurality of parallel multilayer perceptrons and a shared perceptron;

[0016] The multi-layer perceptron is used to process a single modal feature vector to obtain complementary features; the shared perceptron is used to process a plurality of concatenated modal feature vectors to obtain consistent semantic features.

[0017] In some embodiments, the independent search of each modality in a preset multimodal knowledge base based on complementary features and consistent semantic features and guided by consistent semantic features is specifically as follows: calculating the degree of correlation between complementary features and consistent semantic features and knowledge of the same modality type in the multimodal knowledge base, and screening and determining candidate knowledge according to the degree of correlation.

[0018] In some implementations, the cross-modal association retrieval using the degree of association between different modal knowledge is specifically: traversing the candidate knowledge determined by independent retrieval, and selecting other modal knowledge with the highest degree of association therewith.

[0019] In some implementations, the step of processing candidate knowledge of each modality through an answer generation network according to the consistent semantic features to obtain a multimodal answer specifically includes:

[0020] Input the consistent semantic features into the pre-trained large language model to predict the modality type that appears in the next position in the answer;

[0021] According to the consistent semantic features and the predicted modality type, the corresponding candidate knowledge is determined and a multimodal answer is generated.

[0022] In a second aspect, the present invention provides a highway maintenance knowledge question-answering system based on multimodal information;

[0023] A highway maintenance knowledge question-answering system based on multimodal information, comprising:

[0024] The question understanding module is configured to: obtain user question data and convert it into multiple modal feature vectors; process the multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent semantic features;

[0025] The knowledge retrieval module is configured to: based on complementary features and consistent semantic features, perform independent retrieval of each modality in a preset multimodal knowledge base guided by the consistent semantic features, perform cross-modal association retrieval using the correlation degree between knowledge of different modalities, and obtain candidate knowledge of each modality;

[0026] The answer generation module is configured to: process each modality candidate knowledge through the answer generation network according to the consistent semantic features to obtain a multimodal answer.

[0027] In a third aspect, the present invention provides an electronic device;

[0028] An electronic device includes a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the highway maintenance knowledge question-answering method based on multimodal information.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0030] A computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the highway maintenance knowledge question-answering method based on multimodal information.

[0031] In a fifth aspect, the present invention provides a computer program product;

[0032] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The technical solution provided by the present invention designs a multimodal knowledge association rule, integrates the existing local multimodal field data into a question-answering knowledge base, and converts the existing messy and independent highway maintenance knowledge into a form that is easy to represent and store through the strategy of same-modal storage and cross-modal association, while maintaining the association relationship between knowledge to facilitate later retrieval and organization.

[0035] 2. The technical solution provided by the present invention constructs a multimodal question understanding model, decouples the consistency and complementarity information from the user's multimodal questions, integrates the consistency and complementarity information in multiple modalities, obtains the semantic connotation of the question, and realizes the semantic understanding of the multimodal question input.

[0036] 3. The technical solution provided by the present invention uses retrieval enhancement technology to obtain semantically related knowledge and knowledge of other interrelated modalities in the knowledge base, uses this knowledge as answer candidates, and generates answers containing multiple modalities through an answer generation model, thereby improving the accuracy and flexibility of the answers.

[0037] 4. The technical solution provided by the present invention, based on multimodal information retrieval enhancement, constructs a multimodal knowledge base related to highway maintenance, and implements a question-and-answer system containing multimodal data through question understanding, knowledge retrieval and answer generation modules, achieving compatibility with multimodal data formats in the three stages of questioning, retrieval and answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 A schematic diagram of a process flow of a highway maintenance knowledge question-answering method based on multimodal information provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of constructing a multimodal knowledge base provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a framework for highway maintenance knowledge question and answer based on multimodal information provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0045] Embodiment 1

[0046] The highway maintenance knowledge question-answering task has modal diversity. The existing question-answering strategy cannot cope with such scenarios with strong domain knowledge, and its accuracy and flexibility need to be improved. Therefore, the present invention provides a highway maintenance knowledge question-answering method based on multimodal information.

[0047] The modal diversity in the highway maintenance knowledge question answering task involves two aspects:

[0048] On the one hand, users’ questions are multi-modal. Unlike general knowledge questions and answers, in the highway maintenance scenario, users’ questions usually contain at least two modalities. For example, when users ask “Does the road surface need maintenance? What are the potential safety hazards?”, they usually attach pictures or video clips corresponding to the road surface. These modalities complement each other, and removing any one of them usually results in the question not being accurately understood.

[0049] On the other hand, the system's answer may also involve multiple modes. When answering the question "What safety hazards exist?", the system is expected to simultaneously point out the location of the road hazard, the cause demonstration, and the expected effect diagram after the hazard is eliminated.

[0050] Next, combine Figure 1-Figure 3 , a highway maintenance knowledge question-answering method based on multimodal information disclosed in this embodiment is described in detail. The highway maintenance knowledge question-answering method based on multimodal information includes the following steps:

[0051] S1. Obtain user question data and convert it into multiple modal feature vectors.

[0052] In the scenario of highway maintenance knowledge question answering, user question data are mostly multimodal questions. Since user questions are usually composed of multiple modes, and there is heterogeneity between these modes, it is difficult to process them with the same mapping method or feature extraction structure. Therefore, in this embodiment, the original data of user questions is converted into feature vectors through a unimodal encoding network.

[0053] Furthermore, the text information in the user question data is processed through a text encoding network; specifically, first, the original data is preprocessed using strategies such as text cleaning, word segmentation, and stop word removal, and then the bag-of-words model or word embedding model is used to extract the features of the phrases. Finally, common text encoding methods (such as one-hot encoding, TF-IDF encoding, word embedding encoding, etc.) are used to aggregate the phrase features into a text feature vector, that is, the modal feature vector corresponding to the text information.

[0054] The image information in the user question data is processed through an image coding network; specifically, the image coding network is composed of a convolutional structure (for example, the image coding network can be VGG, ResNet, etc.), which retains the spatial position information in the image, maps the two-dimensional image to a one-dimensional feature space, and obtains the modal feature vector corresponding to the image information.

[0055] As an implementation mode, before executing S1, if a multimodal knowledge base is not built, the method further includes:

[0056] Acquire highway maintenance knowledge, store it in a single mode according to the modality type, and establish cross-modal associations between different modalities to form a multimodal knowledge base.

[0057] There is a huge amount of highway maintenance knowledge in local storage, which can be used as a reference for domain knowledge. Since this knowledge is stored in a messy and independent manner in the local or server persistent components, it needs to be converted into a form that is easy to represent and store, while maintaining the association between knowledge to form a multimodal knowledge base that is easy to retrieve and organize. The specific process is as follows:

[0058] (1) Highway maintenance knowledge is read in JSON format, for example: "The skills for cleaning gravel in highway scenarios include:...", "The precautions for cleaning gravel in highway scenarios include:...", "The expected effect of gravel cleaning and the comparison before and after cleaning are shown in the figure: (Figure)", "The typical case of gravel cleaning is shown in the video: (Video)", etc.

[0059] (2) Analyze the highway maintenance knowledge obtained. Each piece of knowledge consists of text data (used to describe the knowledge) and optional other modal data: image data (recording static information of the highway or displaying expected results), video data (recording dynamic information of the highway over a period of time or displaying expected results), audio data (recording information captured by sound sensor equipment), animation data (used for simulation demonstration), etc.

[0060] In this embodiment, the highway maintenance knowledge read from the local storage is unstructured and contains relatively free data forms.

[0061] (3) The highway maintenance knowledge obtained is constructed into a structured multimodal knowledge base, such as Figure 2 As shown, specifically including:

[0062] (301) According to the modality type, each piece of maintenance knowledge is stored in a corresponding knowledge base, such as a "text knowledge base", "image knowledge base", "video knowledge base" and "animation knowledge base".

[0063] Considering that the same modality shares the same storage method and processing strategy, in this embodiment, the highway maintenance knowledge is stored according to the modality type.

[0064] (302) Between knowledge bases of different modalities, cross-modal association weights are added to model the association relationship between knowledge of each modality. The edges where these association weights are located connect the knowledge points in two different modal bases, and finally the knowledge points are modeled as a weighted undirected graph structure.

[0065] Since knowledge is usually described by multiple modalities, there is a logical association relationship between different modalities. In this embodiment, the association relationship between cross-modal knowledge is quantified by cross-modal association weights.

[0066] Exemplarily, when knowledge of different modalities in the knowledge base is stored separately, the cross-modal association weight between different modal data from the same piece of knowledge is initialized to 1, indicating that the degree of association is 100%; correspondingly, the weight between different modal data from different pieces of knowledge is initialized to 0, indicating that the degree of association is 0%.

[0067] Furthermore, if there is naturally knowledge classification in the multimodal knowledge base, on the basis of the above, the cross-modal association weights between different modal data in the same category but different items of knowledge can be set to a∈[0,1], where a is a hyperparameter that can be set manually.

[0068] At the same time, the cross-modal association weights (hyperparameters) in the weighted undirected graph can be updated as trainable parameters. After initialization in the "establishment phase", a reinforcement learning strategy is adopted based on user feedback on the problem. As the network parameters are updated, the cross-modal association weights are also updated.

[0069] S2. Process multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent features.

[0070] The multimodal decoupling network aims to mine the semantic information in the feature vectors of multiple modalities to understand the user's questions. Although the data of different modalities in this link are provided in the form of feature vectors, they contain both consistent semantic information and complementary modality-specific information.

[0071] In this embodiment, the multimodal decoupling network includes multiple multilayer perceptrons (MLPs), and the number of multilayer perceptrons is M+1, where "M" is the number of compatible modalities in the system, which means that each modality corresponds to a modality-specific multilayer perceptron, which takes modal features as input and outputs the complementary features contained therein; "1" represents a shared perceptron, which takes the concatenation of each modal feature as input and outputs the consistent semantic features contained therein.

[0072] Furthermore, the consistent semantic features in the multimodal decoupling network should contain as much information as possible shared by each modality, while being mutually exclusive with complementary features as much as possible.

[0073] Therefore, according to information theory, the optimization objective of the multimodal decoupling network can be expressed as:

[0074]

[0075] In the formula, I(·,·) represents the mutual information of two random variables, β is a balance parameter used to control the information compression and retention preference, and F m 、F con and They represent the characteristics, consistency characteristics and complementary characteristics of mode m respectively, m = {1,...,M}.

[0076] S3. Based on complementary features and consistent semantic features, each modality is independently searched in the preset multimodal knowledge base guided by the consistent semantic features, and cross-modal association search is performed using the degree of association between knowledge in different modalities to obtain candidate knowledge of each modality.

[0077] Given consistent semantic information and complementary information of each modality, relevant multimodal knowledge retrieval aims to retrieve knowledge related to the question from the multimodal knowledge base as candidate data for the answer, guided by semantic information. As an implementation method, S3 specifically includes:

[0078] S301, calculating the correlation between complementary features and consistent semantic features and the same modality type knowledge in the multimodal knowledge base, and screening and determining candidate knowledge according to the correlation.

[0079] Taking the text modality as an example, S301 specifically includes: taking the concatenation of consistent semantic features and complementary text features as input, calculating the relevance with the text knowledge in the knowledge base (i.e., a measurement method such as Euclidean distance or relevance score), sorting the text knowledge in the knowledge base according to the relevance, and selecting the top K most relevant knowledge as the text candidate knowledge for the system answer.

[0080] If it is an image modality, the cosine similarity calculation is used to determine the relevance of the concatenation of consistent semantic features and complementary text features with the image knowledge in the knowledge base; the image knowledge in the knowledge base is sorted according to the relevance, and the top K most relevant image knowledge is selected as the image candidate knowledge for the system answer.

[0081] For vector A and vector B, the cosine similarity calculation is expressed as:

[0082] cos(A,B)=(A·B) / (||A||*||B||).

[0083] Following the same retrieval strategy, candidate knowledge corresponding to other modalities can be obtained.

[0084] S302, traverse the candidate knowledge determined by independent retrieval, and select other modal knowledge with the highest degree of correlation therewith.

[0085] Since the knowledge base records the degree of association between knowledge of different modalities, another key point of retrieval is to select corresponding associated knowledge to expand the candidate knowledge set.

[0086] Taking the text modality as an example, we traverse the existing text candidate knowledge in turn, select the knowledge with the highest degree of relevance (i.e., the largest relevance weight) from each other modality, and add it to the corresponding candidate knowledge. Figure 2 Taking the knowledge base shown in the figure as an example, when "gravel cleaning skills" appears in the text candidate knowledge, the cross-modal association retrieval will add "gravel cleaning skills video" and "roadblock removal effect picture" to the video candidate knowledge and image candidate knowledge respectively.

[0087] S4. Based on the consistent semantic features, the answer generation network is used to process the candidate knowledge of each modality to obtain a multimodal answer. Specifically, it includes:

[0088] S401. Input the consistent semantic features into the pre-trained large language model to predict the modality type that appears in the next position in the answer.

[0089] S402: Determine corresponding candidate knowledge and generate a multimodal answer based on consistent semantic features and predicted modality types.

[0090] Specifically, multimodal answer generation aims to generate the final answer based on the retrieved candidate knowledge of each modality. This part is implemented by an answer generation network with output heads of each modality attached. The answer generation network is a pre-trained large language model (such as GPT, LLaMA, etc.), whose input is the semantic features of the question and predicts the type of modality that should appear in the next position of the answer. According to the prediction of the answer generation network, the output head of the corresponding modality is selected, the semantic features and candidate knowledge of the corresponding modality are received, and the answer is output.

[0091] For example, when a partial answer has been generated: "The highway needs gravel cleaning, and the expected effect is as shown in the figure below", the answer generation network predicts that the next position of the answer should be the image mode, and the output head corresponding to the image mode starts working to generate an image at the next position of the existing answer.

[0092] Since a single round of question and answer cannot usually accurately solve the user's problem, as an implementation method, multiple rounds of dialogue and question and answer (such as Figure 1 ), and after each answer is generated, a feedback option "Did you solve your problem?" is added. If the user continues to ask, it will jump to S2 and repeat S2-S4. If the user chooses "solved" or "unsolved", it will exit this Q&A session, and the system will mark the reward of this Q&A chain as 1 or 0 respectively.

[0093] Use reinforcement learning strategies to update model parameters and cross-modal association weights of weighted undirected graphs in multimodal knowledge bases. t ,a t ,...,q T ,a T ,r], then the optimization goal of the system is to maximize the feedback reward, which can be expressed as:

[0094]

[0095] Among them, t and T represent the round index and total number of rounds of question and answer, respectively, and q t represents the user's question in round t, a t represents the model answer in round t, r(q t ,a t ) represents the corresponding feedback reward when the user asks the question in round t and the model answers the question in round t, for example, r(q t ,a t) can score the user when the user asks a question in round t and the model answers the question in round t. If the user chooses "solved", the corresponding user score is 1; if the user chooses "unsolved", the corresponding user score is 0.

[0096] Specifically, the cross-modal association weights can be modeled as a set of trainable parameters a. When constructing a multimodal knowledge base, the weights are initialized, that is, ω is set to a number between [0,1]. When the question and answer session is over, the user's feedback on the question is obtained. According to the "maximize feedback reward" objective function R, taking the gradient descent optimization algorithm as an example, the partial derivatives of the various parameters in the model are calculated. At this time, ω is used as one of the parameters and is updated in the opposite direction of its gradient, that is:

[0097]

[0098] Embodiment 2

[0099] Combination Figure 3 This embodiment discloses a highway maintenance knowledge question-answering system based on multimodal information, including:

[0100] The question understanding module is configured to: obtain user question data and convert it into multiple modal feature vectors; process the multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent semantic features;

[0101] The knowledge retrieval module is configured to: based on complementary features and consistent semantic features, perform independent retrieval of each modality in a preset multimodal knowledge base guided by the consistent semantic features, perform cross-modal association retrieval using the correlation degree between knowledge of different modalities, and obtain candidate knowledge of each modality;

[0102] The answer generation module is configured to: process each modality candidate knowledge through the answer generation network according to the consistent semantic features to obtain a multimodal answer.

[0103] It should be noted that the above-mentioned question understanding module, knowledge retrieval module and answer generation module correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0104] Embodiment 3

[0105] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the highway maintenance knowledge question-and-answer method based on multimodal information are completed.

[0106] Embodiment 4

[0107] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned highway maintenance knowledge question and answer method based on multimodal information are completed.

[0108] Embodiment 5

[0109] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A highway maintenance knowledge question-answering method based on multimodal information, characterized in that: include: Obtain user question data and convert it into multiple modal feature vectors; The multi-modal feature vectors are processed through a preset multi-modal decoupling network to obtain complementary features and consistent semantic features; Based on complementary features and consistent semantic features, each modality is independently searched in the preset multimodal knowledge base with the consistent semantic features as the guide, and cross-modal association search is performed using the correlation degree between different modal knowledge to obtain candidate knowledge of each modality; According to the consistent semantic features, the candidate knowledge of each modality is processed through the answer generation network to obtain multimodal answers.

2. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: Before obtaining user question data, it also includes: Expressway maintenance knowledge is acquired, the expressway maintenance knowledge is stored in a single mode according to the modality type, and cross-modal associations between different modalities are established to form a multimodal knowledge base.

3. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The multimodal decoupling network includes multiple parallel multi-layer perceptrons and shared perceptrons; The multi-layer perceptron is used to process a single modal feature vector to obtain complementary features; the shared perceptron is used to process a plurality of concatenated modal feature vectors to obtain consistent semantic features.

4. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The independent retrieval of each modality in a preset multimodal knowledge base based on complementary features and consistent semantic features and guided by consistent semantic features is specifically as follows: calculating the correlation between complementary features and consistent semantic features and knowledge of the same modality type in the multimodal knowledge base, and screening and determining candidate knowledge according to the correlation.

5. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The cross-modal association retrieval using the association degree between different modal knowledge is specifically as follows: traversing the candidate knowledge determined by independent retrieval, and selecting other modal knowledge with the highest association degree therewith.

6. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The method of processing candidate knowledge of each modality through an answer generation network according to the consistent semantic features to obtain a multimodal answer specifically includes: Input the consistent semantic features into the pre-trained large language model to predict the modality type that appears in the next position in the answer; According to the consistent semantic features and the predicted modality type, the corresponding candidate knowledge is determined and a multimodal answer is generated.

7. Highway maintenance knowledge question-answering system based on multimodal information, characterized by: include: The question understanding module is configured to: obtain user question data and convert it into multiple modal feature vectors; process the multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent semantic features; The knowledge retrieval module is configured to: based on complementary features and consistent semantic features, perform independent retrieval of each modality in a preset multimodal knowledge base guided by the consistent semantic features, perform cross-modal association retrieval using the correlation degree between knowledge of different modalities, and obtain candidate knowledge of each modality; The answer generation module is configured to: process each modality candidate knowledge through the answer generation network according to the consistent semantic features to obtain a multimodal answer.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the highway maintenance knowledge question-answering method based on multimodal information as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the highway maintenance knowledge question-answering method based on multimodal information as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the highway maintenance knowledge question-answering method based on multimodal information as described in any one of claims 1 to 6 are implemented.

Images