Highway maintenance knowledge question and answer method and system based on multi-modal information

Abstract

The application discloses a kind of based on multi-modal information's expressway maintenance knowledge question and answer method and system, belong to knowledge question and answer technical field. Including: obtaining user question data and converting into multiple modal feature vectors;Multiple modal feature vectors are processed by the preset multi-modal decoupling network, and complementary features and consistent semantic features are obtained;Based on complementary features and consistent semantic features, consistent semantic features are guided in the preset multi-modal knowledge base to carry out each modal independent search, and the correlation degree between different modal knowledge is used to carry out cross-modal correlation search, and obtain each modal candidate knowledge;According to consistent semantic features, each modal candidate knowledge is processed by answer generation network, and multiple modal answers are obtained. It can improve the accuracy and flexibility of expressway maintenance knowledge answer, solve the problem that the accuracy of existing multi-modal expressway maintenance knowledge question and answer task processing is not high.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge question answering, and in particular to a highway maintenance knowledge question answering method and system based on multimodal information. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Highway maintenance knowledge and answers play a vital role in maintaining the value of public assets and improving the efficiency of highway operations. Highways are crucial public infrastructure, and their maintenance is crucial not only for smooth and safe traffic but also for preserving and increasing the value of public investment. Knowledge and answers can integrate and apply extensive maintenance data and historical experience to help decision-makers develop informed maintenance plans and strategies. Furthermore, based on advanced data analysis techniques, the knowledge and answers system can predict the degradation trends and maintenance needs of road infrastructure, helping managers conduct timely preventative maintenance. This reduces the frequency and repair time of road failures, avoids costly overhauls and unexpected accidents, and improves the efficiency and reliability of road operations.

[0004] Building an effective and flexible highway maintenance knowledge question-answering system is the premise and foundation for realizing the above functions. However, due to the complexity and diversity of sensor forms and data modalities in the highway environment, the design of the maintenance knowledge question-answering system has always been pending.

[0005] Existing question-answering strategies can be categorized as rule-based, learning-based, and retrieval-enhanced methods. Rule-based methods typically target smaller, fixed scenarios, achieving simple question-answering through a few sets of manually preset answers. These methods are unable to cope with the complex and diverse nature of highway Q&A. Learning-based methods typically leverage large language models trained on massive amounts of data. While these methods offer more flexible Q&A, they perform poorly in scenarios requiring strong domain knowledge. Retrieval-enhanced methods, based on large language models and combined with local domain knowledge bases, enable flexible and professional Q&A interactions. However, due to limitations in model architecture and data format, existing methods still struggle to effectively handle Q&A tasks involving multimodal highway maintenance knowledge. Summary of the Invention

[0006] To address the deficiencies of the prior art, the present invention provides a method, system, electronic device, computer-readable storage medium, and computer program product for answering questions about highway maintenance based on multimodal information. These methods achieve compatibility with multimodal data formats in the three stages of questioning, retrieval, and answering, thereby improving the accuracy and flexibility of highway maintenance knowledge question answering.

[0007] In a first aspect, the present invention provides a highway maintenance knowledge question-answering method based on multimodal information;

[0008] A highway maintenance knowledge question-answering method based on multimodal information includes:

[0009] Obtain user question data and convert it into multiple modal feature vectors;

[0010] The multi-modal feature vectors are processed through a preset multi-modal decoupling network to obtain complementary features and consistent semantic features;

[0011] Based on complementary features and consistent semantic features, each modality is searched independently in the preset multimodal knowledge base with the consistent semantic features as the guide, and cross-modal association search is performed using the correlation degree between different modal knowledge to obtain candidate knowledge of each modality;

[0012] According to the consistent semantic features, the candidate knowledge of each modality is processed through the answer generation network to obtain multimodal answers.

[0013] In some implementations, before obtaining the user question data, the process further includes:

[0014] Expressway maintenance knowledge is acquired, the expressway maintenance knowledge is stored in a single mode according to the modality type, and cross-modal associations between different modalities are established to form a multimodal knowledge base.

[0015] In some embodiments, the multimodal decoupling network includes multiple parallel multilayer perceptrons and shared perceptrons;

[0016] The multi-layer perceptron is used to process a single modal feature vector to obtain complementary features; the shared perceptron is used to process multiple concatenated modal feature vectors to obtain consistent semantic features.

[0017] In some embodiments, the independent retrieval of each modality in a preset multimodal knowledge base based on complementary features and consistent semantic features and guided by consistent semantic features is specifically as follows: calculating the degree of correlation between complementary features and consistent semantic features and knowledge of the same modality type in the multimodal knowledge base, and screening and determining candidate knowledge according to the degree of correlation.

[0018] In some embodiments, the cross-modal association retrieval using the degree of association between different modal knowledge is specifically performed by traversing the candidate knowledge determined by independent retrieval and selecting other modal knowledge with the highest degree of association therewith.

[0019] In some embodiments, processing candidate knowledge of each modality through an answer generation network based on consistent semantic features to obtain a multimodal answer specifically includes:

[0020] Input the consistent semantic features into the pre-trained large language model to predict the modality type that appears next in the answer;

[0021] According to the consistent semantic features and the predicted modality type, the corresponding candidate knowledge is determined and a multimodal answer is generated.

[0022] In a second aspect, the present invention provides a highway maintenance knowledge question-answering system based on multimodal information;

[0023] A highway maintenance knowledge question-answering system based on multimodal information, comprising:

[0024] The question understanding module is configured to: obtain user question data and convert it into multiple modal feature vectors; process the multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent semantic features;

[0025] The knowledge retrieval module is configured to: based on complementary features and consistent semantic features, use the consistent semantic features as a guide to perform independent searches for each modality in a preset multimodal knowledge base, and use the degree of association between knowledge in different modalities to perform cross-modal association searches to obtain candidate knowledge for each modality;

[0026] The answer generation module is configured to process candidate knowledge of each modality through the answer generation network according to the consistent semantic features to obtain a multimodal answer.

[0027] In a third aspect, the present invention provides an electronic device;

[0028] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0030] A computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information.

[0031] In a fifth aspect, the present invention provides a computer program product;

[0032] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The technical solution provided by this invention designs a multimodal knowledge association rule to integrate existing local multimodal domain data into a question-and-answer knowledge base. Through the strategy of homomodal storage and cross-modal association, the existing chaotic and independent highway maintenance knowledge is converted into a form that is easy to represent and store, while maintaining the association relationship between knowledge, facilitating subsequent retrieval and organization.

[0035] 2. The technical solution provided by the present invention constructs a multimodal question understanding model, decouples the consistency and complementarity information from the user's multimodal questions, integrates the consistency and complementarity information in multiple modalities, obtains the semantic connotation of the question, and realizes the semantic understanding of the multimodal question input.

[0036] 3. The technical solution provided by the present invention uses retrieval enhancement technology to obtain semantically related knowledge and knowledge of other interrelated modalities in the knowledge base, uses this knowledge as answer candidates, and generates answers containing multiple modalities through the answer generation model, thereby improving the accuracy and flexibility of the answers.

[0037] 4. The technical solution provided by the present invention, based on multimodal information retrieval enhancement, constructs a multimodal knowledge base related to highway maintenance, and implements a question-answering system containing multimodal data through question understanding, knowledge retrieval and answer generation modules, achieving compatibility with multimodal data formats in the three stages of questioning, retrieval and answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 A schematic diagram of a process flow of a highway maintenance knowledge question-answering method based on multimodal information provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of constructing a multimodal knowledge base provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a framework for highway maintenance knowledge question-answering based on multimodal information provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0045] Example 1

[0046] Highway maintenance knowledge question-answering tasks have modal diversity. Existing question-answering strategies are unable to cope with such scenarios with strong domain knowledge, and their accuracy and flexibility need to be improved. Therefore, the present invention provides a highway maintenance knowledge question-answering method based on multimodal information.

[0047] Modal diversity in highway maintenance knowledge question answering tasks involves two aspects:

[0048] First, user questions are modally diverse. Unlike general knowledge Q&A, in highway maintenance scenarios, user questions typically encompass at least two modalities. For example, when a user asks, "Does this road surface need maintenance? What safety hazards exist?" they often include pictures or video clips of the road surface. These modalities complement each other; removing any one of them often results in a lack of accurate understanding of the question.

[0049] On the other hand, the system's response may also involve multiple modalities. When answering the question "What safety hazards exist?", the system is expected to simultaneously indicate the location of the road hazard, a demonstration of the cause, and a diagram of the expected effect after the hazard is eliminated.

[0050] Next, combine Figure 1-Figure 3 , a highway maintenance knowledge question-answering method based on multimodal information disclosed in this embodiment is described in detail. The highway maintenance knowledge question-answering method based on multimodal information includes the following steps:

[0051] S1. Obtain user question data and convert it into multiple modal feature vectors.

[0052] In the highway maintenance Q&A scenario, user question data is often multimodal. Since user questions typically consist of multiple modalities and these modalities are heterogeneous, they are difficult to process using the same mapping method or feature extraction structure. Therefore, in this embodiment, a unimodal encoding network is used to convert the raw user question data into feature vectors.

[0053] Furthermore, the text information in the user question data is processed through a text encoding network. Specifically, first, the original data is preprocessed using strategies such as text cleaning, word segmentation, and stop word removal. Then, the bag-of-words model or word embedding model is used to extract the features of the phrases. Finally, common text encoding methods (such as one-hot encoding, TF-IDF encoding, word embedding encoding, etc.) are used to aggregate the phrase features into a text feature vector, that is, the modal feature vector corresponding to the text information.

[0054] The image information in the user question data is processed through an image coding network. Specifically, the image coding network is composed of a convolutional structure (for example, the image coding network can be VGG, ResNet, etc.), which retains the spatial position information in the image and maps the two-dimensional image to a one-dimensional feature space to obtain the modal feature vector corresponding to the image information.

[0055] As an implementation method, before executing S1, if a multimodal knowledge base has not been built, the following steps are further included:

[0056] Acquire highway maintenance knowledge, store it in a single modality according to the modality type, and establish cross-modal associations between different modalities to form a multimodal knowledge base.

[0057] There is a massive amount of highway maintenance knowledge in local storage that can be used as a domain knowledge reference. Since this knowledge is stored in a disorganized and independent manner in persistent components locally or on the server, it needs to be converted into a form that is easy to represent and store while maintaining the relationships between knowledge to form a multimodal knowledge base that is easy to retrieve and organize. The specific process is as follows:

[0058] (1) Highway maintenance knowledge is read in JSON format, for example: "The skills for cleaning gravel in highway scenarios include:...", "The precautions for cleaning gravel in highway scenarios include:...", "The expected effect of gravel cleaning and the comparison before and after cleaning are shown in the figure: (Figure)", "A typical case of gravel cleaning is shown in the video: (Video)", etc.

[0059] (2) The read highway maintenance knowledge is analyzed. The composition of each piece of knowledge includes text data (used for knowledge description) and optional other modal data: image data (recording static information of the highway or displaying expected results), video data (recording dynamic information of the highway over a period of time or displaying expected results), audio data (recording information captured by sound sensor equipment), animation data (used for simulation demonstration), etc.

[0060] In this embodiment, the highway maintenance knowledge read from the local storage is unstructured and contains relatively free data forms.

[0061] (3) Construct the read highway maintenance knowledge into a structured multimodal knowledge base, such as Figure 2 As shown, specifically including:

[0062] (301) According to the modality type, each piece of maintenance knowledge is stored in the corresponding knowledge base, such as the “text knowledge base”, “image knowledge base”, “video knowledge base” and “animation knowledge base”.

[0063] Considering that the same modality shares the same storage method and processing strategy, in this embodiment, the highway maintenance knowledge is stored according to the modality type.

[0064] (302) Between knowledge bases of different modalities, cross-modal association weights are added to model the association relationship between knowledge of each modality. The edges containing these association weights connect the knowledge points in two different modal bases, and ultimately the knowledge points are modeled as a weighted undirected graph structure.

[0065] Since knowledge is usually described by multiple modalities, there is a logical association relationship between different modalities. In this embodiment, the association relationship between cross-modal knowledge is quantified by cross-modal association weights.

[0066] For example, when storing knowledge of different modalities in the knowledge base separately, the cross-modal association weight between different modal data from the same piece of knowledge is initialized to 1, indicating that the degree of association is 100%; correspondingly, the weight between different modal data from different pieces of knowledge is initialized to 0, indicating that the degree of association is 0%.

[0067] Furthermore, if knowledge classification naturally exists in the multimodal knowledge base, based on the above, the cross-modal association weights between different modal data in the same category but different items of knowledge can be set to a∈[0,1], where a is a hyperparameter and can be set manually.

[0068] At the same time, the cross-modal association weights (hyperparameters) in the weighted undirected graph can be updated as trainable parameters. After initialization in the "establishment phase", a reinforcement learning strategy is adopted based on user feedback on the problem. As the network parameters are updated, the cross-modal association weights are also updated.

[0069] S2. Process multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent features.

[0070] The multimodal decoupling network aims to mine semantic information from feature vectors of multiple modalities to understand user questions. Although the data from different modalities is provided in the form of feature vectors, they contain both consistent semantic information and complementary modality-specific information.

[0071] In this embodiment, the multimodal decoupling network includes multiple multilayer perceptrons (MLPs), and the number of multilayer perceptrons is M+1, where "M" is the number of compatible modalities in the system, meaning that each modality corresponds to a modality-specific multilayer perceptron, which takes modal features as input and outputs the complementary features contained therein; "1" represents a shared perceptron, which takes the concatenation of the features of each modality as input and outputs the consistent semantic features contained therein.

[0072] Furthermore, the consistent semantic features in the multimodal decoupling network should contain as much information as possible shared by each modality, while being mutually exclusive with complementary features as much as possible.

[0073] Therefore, according to information theory, the optimization objective of the multimodal decoupling network can be expressed as:

[0074]

[0075] Where I(·,·) represents the mutual information of two random variables, β is a balance parameter used to control the preference of information compression and retention, and F m 、F con and They represent the features, consistency features and complementary features of mode m, respectively, where m = {1,...,M}.

[0076] S3. Based on complementary features and consistent semantic features, each modality is searched independently in the preset multimodal knowledge base with the consistent semantic features as the guide, and cross-modal association search is performed using the degree of association between different modal knowledge to obtain candidate knowledge of each modality.

[0077] Given consistent semantic information and complementary information across modalities, relevant multimodal knowledge retrieval aims to use semantic information as a guide to retrieve relevant knowledge from the multimodal knowledge base as candidate data for the answer. As an implementation, S3 specifically includes:

[0078] S301. Calculate the correlation between the complementary features and the consistent semantic features and the knowledge of the same modality in the multimodal knowledge base, and screen and determine candidate knowledge based on the correlation.

[0079] Taking the text modality as an example, S301 specifically involves taking the concatenation of consistent semantic features and complementary text features as input and calculating the relevance (i.e., using metrics such as Euclidean distance or relevance score) to the text knowledge in the knowledge base. The text knowledge in the knowledge base is sorted by relevance, and the top K most relevant pieces of knowledge are selected as candidate text knowledge for the system's answer.

[0080] If it is an image modality, the cosine similarity calculation is used to determine the relevance between the splicing of consistent semantic features and complementary text features and the image knowledge in the knowledge base; the image knowledge in the knowledge base is sorted according to the relevance, and the top K most relevant image knowledge are selected as the image candidate knowledge for the system answer.

[0081] For vector A and vector B, the cosine similarity calculation is expressed as:

[0082] cos(A,B)=(A·B) / (||A||*||B||).

[0083] Following the same retrieval strategy, candidate knowledge corresponding to other modalities can be obtained.

[0084] S302: Traverse the candidate knowledge determined by independent retrieval and select other modal knowledge with the highest degree of correlation.

[0085] Since the knowledge base records the degree of association between different modal knowledge, another key point of retrieval is to select corresponding associated knowledge to expand the candidate knowledge set.

[0086] Taking the text modality as an example, we traverse the existing text candidate knowledge in turn, select the knowledge with the highest degree of correlation (i.e. the largest correlation weight) from each other modality, and add it to the corresponding candidate knowledge. Figure 2 Taking the knowledge base shown in the figure as an example, when "stone cleaning skills" appears in the text candidate knowledge, the cross-modal association retrieval will add "stone cleaning skills video" and "roadblock removal effect picture" to the video candidate knowledge and image candidate knowledge respectively.

[0087] S4. Based on the consistent semantic features, the answer generation network processes the candidate knowledge of each modality to obtain a multimodal answer. Specifically, it includes:

[0088] S401. Input the consistent semantic features into the pre-trained large language model to predict the modal type that will appear in the next position in the answer.

[0089] S402: Determine the corresponding candidate knowledge and generate a multimodal answer based on the consistent semantic features and the predicted modality type.

[0090] Specifically, multimodal answer generation aims to generate the final answer based on the retrieved candidate knowledge from each modality. This is achieved by attaching an answer generation network to each modality's output head. The answer generation network is a pre-trained large language model (such as GPT or LLaMA), which takes as input the semantic features of the question and predicts the modality type that should appear next in the answer. Based on the answer generation network's prediction, the output head for the corresponding modality is selected, receives the semantic features and candidate knowledge from the corresponding modality, and outputs the answer.

[0091] For example, when a partial answer has been generated: "The highway needs to be cleared of gravel, and the expected effect is as shown in the figure below", the answer generation network predicts that the next position of the answer should be the image mode, and the output head corresponding to the image mode starts working to generate an image at the next position of the existing answer.

[0092] Since a single round of question and answer cannot usually accurately solve the user's problem, as an implementation method, multiple rounds of dialogue and question and answer (such as Figure 1 ), and after each answer is generated, a feedback option "Did it solve your question?" is added. If the user continues to ask, the process jumps to S2 and repeats S2-S4. If the user selects "Solved" or "Unsolved," the Q&A session is exited, and the system marks the reward for this Q&A chain as 1 or 0, respectively.

[0093] Use reinforcement learning strategy to update model parameters and cross-modal association weights of weighted undirected graphs in multimodal knowledge base. t ,a t ,...,q T ,a T ,r], then the optimization goal of the system is to maximize the feedback reward, which can be expressed as:

[0094]

[0095] Among them, t and T represent the round index and total number of rounds of question and answer respectively, q t Indicates the user question in round t, a t represents the model answer in round t, r(q t ,a t ) represents the corresponding feedback reward when the user asks the question in round t and the model answers the question in round t, for example, r(q t ,a t) can score the corresponding user when the user asks a question in round t and the model answers in round t. If the user chooses "solved", the corresponding user score is 1. If the user chooses "unsolved", the corresponding user score is 0.

[0096] Specifically, the cross-modal association weights can be modeled as a set of trainable parameters a. When constructing a multimodal knowledge base, the weights are initialized, i.e., ω is set to a number between [0, 1]. When the question-answering process is complete, user feedback is obtained. Based on the "maximizing feedback reward" objective function R, using the gradient descent optimization algorithm as an example, the partial derivatives of the various parameters in the model are calculated. In this case, ω is used as one of the parameters and is updated in the opposite direction of its gradient, i.e.,

[0097]

[0098] Example 2

[0099] Combine Figure 3 This embodiment discloses a highway maintenance knowledge question-answering system based on multimodal information, including:

[0100] The question understanding module is configured to: obtain user question data and convert it into multiple modal feature vectors; process the multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent semantic features;

[0101] The knowledge retrieval module is configured to: based on complementary features and consistent semantic features, use the consistent semantic features as a guide to perform independent searches for each modality in a preset multimodal knowledge base, and use the degree of association between knowledge in different modalities to perform cross-modal association searches to obtain candidate knowledge for each modality;

[0102] The answer generation module is configured to process candidate knowledge of each modality through the answer generation network according to the consistent semantic features to obtain a multimodal answer.

[0103] It should be noted that the question understanding module, knowledge retrieval module, and answer generation module described above correspond to the steps in Example 1. The examples and application scenarios implemented by these modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0104] Example 3

[0105] A third embodiment of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information are completed.

[0106] Example 4

[0107] A fourth embodiment of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information are completed.

[0108] Example 5

[0109] A fifth embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned highway maintenance knowledge question-answering method based on multimodal information.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0113] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A highway maintenance knowledge question-answering method based on multimodal information, characterized by: include: Obtain user question data and convert it into multiple modal feature vectors; The multi-modal feature vectors are processed through a preset multi-modal decoupling network to obtain complementary features and consistent semantic features; Based on complementary features and consistent semantic features, each modality is searched independently in the preset multimodal knowledge base with the consistent semantic features as the guide, and cross-modal association search is performed using the correlation degree between different modal knowledge to obtain candidate knowledge of each modality; Based on the consistent semantic features, the answer generation network processes the candidate knowledge of each modality to obtain multimodal answers; The independent retrieval of each modality in a preset multimodal knowledge base based on complementary features and consistent semantic features and guided by consistent semantic features is specifically as follows: calculating the correlation between the complementary features and consistent semantic features and the knowledge of the same modality type in the multimodal knowledge base, and screening and determining the candidate knowledge according to the correlation degree.

2. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: Before obtaining user question data, it also includes: Expressway maintenance knowledge is acquired, the expressway maintenance knowledge is stored in a single mode according to the modality type, and cross-modal associations between different modalities are established to form a multimodal knowledge base.

3. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The multimodal decoupling network includes multiple parallel multi-layer perceptrons and shared perceptrons; The multi-layer perceptron is used to process a single modal feature vector to obtain complementary features; the shared perceptron is used to process multiple concatenated modal feature vectors to obtain consistent semantic features.

4. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The cross-modal association retrieval using the degree of association between different modal knowledge is specifically as follows: traversing the candidate knowledge determined by independent retrieval, and selecting other modal knowledge with the highest degree of association therewith.

5. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1, characterized in that: The method of processing candidate knowledge of each modality through the answer generation network based on the consistent semantic features to obtain a multimodal answer specifically includes: Input the consistent semantic features into the pre-trained large language model to predict the modality type that appears next in the answer; According to the consistent semantic features and the predicted modality type, the corresponding candidate knowledge is determined and a multimodal answer is generated.

6. Highway maintenance knowledge question-answering system based on multimodal information, characterized by: include: The question understanding module is configured to: obtain user question data and convert it into multiple modal feature vectors; process the multiple modal feature vectors through a preset multimodal decoupling network to obtain complementary features and consistent semantic features; The knowledge retrieval module is configured to: based on complementary features and consistent semantic features, use the consistent semantic features as a guide to perform independent searches for each modality in a preset multimodal knowledge base, and use the degree of association between knowledge in different modalities to perform cross-modal association searches to obtain candidate knowledge for each modality; The answer generation module is configured to: process candidate knowledge of each modality through the answer generation network according to the consistent semantic features to obtain a multimodal answer; The independent retrieval of each modality in a preset multimodal knowledge base based on complementary features and consistent semantic features and guided by consistent semantic features is specifically as follows: calculating the correlation between the complementary features and consistent semantic features and the knowledge of the same modality type in the multimodal knowledge base, and screening and determining the candidate knowledge according to the correlation degree.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the highway maintenance knowledge question-answering method based on multimodal information as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the highway maintenance knowledge question-answering method based on multimodal information as described in any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the highway maintenance knowledge question-answering method based on multimodal information as described in any one of claims 1 to 5 are implemented.

Images