Expressway maintenance knowledge question-answering method and system based on multi-modal information

Abstract

The invention relates to the technical field of expressway intelligent maintenance, and discloses an expressway maintenance knowledge question-answering method and system based on multi-modal information, and the system comprises a multi-dimensional collection module and an intelligent management module. According to the system, all expressway maintenance knowledge and question data of all users are acquired through the multi-dimensional acquisition module and are classified to form a data set, the intelligent management module analyzes entities and connection relationships in the expressway maintenance knowledge and constructs a knowledge graph, the multi-dimensional integration accuracy is high, the intelligent management module analyzes question contents in different forms, and the knowledge graph is integrated with the expressway maintenance knowledge. Querying a corresponding target triple in the knowledge graph, preferentially returning a result with the strongest relevance, counting the query time, analyzing the reply speed and the quality score when each user answers questions and answers, evaluating the query efficiency and the query quality of the expressway maintenance knowledge, and outputting a corresponding query instruction. The intelligent and professional solution of intelligent maintenance is realized, and the knowledge query efficiency is high.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent highway maintenance technology, and specifically to a highway maintenance knowledge question-answering method and system based on multimodal information. Background Art

[0002] With the continuous growth of highway mileage, highway maintenance is becoming increasingly important. To address complex road networks and diverse maintenance needs, industry technical standards and knowledge systems are constantly being upgraded. A growing number of maintenance technical standards and implementation recommendations are available, and data is scattered across various modalities, such as text reports, sensors, and images, resulting in inefficient information integration. Traditional question-and-answer systems are unable to understand professional terminology and context, resulting in inaccurate responses, delayed maintenance knowledge updates, and a lack of dynamic learning mechanisms, making it difficult to adapt to new highway maintenance regulations in a timely manner.

[0003] Addressing these challenges presents numerous challenges. The first is the bottleneck of understanding specialized terminology. Highway maintenance involves specialized terminology such as crack types, material ratios, and process standards, and the accuracy of general NLP models may be less than 60%. Second, adapting to multimodal scenarios is difficult. Maintenance personnel often need to describe damage using images or annotate repair areas on CAD drawings, which existing single-text question-answering models are unable to handle. Third, in the process of data acquisition and knowledge base construction, there are numerous challenges, such as the fragmentation of unstructured data and regional differences in standards. Fourth, there is a lag in standard updates. Maintenance specifications are updated 3-5% annually, while the update cycle of traditional knowledge bases needs to be shortened from 3 months to 7 days. This situation makes the development of an AI-powered question-answering bot that can cover over 80% of the industry's most frequently asked questions, support dynamic knowledge updates, multiple rounds of follow-up questions and contextual understanding, and support voice and image input a critical and urgent task. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a highway maintenance knowledge question-and-answer method and system based on multimodal information, which has the advantages of high multi-dimensional integration accuracy and high knowledge query efficiency, and solves the problems of low information integration efficiency, delayed update and slow response speed in traditional highway maintenance knowledge question-and-answer systems.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solution: a highway maintenance knowledge question-answering method based on multimodal information, comprising the following steps: Step 1: Connect the database and big data platform through the network to obtain all highway maintenance knowledge and all user question data, and classify them into knowledge dataset and question dataset; Step 2: Based on the professional data set, analyze the entities and connection relationships in the highway maintenance knowledge and build the corresponding knowledge graph ; Step 3: Analyze different forms of question content based on the question dataset and generate Query the corresponding target triples in , and count the corresponding query duration ; Step 4: Based on the question dataset and query duration , analyze the response speed of each user when answering questions and quality rating ; Step 5: Set a fixed speed threshold , combined with the response speed and quality rating ,evaluate the query efficiency and query quality of highway maintenance knowledge, and output corresponding query instructions.

[0006] Preferably, in step 1, the knowledge data set includes maintenance specifications, technical standards, operation manuals, historical maintenance reports, inspection reports, expert experience documents and research papers.

[0007] Preferably, in step 2, the question data set includes text content, voice content and image content of each user.

[0008] Preferably, in step 2, the knowledge graph The build process is as follows: S11. Use natural language processing technology to identify entities in the knowledge dataset, including roadbeds, pavements, bridges, tunnels, safety facilities, green plants, and emergency supplies; S12. Count the number of entities in the knowledge dataset, and then set corresponding nodes based on each entity, with the number of nodes being consistent with the number of entities; S13. Analyze the connection relationship between each entity in the knowledge data set through natural language processing technology, and then set corresponding edges based on the connection relationship between each entity; S14: Integrate triples using natural language processing technology, arrange all entities and connections in S11 to S13 in the form of "entity → relationship → entity", and then save them in the form of "node → edge → node". If a single entity has a connection relationship with multiple entities, they are arranged in the form of "entity → relationship → entity → relationship → entity" and then saved in the form of "node → edge → node → edge → node"; Knowledge Graph It consists of several triplets.

[0009] Preferably, in step 3, the target triplet The query process is as follows: S21. According to the question dataset, extract the The question data of each user, and Each time a user asks a question, it is marked as , to Indicates the Users from the first time to The content of the questions asked; S22. According to The input form of the user's question content is used to query the corresponding target triples , as follows: Jordi The first question asked by a user For textual questions, natural language processing techniques should be used to identify entities in the question content; Jordi The first question asked by a user For voice-based questions, voice processing software should be used to identify entities in the input question content; Jordi The first question asked by a user If the question is in image form, image processing software should be used to identify entities in the input question content; The first The first time user asked a question Entities in Question content The entities in ; S23. In the knowledge graph , query all containing entities The triples of Arrange the numbers from most to least and mark them as , to Represented in the knowledge graph In the first to Group contains entities The first set of triples Contains entities The first group of triples has the largest number of The target triplet .

[0010] Preferably, in step 4, the recovery speed The calculation process is as follows:

[0011] In the formula, Indicates the The query duration of each user, Indicates the Number of questions asked by users, Reply The time required for each question by each user is the The speed of reply when users ask questions .

[0012] Preferably, in step 4, the quality score The evaluation process is as follows: Jordi The first question asked by a user Entities in To Question content Entities in are all different, indicating the output target triples Accurately meet the user's query needs for highway maintenance knowledge, the first Quality rating of questions and answers by users Score 1 point; Jordi The first question asked by a user Entities in To Question content Entities in All the same, indicating the output target triples It does not accurately meet the user's query needs for highway maintenance knowledge. Quality rating of users' Q&A Score 0 points.

[0013] Preferably, in step five, the recovery speed ≥Speed ​​threshold , it indicates that the query efficiency is low, and the user should be advised to simplify the input content to narrow the knowledge graph. The query scope in .

[0014] Preferably, in step 5, quality scoring When the score is 0, it means the query quality is low and natural language processing technology should be used to mine synonymous entities to expand the knowledge graph. The query scope in .

[0015] A highway maintenance knowledge question-answering system based on multimodal information, including a multi-dimensional acquisition module and an intelligent management module; The multi-dimensional acquisition module is composed of a professional data unit and a user data unit. The professional data unit collects a knowledge data set through a network connection to a database. The knowledge data set includes all highway maintenance knowledge. The user data unit collects a question data set through a network connection to a big data platform. The question data set includes question data from all users. The intelligent management module consists of a graph analysis unit, a knowledge query unit, a response evaluation unit, and a question-answer management unit. The graph analysis unit analyzes the entities and connection relationships in the highway maintenance knowledge based on professional data sets and constructs the corresponding knowledge graph. The knowledge query unit analyzes different forms of question content based on the question data set and generates Query the corresponding target triples in , and count the corresponding query duration The response evaluation unit is based on the question data set and query duration , analyze the response speed of each user when answering questions and quality rating The question and answer management unit is set with a fixed value speed threshold , combined with the response speed and quality rating ,evaluate the query efficiency and query quality of highway maintenance knowledge, and output corresponding query instructions.

[0016] Compared with the existing technology, the present invention provides a highway maintenance knowledge question-answering method and system based on multimodal information, which has the following beneficial effects: 1. The present invention connects the database and the big data platform through a multi-dimensional acquisition module network to obtain all highway maintenance knowledge and all user question data, and classifies them into knowledge data sets and question data sets. The intelligent management module analyzes the entities and connection relationships in the highway maintenance knowledge based on the professional data sets and constructs the corresponding knowledge graph. , converting unstructured text into a semantic association network, upgrading the query from "keyword matching" to "semantic association search", greatly improving the accuracy, effectively solving the problems of scattered and difficult retrieval of traditional maintenance knowledge, avoiding users' repeated query operations between different systems, and significantly reducing data acquisition time, with high multi-dimensional integration accuracy.

[0017] 2. The present invention uses the intelligent management module to analyze the content of different forms of questions based on the question data set, and Query the corresponding target triples in , prioritize returning the most relevant results and count the corresponding query duration , effectively adapting to the disease identification needs of on-site staff, lowering the usage threshold, and the intelligent management module based on the question data set and query time , analyze the response speed of each user when answering questions and quality rating , set a fixed value speed threshold , combined with the response speed and quality rating , evaluates the query efficiency and query quality of highway maintenance knowledge, and outputs corresponding query instructions, realizing intelligent and professional answers to smart maintenance, with high knowledge query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a step diagram of the method of the present invention; Figure 2 This is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] Since the traditional highway maintenance knowledge question-answering system has low information integration efficiency, delayed update and slow response speed, please refer to Figure 1-Figure 2 The present invention provides a highway maintenance knowledge question-answering method and system based on multimodal information, specifically as follows: The highway maintenance knowledge question answering method based on multimodal information includes the following steps: Step 1: Connect the database and big data platform through the network to obtain all highway maintenance knowledge and all user question data, and classify them into knowledge dataset and question dataset; The knowledge dataset includes maintenance specifications, technical standards, operation manuals, historical maintenance reports, inspection reports, expert experience documents, and research papers; Specifically, the data attributes of highway maintenance specifications, technical standards and operation manuals are generally structured data, while the data attributes of highway maintenance history reports, inspection reports, expert experience documents and research papers are generally unstructured data. The use of natural language processing technology to integrate and analyze structured data and unstructured data ensures the knowledge graph compatibility; The question dataset includes the text content, voice content, and image content of each user; Step 2: Based on the professional data set, analyze the entities and connection relationships in the highway maintenance knowledge and build the corresponding knowledge graph ; Knowledge Graph The build process is as follows: S11. Use natural language processing technology to identify entities in the knowledge dataset, including roadbeds, pavements, bridges, tunnels, safety facilities, green plants, and emergency supplies; S12. Count the number of entities in the knowledge dataset, and then set corresponding nodes based on each entity, with the number of nodes being consistent with the number of entities; S13. Analyze the connection relationship between each entity in the knowledge data set through natural language processing technology, and then set corresponding edges based on the connection relationship between each entity; S14: Integrate triples using natural language processing technology, arrange all entities and connections in S11 to S13 in the form of "entity → relationship → entity", and then save them in the form of "node → edge → node". If a single entity has a connection relationship with multiple entities, they are arranged in the form of "entity → relationship → entity → relationship → entity" and then saved in the form of "node → edge → node → edge → node"; Knowledge Graph It is composed of several triplets; Specifically, knowledge graph Converting unstructured text into a semantic association network upgrades queries from "keyword matching" to "semantic association search," significantly improving accuracy and effectively resolving the issues of fragmented and difficult retrieval of traditional maintenance knowledge. This avoids users from having to perform repeated queries across different systems, significantly reduces data acquisition time, and offers high accuracy in multi-dimensional integration. Step 3: Analyze different forms of question content based on the question dataset and generate Query the corresponding target triples in , and count the corresponding query duration ; Target triplet The query process is as follows: S21. According to the question dataset, extract the The question data of each user, and Each time a user asks a question, it is marked as , to Indicates the Users from the first time to The content of the questions asked; S22. According to The input form of the user's question content is used to query the corresponding target triples , as follows: Jordi The first question asked by a user For textual questions, natural language processing techniques should be used to identify entities in the question content; Jordi The first question asked by a user For voice-based questions, voice processing software should be used to identify entities in the input question content; Jordi The first question asked by a user If the question is in image form, image processing software should be used to identify entities in the input question content; The first The first time user asked a question Entities in Question content The entities in ; Specifically, query the corresponding target triples according to the input form of the user's question , effectively adapting to the disease identification needs of on-site staff and lowering the threshold for use; S23. In the knowledge graph , query all containing entities The triples of Arrange the numbers from most to least and mark them as , to Represented in the knowledge graph In the first to Group contains entities The first set of triples Contains entities The first group of triples has the largest number of The target triplet ; Specifically, sort the target triples according to the frequency of entity occurrence , prioritize returning the results with the strongest relevance; Step 4: Based on the question dataset and query duration , analyze the response speed of each user when answering questions and quality rating ; Response speed The calculation process is as follows:

[0021] In the formula, Indicates the The query duration of each user, Indicates the Number of questions asked by users, Reply The time required for each question by each user is the The speed of reply when users ask questions , objectively reflects the system efficiency; Quality Rating The evaluation process is as follows: Jordi The first question asked by a user Entities in To Question content Entities in are all different, indicating the output target triples Accurately meet the user's query needs for highway maintenance knowledge, the first Quality rating of users' Q&A Score 1 point; Jordi The first question asked by a user Entities in To Question content Entities in All the same, indicating the output target triples It does not accurately meet the user's query needs for highway maintenance knowledge. Quality rating of users' Q&A Score 0 points; Step 5: Set a fixed speed threshold , combined with the response speed and quality rating ,Evaluate the query efficiency and query quality of highway maintenance knowledge, and output corresponding query instructions; Response speed ≥Speed ​​threshold When , it indicates that the query efficiency is low, and users should be advised to simplify the input content and focus on core entities to narrow the knowledge graph. The query scope in ; Quality Rating When the score is 0, it means the query quality is low and natural language processing technology should be used to mine synonymous entities to expand the knowledge graph. The query scope in ; Specifically, the integrated analysis provides answers to maintenance knowledge and daily maintenance data, realizing intelligent and professional answers to smart maintenance, improving the efficiency, quality and timeliness of maintenance knowledge query, and enhancing user experience and work efficiency.

[0022] A highway maintenance knowledge question-answering system based on multimodal information, including a multi-dimensional acquisition module and an intelligent management module; The multi-dimensional acquisition module consists of a professional data unit and a user data unit. The professional data unit connects to the database via the network to collect the knowledge data set, which includes all highway maintenance knowledge. The user data unit connects to the big data platform via the network to collect the question data set, which includes all users' question data. The intelligent management module consists of a graph analysis unit, a knowledge query unit, a response evaluation unit, and a question-answer management unit. The graph analysis unit analyzes the entities and connection relationships in highway maintenance knowledge based on professional data sets and constructs the corresponding knowledge graph. , the knowledge query unit analyzes different forms of question content based on the question dataset, and Query the corresponding target triples in , and count the corresponding query duration ,Multi-dimensional integration has high accuracy, and the response evaluation unit is based on the question dataset and query duration , analyze the response speed of each user when answering questions and quality rating , the Q&A management unit sets a fixed value speed threshold , combined with the response speed and quality rating ,Evaluate the query efficiency and query quality of highway maintenance knowledge, and output corresponding query instructions,with high knowledge query efficiency.

[0023] Example 1 In this experiment, a user who asked questions 10 times within an hour was selected as the experimental subject. According to statistics, the query time of the highway maintenance knowledge question-answering system when the user asked questions was The response speed is 110 seconds. The calculation process is as follows:

[0024] In the formula, Indicates the time required to respond to each question of the user, that is, the response speed of the user when answering questions , speed threshold Set to 10 seconds. After judgment, when the user asks questions, the highway maintenance knowledge question answering system responds quickly. ≥Speed ​​threshold , indicating that the query efficiency is low, the user should be advised to simplify the input content to narrow the knowledge graph The query scope in .

[0025] Example 2 In this experiment, the Technical Specifications for Highway Asphalt Pavement Maintenance was selected as the experimental object, and the knowledge graph The build process is as follows: S11. Use natural language processing technology to identify entities in the "Technical Specifications for Highway Asphalt Pavement Maintenance," including roadbed, pavement, bridges, tunnels, safety facilities, and emergency supplies. S12. Count the number of entities in the "Technical Specifications for Maintenance of Highway Asphalt Pavement" and then set corresponding nodes for each entity, with the number of nodes being consistent with the number of entities; Nodes include roadbed, pavement, bridges, tunnels, safety facilities and emergency supplies; S13. Using natural language processing technology, the connection relationship between each entity in the "Technical Specifications for Highway Asphalt Pavement Maintenance" is determined, and then corresponding edges are set based on the connection relationship between each entity; Edges include the connection between the roadbed and the road surface, the connection between the bridge and the road surface, the connection between the tunnel and the road surface, and the connection between the safety facilities and the road surface; S14: Integrate triples using natural language processing technology, arrange all entities and connections in S11 to S13 in the form of "entity → relationship → entity", and then save them in the form of "node → edge → node". If a single entity has a connection relationship with multiple entities, they are arranged in the form of "entity → relationship → entity → relationship → entity" and then saved in the form of "node → edge → node → edge → node"; The triplet includes roadbed → located at the bottom layer → road surface, bridge → located at the bottom layer → road surface, tunnel → interconnected → road surface, safety facilities → located on both sides → road surface; Knowledge Graph It consists of several triplets.

[0026] The threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by technicians in this field for each set of sample data; as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0027] The above formulas are obtained by collecting a large amount of data and performing software simulation, and a formula close to the actual value is selected. The coefficients in the formula are set by those skilled in the art according to actual conditions. The above is only a preferred specific implementation method of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A highway maintenance knowledge question-answering method based on multimodal information, characterized by: The following steps are involved: Step 1: Connect the database and big data platform through the network to obtain all highway maintenance knowledge and all user question data, and classify them into knowledge dataset and question dataset; Step 2: Based on the professional data set, analyze the entities and connection relationships in the highway maintenance knowledge and build the corresponding knowledge graph ; Step 3: Analyze different forms of question content based on the question dataset and generate the data in the knowledge graph. Query the corresponding target triplet in , and count the corresponding query duration ; Step 4: Based on the question dataset and query duration , analyze the response speed of each user when answering questions and quality rating ; Step 5: Set a fixed speed threshold , combined with the response speed and quality rating ,evaluate the query efficiency and query quality of highway maintenance knowledge, and output corresponding query instructions.

2. The highway maintenance knowledge question-answering method based on multimodal information according to claim 1 is characterized by: In step 1, the knowledge dataset includes maintenance specifications, technical standards, operation manuals, historical maintenance reports, inspection reports, expert experience documents and research papers.

3. The highway maintenance knowledge question-answering method based on multimodal information according to claim 2 is characterized by: In step 1, the question data set includes text content, voice content and image content of each user.

4. The highway maintenance knowledge question-answering method based on multimodal information according to claim 3 is characterized by: In the step 2, the knowledge graph The build process is as follows: S11. Use natural language processing technology to identify entities in the knowledge dataset, including roadbeds, pavements, bridges, tunnels, safety facilities, green plants, and emergency supplies; S12. Count the number of entities in the knowledge dataset, and then set corresponding nodes based on each entity, with the number of nodes being consistent with the number of entities; S13. Analyze the connection relationship between each entity in the knowledge data set through natural language processing technology, and then set corresponding edges based on the connection relationship between each entity; S14: Integrate triples using natural language processing technology, arrange all entities and connections in S11 to S13 in the form of "entity → relationship → entity", and then save them in the form of "node → edge → node". If a single entity has a connection relationship with multiple entities, they are arranged in the form of "entity → relationship → entity → relationship → entity" and then saved in the form of "node → edge → node → edge → node"; Knowledge Graph It consists of several triplets.

5. The highway maintenance knowledge question-answering method based on multimodal information according to claim 4 is characterized by: In step 3, the target triplet The query process is as follows: S21. Extract the first The user's question data, and Each time a user asks a question, it is marked as , to Indicates the Users from the first time to The content of the questions asked; S22. According to The input form of the user's question content is used to query the corresponding target triples , as follows: Jordi The first question asked by a user For textual questions, natural language processing techniques should be used to identify entities in the question content; Jordi The first question asked by a user For voice-based questions, voice processing software should be used to identify entities in the input question content; Jordi The first question asked by a user If the question is in image form, image processing software should be used to identify entities in the input question content; The first The first time user asked a question Entities in Question content The entities in ; S23. In the knowledge graph , query all containing entities The triples of Arrange the numbers from most to least and mark them as , to Represented in the knowledge graph In the first to Group contains entities The first set of triples Contains entities The first group of triples has the largest number of The target triplet .

6. The highway maintenance knowledge question-answering method based on multimodal information according to claim 5 is characterized by: In step 4, the response speed The calculation process is as follows: In the formula, Indicates the The query duration of each user, Indicates the Number of questions asked by users, Reply The time required for each question by each user is the The speed of reply when users ask questions .

7. The highway maintenance knowledge question-answering method based on multimodal information according to claim 6 is characterized by: In step 4, quality scoring The evaluation process is as follows: Jordi The first question asked by a user Entities in To Question content Entities in are all different, indicating the target triples to be output Accurately meet the user's query needs for highway maintenance knowledge, the first Quality rating of users' Q&A Score 1 point; Jordi The first question asked by a user Entities in To Question content Entities in All the same, indicating the output target triples It does not accurately meet the user's query needs for highway maintenance knowledge. Quality rating of users' Q&A Score 0 points.

8. The highway maintenance knowledge question-answering method based on multimodal information according to claim 7 is characterized by: Step 5: Responding to the speed ≥Speed ​​threshold , it indicates that the query efficiency is low, and the user should be advised to simplify the input content to narrow the knowledge graph. The query scope in .

9. The highway maintenance knowledge question-answering method based on multimodal information according to claim 8 is characterized by: Step 5: Quality Scoring When the score is 0, it means the query quality is low and natural language processing technology should be used to mine synonymous entities to expand the knowledge graph. The query scope in .

10. A highway maintenance knowledge question-answering system based on multimodal information, applied to the highway maintenance knowledge question-answering method based on multimodal information as claimed in any one of claims 1 to 9, characterized in that: Including multi-dimensional acquisition module and intelligent management module; The multi-dimensional acquisition module is composed of a professional data unit and a user data unit. The professional data unit collects a knowledge data set through a network connection to a database. The knowledge data set includes all highway maintenance knowledge. The user data unit collects a question data set through a network connection to a big data platform. The question data set includes question data from all users. The intelligent management module consists of a graph analysis unit, a knowledge query unit, a response evaluation unit, and a question-answer management unit. The graph analysis unit analyzes the entities and connection relationships in the highway maintenance knowledge based on professional data sets and constructs the corresponding knowledge graph. The knowledge query unit analyzes different forms of question content based on the question data set and generates Query the corresponding target triplet in , and count the corresponding query duration The response evaluation unit is based on the question data set and query duration , analyze the response speed of each user when answering questions and quality rating The question and answer management unit is set with a fixed value speed threshold , combined with the response speed and quality rating ,evaluate the query efficiency and query quality of highway maintenance knowledge, and output corresponding query instructions.

Images