Small sample traffic schematic diagram question and answer method and device based on multi-modal context

Abstract

The invention discloses a multi-modal context-based small sample traffic schematic diagram question and answer method and device, and solves the problem that traffic schematic diagrams are difficult to characterize under the conditions of limited sample scale and semantic confusion, and the method comprises the following steps: obtaining a sample set containing a plurality of traffic schematic diagrams and a plurality of question texts; extracting visual area features, block features and block description features of the traffic schematic diagram and lexical features of the problem text, and constructing a multi-modal feature data set; and training a pre-constructed traffic schematic diagram question-answering model by using the multi-modal feature data set to obtain a trained traffic schematic diagram question-answering model, then screening the trained traffic schematic diagram question-answering model, deploying the screened traffic schematic diagram question-answering model on electronic equipment to answer a traffic schematic diagram question to be answered, and displaying the traffic schematic diagram question to a user. According to the method, the accuracy of traffic schematic diagram questions and answers is guaranteed under the condition that visual semantic confusion exists in the samples, and good technical support is provided for applications such as multi-modal retrieval and automatic driving in an intelligent traffic scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal traffic scene understanding, and specifically relates to a small sample traffic diagram question and answer method and device based on multimodal context. Background Art

[0002] Traffic diagram question answering aims to capture the association between traffic diagrams and question texts, and plays an important role in applications such as traffic knowledge retrieval and intelligent driving decision recommendation. Current traditional multimodal representation methods tend to learn common high-frequency content in vision and text, while ignoring the difficulty of representing traffic diagrams in the presence of limited sample size and semantic confusion. Traffic diagrams contain professional traffic domain knowledge, and there is visual-semantic confusion in traffic diagrams. Traditional methods cannot be directly applied to traffic diagram representation.

[0003] Therefore, there is an urgent need for a traffic diagram question-answering method that can solve the semantic confusion problem of traffic diagrams in small sample scenarios. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a small sample traffic diagram question answering method and device based on multimodal context. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] In a first aspect, the present invention provides a small sample traffic diagram question answering method based on multimodal context, comprising:

[0006] S100, obtaining a sample set including a plurality of traffic schematic diagrams and a plurality of question texts; each traffic schematic diagram in the sample set corresponds to at least one question text;

[0007] S200, extracting visual area features and visual block features of each traffic schematic diagram in the sample set, and extracting word element features of each question text in the sample set, and constructing a multimodal feature data set using the visual area features, visual block features and word element features; wherein the visual block features include block features and block description features;

[0008] S300, training a pre-built traffic diagram question-answering model using the multimodal feature data set to obtain a trained traffic diagram question-answering model;

[0009] S400, screening the trained traffic diagram question-answering model, and after screening, deploying it on an electronic device to answer the traffic diagram questions to be answered and displaying it to the user.

[0010] In a second aspect, the present invention provides a small sample traffic diagram question-answering device based on multimodal context, comprising:

[0011] An acquisition module is configured to acquire a sample set including a plurality of traffic diagrams and a plurality of question texts;

[0012] The extraction module is configured to extract the visual area features and visual block features of each traffic schematic diagram in the sample set, and extract the word element features of each question text in the sample set, and construct a multimodal feature data set using the visual area features, visual block features and word element features; wherein the visual block features include block features and block description features;

[0013] A training module is configured to train a pre-built traffic diagram question-answering model using the multimodal feature data set to obtain a trained traffic diagram question-answering model;

[0014] The question-answering module is configured to screen the trained traffic diagram question-answering model, and after the screening is completed, it is deployed on the electronic device to answer the traffic diagram questions to be answered and display them to the user.

[0015] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0016] (1) The present invention designs a contextual interactive question representation network and an enhanced fine-grained enhanced traffic diagram representation network. With the help of the block description generated by the traffic diagram itself, the semantic knowledge contained in the traffic diagram is enriched without the need for additional annotation costs. Through the fine-grained feature interaction between the traffic diagram and the question text, it helps to mine the key content in the traffic diagram and the question text while ignoring irrelevant background noise, thereby improving the accuracy of the traffic diagram question answering model.

[0017] (2) The present invention designs an adaptive question attention dispersion mechanism, which uses the word-unit features of the question text as a guide to screen the parts of the traffic diagram blocks and block descriptions that are most relevant to the current word-unit. Based on this, it is determined whether the attention of the question text on the traffic diagram should be concentrated or dispersed. This helps the model to adaptively adjust the learning of difficult questions (attention dispersion) and simple questions (attention concentration), and ensures the accuracy of traffic diagram question and answering when there is visual semantic confusion in the sample.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a small sample traffic diagram question-answering method based on multimodal context provided by the present invention;

[0020] Figure 2This is an example of annotation of a sample set of traffic diagrams provided by the present invention;

[0021] Figure 3 It is a training diagram of a small sample traffic diagram question-answering model based on multimodal context provided by the present invention;

[0022] Figure 4 It is a schematic diagram of a small sample traffic diagram question and answer device based on multimodal context provided by the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0024] First, as Figure 1 As shown, the present invention provides a small sample traffic diagram question answering method based on multimodal context, including:

[0025] S100, obtaining a sample set including a plurality of traffic schematic diagrams and a plurality of question texts; each traffic schematic diagram in the sample set corresponds to at least one question text;

[0026] For example, refer to Figure 2 ,Firstly, any traffic diagram is selected and the corresponding question text is ,labeled for each traffic diagram to make a sample set of traffic diagrams. ,The sample set contains 1100 traffic diagrams, and each traffic diagram corresponds to 1 to 3 question texts.

[0027] S200, extracting visual area features and visual block features of each traffic schematic diagram in the sample set, and extracting word element features of each question text in the sample set, and constructing a multimodal feature data set using the visual area features, visual block features and word element features; wherein the visual block features include block features and block description features;

[0028] For example, through the three-dimensional traffic schematic diagram, the visual area feature F of the traffic schematic diagram is extracted R , block feature F P , block description feature F C ; Extract the word features F of the question text through the question text W ; Then combine all features to build a multimodal feature dataset.

[0029] S300, training a pre-built traffic diagram question-answering model using the multimodal feature data set to obtain a trained traffic diagram question-answering model;

[0030] S400, screening the trained traffic diagram question-answering model, and after screening, deploying it on an electronic device to answer the traffic diagram questions to be answered and displaying it to the user.

[0031] As a specific implementation of the present invention, S200 includes:

[0032] S210, dividing each traffic schematic diagram in the sample set into blocks of the same size, and using a pre-trained convolutional network to obtain block features of each block;

[0033] S220, uses the pre-trained Faster-rcnn model to obtain the visual area features of each traffic diagram;

[0034] S230, using a pre-trained description generation model to obtain a tile description corresponding to each tile, and for each tile description, using a pre-trained BERT model to obtain a tile description feature from the tile description;

[0035] S240, using the pre-trained BERT model to obtain the word features of each question text.

[0036] refer to Figure 3 , Figure 3 The figure is a schematic diagram of the construction process of the traffic diagram question-answering model of the present invention. S300 of the present invention includes:

[0037] S310, defining the contextual interactive question representation network, the fine-grained enhanced traffic diagram representation network, the multimodal question fusion network, the answer selector and the optimization loss function in the traffic diagram question answering model;

[0038] It is worth noting that the contextual interaction problem representation network, fine-grained enhanced traffic diagram representation network, and multimodal problem fusion network of the present invention can all be implemented based on existing network structures. For example, the contextual interaction problem representation network and the fine-grained enhanced traffic diagram representation network are implemented using the Transformer architecture, and the multimodal problem fusion network is implemented using the cross-attention network architecture.

[0039] S320, in the current iteration, selecting a visual block feature of a traffic diagram and a word feature of a corresponding question text from the multimodal feature data set as input, and inputting them into the context interaction question representation network to obtain a word feature enhanced by context interaction, a global question feature, and a visual block feature enhanced by word features;

[0040] S330, inputting the word-unit features enhanced by the contextual interaction, the global question features, the word-unit enhanced visual block features and the visual area features of the traffic schematic diagram in S320 into the fine-grained enhanced traffic schematic diagram representation network to perform representation enhancement of the traffic schematic diagram, thereby obtaining fine-grained enhanced visual area features and global traffic schematic diagram features;

[0041] S340, inputting the fine-grained enhanced visual area features, the global traffic schematic diagram features, the context-interaction-enhanced word-unit features, and the global question features into the multimodal question fusion network to obtain a fused multimodal traffic schematic diagram question feature;

[0042] S350, inputting the fused multimodal traffic diagram question features into the answer selector to obtain a probability distribution of candidate answers;

[0043] S360, calculating the loss value of the input traffic diagram question and answer using the optimized loss function according to the probability distribution described in S350;

[0044] S370, using the loss value to adjust the weight parameters of the traffic diagram question answering model, and reselecting a visual area feature, a visual block feature, and a word feature of a corresponding question text of a traffic diagram in the multimodal feature dataset as input;

[0045] S380, repeat S320 to S380 until the iteration end condition is reached, and a trained traffic diagram question-answering model is obtained.

[0046] As a specific implementation of the present invention, S320 includes:

[0047] S321, in the current iteration, a visual block feature of a traffic diagram and a word feature of a corresponding question text are selected from the multimodal feature data set as input, and are input into the contextual interactive question representation network, so that the contextual interactive question representation network uses the word feature as context, selects the block feature most relevant to the word feature, and uses it as the word-enhanced block feature f WP ; Calculate the attention distribution Att between the word feature and the tile feature WP , and calculate the mean by column to obtain the first attention vector of the question on the tile

[0048]

[0049] Among them, W W1 and W P are two learnable parameter matrices, AvgPool is the average pooling operation, and d represents the mapping channel dimension.

[0050] S322, the context interaction problem representation network uses the word element feature as context, selects the tile description feature most relevant to the word element feature, and uses it as the word element enhanced tile description feature F WC Calculate the attention distribution Att between the word feature and the tile description feature WC , calculate the mean by column to obtain the second attention vector of the description of the question on the tile

[0051]

[0052] Among them, W W2 and W C are two learnable parameter matrices, AvgPool is the average pooling operation, and d represents the mapping channel dimension;

[0053] S323, calculating the index i of the maximum value in the first attention vector and the index j of the maximum value in the second attention vector;

[0054] S324, if the index value i=j, it indicates that the question text focuses on a specific content in the traffic diagram, and the cascade features of the tile feature corresponding to the index value i and the tile description feature corresponding to the index value j are used as the key and value, and the word-unit feature is used as the query to obtain the word-unit feature enhanced by context interaction; then the word-unit feature enhanced by context interaction is averaged and pooled to obtain the global question feature F of the question text. Q ;

[0055] S325, if the index value i≠j, it indicates that the question text focuses on multiple contents in the traffic diagram, and the cascade features of all the tile features and all the tile description features are used as keys and values, and the word-unit features are used as queries to obtain word-unit features enhanced by context interaction; and then the word-unit features enhanced by context interaction are average pooled to obtain the global question feature F of the question text. Q .

[0056] As a specific implementation of the present invention, S330 includes:

[0057] S331, inputting the fine-grained enhanced visual area features, the global traffic schematic diagram features, the word-unit features enhanced by contextual interaction, and the global question features into the multimodal question fusion network, so that the multimodal question fusion network uses the visual area features of the traffic schematic diagram as a query, performs a self-attention operation and filters out its own background noise, thereby obtaining self-attention enhanced visual area features;

[0058] S332, using the self-attention enhanced visual area feature as a query and the context interaction enhanced word element feature as a key and a value, obtaining a word element feature enhanced visual area feature F WR ;

[0059] S333, using the word unit to enhance the block feature F WP and word-enhanced tile description features F WC As input, the semantics of the block features of the traffic diagram are refined through dot product operations to obtain the multimodal block features F of the traffic diagram PC ;

[0060] S334, visual area feature F enhanced by the word element feature WR As a query, the multimodal patch feature F PC As keys and values, we obtain the visual region features enhanced by fine-grained patches;

[0061] S335, performing an average pooling operation on the visual area features enhanced by the fine-grained image blocks to obtain a global traffic diagram feature F D .

[0062] As a specific implementation of the present invention, S340 includes:

[0063] S341, defining the number of layers L of the multimodal question fusion network and the number of heads H of the multi-head attention;

[0064] S342, the global problem feature F Q and the global traffic diagram feature F D Perform dot product operation to obtain the characteristics of multimodal traffic diagram problem;

[0065] S343, using the multimodal traffic diagram problem feature as a query, the word feature enhanced by the context interaction and the visual area feature enhanced by the fine-grained tile in S334 as context, to obtain a fused multimodal traffic diagram problem feature.

[0066] As a specific implementation of the present invention, S350 includes:

[0067] The fused multimodal traffic diagram question features are input into the answer selector, so that the answer selector performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers.

[0068] As a specific implementation of the present invention, S360 includes:

[0069] S361, input the probability distribution of the candidate answers into the optimization loss function, and calculate the loss value of the input traffic diagram question and answer, which is expressed as:

[0070]

[0071] Where N represents the number of question texts corresponding to the input traffic diagram, C represents the number of candidate answers, and y ij represents the true answer distribution of the input question text, represents the predicted answer distribution.

[0072] As a specific implementation of the present invention, S400 includes:

[0073] S410, obtaining a test traffic diagram and a test question text, and answering the test traffic diagram and the test question text using the trained traffic diagram question-answering model to obtain answers corresponding to the test traffic diagram and the test question text;

[0074] S420, evaluating the performance of the traffic diagram question answering model using the test traffic diagram and the answer corresponding to the test question text;

[0075] S430, using the traffic diagram question-answering model whose performance meets the conditions as the screened traffic diagram question-answering model;

[0076] S440, deploying the screened traffic diagram question-answering model on an electronic device to answer the traffic diagram question to be answered and displaying it to the user.

[0077] Second, reference Figure 4 The present invention provides a small sample traffic diagram question-answering device based on multimodal context, comprising:

[0078] An acquisition module is configured to acquire a sample set including a plurality of traffic diagrams and a plurality of question texts;

[0079] The extraction module is configured to extract the visual area features and visual block features of each traffic schematic diagram in the sample set, and extract the word element features of each question text in the sample set, and construct a multimodal feature data set using the visual area features, visual block features and word element features; wherein the visual block features include block features and block description features;

[0080] A training module is configured to train a pre-built traffic diagram question-answering model using the multimodal feature data set to obtain a trained traffic diagram question-answering model;

[0081] The question-answering module is configured to screen the trained traffic diagram question-answering model, and after the screening is completed, it is deployed on the electronic device to answer the traffic diagram questions to be answered and display them to the user.

[0082] The present invention discloses a small sample traffic diagram question-answering method and device based on multimodal context, which solves the problem of difficulty in representing traffic diagrams when the sample size is limited and there is semantic confusion, including: obtaining a sample set containing multiple traffic diagrams and multiple question texts; then extracting the visual area features, block features, block description features and word features of the traffic diagrams, and then constructing a multimodal feature data set; using the multimodal feature data set to train a pre-constructed traffic diagram question-answering model to obtain a trained traffic diagram question-answering model, and then screening it. After the screening is completed, it is deployed on an electronic device to answer the traffic diagram questions to be answered and display them to the user. The present invention achieves the accuracy of traffic diagram question-answering in the presence of visual semantic confusion in the sample, and provides good technical support for multimodal retrieval, automatic driving and other applications in intelligent traffic scenarios.

[0083] It is worth noting that the terms "first" and "second" in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0084] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.

[0085] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A small sample traffic diagram question answering method based on multimodal context, characterized in that: include: S100, obtaining a sample set including a plurality of traffic schematic diagrams and a plurality of question texts; Each traffic diagram in the sample set corresponds to at least one question text; S200, extracting visual area features and visual block features of each traffic schematic diagram in the sample set, and extracting word element features of each question text in the sample set, and constructing a multimodal feature data set using the visual area features, visual block features and word element features; wherein the visual block features include block features and block description features; S300, training a pre-built traffic diagram question-answering model using the multimodal feature data set to obtain a trained traffic diagram question-answering model; S400, screening the trained traffic diagram question-answering model, and after screening, deploying it on an electronic device to answer the traffic diagram questions to be answered and displaying it to the user.

2. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: S200 includes: S210, dividing each traffic schematic diagram in the sample set into blocks of the same size, and using a pre-trained convolutional network to obtain block features of each block; S220, uses the pre-trained Faster-rcnn model to obtain the visual area features of each traffic diagram; S230, using a pre-trained description generation model to obtain a tile description corresponding to each tile, and for each tile description, using a pre-trained BERT model to obtain a tile description feature from the tile description; S240, using the pre-trained BERT model to obtain the word features of each question text.

3. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: S300 includes: S310, defining the contextual interactive question representation network, the fine-grained enhanced traffic diagram representation network, the multimodal question fusion network, the answer selector and the optimization loss function in the traffic diagram question answering model; S320, in the current iteration, selecting a visual block feature of a traffic diagram and a word feature of a corresponding question text from the multimodal feature data set as input, and inputting them into the context interaction question representation network to obtain a word feature enhanced by context interaction, a global question feature, and a visual block feature enhanced by word features; S330, inputting the word-unit features enhanced by the contextual interaction, the global question features, the word-unit enhanced visual block features and the visual area features of the traffic schematic diagram in S320 into the fine-grained enhanced traffic schematic diagram representation network to perform representation enhancement of the traffic schematic diagram, thereby obtaining fine-grained enhanced visual area features and global traffic schematic diagram features; S340, inputting the fine-grained enhanced visual area features, the global traffic schematic diagram features, the context-interaction-enhanced word-unit features, and the global question features into the multimodal question fusion network to obtain a fused multimodal traffic schematic diagram question feature; S350, inputting the fused multimodal traffic diagram question features into the answer selector to obtain a probability distribution of candidate answers; S360, calculating the loss value of the input traffic diagram question and answer using the optimized loss function according to the probability distribution described in S350; S370, using the loss value to adjust the weight parameters of the traffic diagram question answering model, and reselecting a visual area feature, a visual block feature, and a word feature of a corresponding question text of a traffic diagram in the multimodal feature dataset as input; S380, repeat S320 to S380 until the iteration end condition is reached, and a trained traffic diagram question-answering model is obtained.

4. The small sample traffic diagram question answering method based on multimodal context according to claim 3 is characterized in that: S320 includes: S321, in the current iteration, a visual block feature of a traffic diagram and a word feature of a corresponding question text are selected from the multimodal feature data set as input, and are input into the contextual interactive question representation network, so that the contextual interactive question representation network uses the word feature as context, selects the block feature most relevant to the word feature, and uses it as the word-enhanced block feature F WP ; Calculate the attention distribution Att between the word feature and the tile feature WP , and obtain the first attention vector of the question to the image block by taking the average value by column; S322, the context interaction problem representation network uses the word element feature as context, selects the tile description feature most relevant to the word element feature, and uses it as the word element enhanced tile description feature F WC Calculate the attention distribution Att between the word feature and the tile description feature WC , obtaining a second attention vector of the description of the question on the image block by taking the average value by column; S323, calculating the index i of the maximum value in the first attention vector and the index j of the maximum value in the second attention vector; S324, if the index value i=j, it indicates that the question text focuses on a specific content in the traffic diagram, and the block feature corresponding to the index value i and the index value, the cascade feature of the corresponding block description feature are used as the key and value, and the word element feature is used as the query to obtain the word element feature enhanced by context interaction; then the word element feature enhanced by context interaction is averaged and pooled to obtain the global question feature F of the question text. Q ; S325, if the index value i≠j, it indicates that the question text focuses on multiple contents in the traffic diagram, and the cascade features of all the tile features and all the tile description features are used as keys and values, and the word-unit features are used as queries to obtain word-unit features enhanced by context interaction; and then the word-unit features enhanced by context interaction are average pooled to obtain the global question feature F of the question text. Q .

5. The small sample traffic diagram question answering method based on multimodal context according to claim 3 is characterized in that: S330 includes: S331, inputting the fine-grained enhanced visual area features, the global traffic schematic diagram features, the word-unit features enhanced by contextual interaction, and the global question features into the multimodal question fusion network, so that the multimodal question fusion network uses the visual area features of the traffic schematic diagram as a query, performs a self-attention operation and filters out its own background noise, thereby obtaining self-attention enhanced visual area features; S332, using the self-attention enhanced visual area feature as a query and the context interaction enhanced word element feature as a key and a value, obtaining a word element feature enhanced visual area feature F WR ; S333, using the word unit to enhance the block feature F WP and word-enhanced tile description features F WC As input, the semantics of the block features of the traffic diagram are refined through dot product operations to obtain the multimodal block features F of the traffic diagram PC ; S334, visual area feature F enhanced by the word element feature WR As a query, the multimodal patch feature F PC As keys and values, we obtain the visual region features enhanced by fine-grained patches; S335, performing an average pooling operation on the visual area features enhanced by the fine-grained image blocks to obtain a global traffic diagram feature F D .

6. The small sample traffic diagram question answering method based on multimodal context according to claim 3 is characterized in that: S340 includes: S341, defining the number of layers L of the multimodal question fusion network and the number of heads H of the multi-head attention; S342, the global problem feature F Q and the global traffic diagram feature F D Perform dot product operation to obtain the characteristics of multimodal traffic diagram problem; S343, using the multimodal traffic diagram problem feature as a query, the word feature enhanced by the context interaction and the visual area feature enhanced by the fine-grained tile in S334 as context, to obtain a fused multimodal traffic diagram problem feature.

7. The small sample traffic diagram question answering method based on multimodal context according to claim 3 is characterized in that: S350 includes: The fused multimodal traffic diagram question features are input into the answer selector, so that the answer selector performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers.

8. The small sample traffic diagram question answering method based on multimodal context according to claim 3 is characterized in that: S360 includes: S361, input the probability distribution of the candidate answers into the optimization loss function, and calculate the loss value of the input traffic diagram question and answer, which is expressed as: Where N represents the number of question texts corresponding to the input traffic diagram, C represents the number of candidate answers, and y ij represents the true answer distribution of the input question text, represents the predicted answer distribution.

9. The small sample traffic diagram question answering method based on multimodal context according to claim 1, characterized in that: S400 includes: S410, obtaining a test traffic diagram and a test question text, and answering the test traffic diagram and the test question text using the trained traffic diagram question-answering model to obtain answers corresponding to the test traffic diagram and the test question text; S420, evaluating the performance of the traffic diagram question answering model using the test traffic diagram and the answer corresponding to the test question text; S430, using the traffic diagram question-answering model whose performance meets the conditions as the screened traffic diagram question-answering model; S440, deploying the screened traffic diagram question-answering model on an electronic device to answer the traffic diagram question to be answered and displaying it to the user.

10. A small sample traffic diagram question-answering device based on multimodal context, characterized in that: include: An acquisition module is configured to acquire a sample set including a plurality of traffic diagrams and a plurality of question texts; The extraction module is configured to extract the visual area features and visual block features of each traffic schematic diagram in the sample set, and extract the word element features of each question text in the sample set, and construct a multimodal feature data set using the visual area features, visual block features and word element features; wherein the visual block features include block features and block description features; A training module is configured to train a pre-built traffic diagram question-answering model using the multimodal feature data set to obtain a trained traffic diagram question-answering model; The question-answering module is configured to screen the trained traffic diagram question-answering model, and after the screening is completed, it is deployed on the electronic device to answer the traffic diagram questions to be answered and display them to the user.

Images