Small sample traffic diagram question answering method and device based on multimodal context

Abstract

The present invention discloses a small-sample traffic diagram question-answering method and device based on multimodal context, which solves the problem of difficulty in representing traffic diagrams when the sample size is limited and semantic confusion exists. The method comprises: obtaining a sample set containing multiple traffic diagrams and multiple question texts; then extracting the visual area features, block features, block description features, and word features of the traffic diagrams, and then constructing a multimodal feature dataset; using the multimodal feature dataset to train a pre-constructed traffic diagram question-answering model to obtain a trained traffic diagram question-answering model, then screening the model, and deploying the model on an electronic device after screening to answer the traffic diagram questions to be answered and presenting them to the user. The present invention ensures the accuracy of traffic diagram question-answering in the presence of visual semantic confusion in the sample, and provides good technical support for applications such as multimodal retrieval and autonomous driving in intelligent traffic scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal traffic scene understanding, and specifically relates to a small sample traffic diagram question and answer method and device based on multimodal context. Background Art

[0002] Traffic diagram question answering aims to capture the relationship between traffic diagrams and question text, playing a vital role in applications such as traffic knowledge retrieval and intelligent driving decision-making recommendations. Current traditional multimodal representation methods tend to learn high-frequency content common in both visual and text, while ignoring the difficulty of representing traffic diagrams in the presence of limited sample size and semantic confusion. Traffic diagrams contain specialized traffic domain knowledge and suffer from visual-semantic confusion, making traditional methods incapable of direct application to traffic diagram representation.

[0003] Therefore, there is an urgent need for a traffic diagram question-answering method that can solve the semantic confusion problem of traffic diagrams in small sample scenarios. Summary of the Invention

[0004] To address the aforementioned issues in the prior art, the present invention provides a method and device for answering questions about small-sample traffic diagrams based on multimodal context. The technical issues addressed by the present invention are achieved through the following technical solutions:

[0005] In a first aspect, the present invention provides a small sample traffic diagram question answering method based on multimodal context, comprising:

[0006] S100, obtaining a sample set comprising a plurality of traffic diagrams and a plurality of question texts; each traffic diagram in the sample set corresponds to at least one question text;

[0007] S200, extracting visual area features and visual block features of each traffic diagram in the sample set, and extracting word-unit features of each question text in the sample set, and constructing a multimodal feature dataset using the visual area features, visual block features, and word-unit features; wherein the visual block features include block features and block description features;

[0008] S300, using the multimodal feature dataset to train a pre-built traffic diagram question-answering model to obtain a trained traffic diagram question-answering model;

[0009] S400: Screening the trained traffic diagram question-answering model. After screening, the model is deployed on an electronic device to answer the traffic diagram question and present it to the user.

[0010] In a second aspect, the present invention provides a small sample traffic diagram question-answering device based on multimodal context, comprising:

[0011] An acquisition module is configured to acquire a sample set including a plurality of traffic diagrams and a plurality of question texts;

[0012] an extraction module configured to extract visual area features and visual block features of each traffic diagram in the sample set, and extract word-unit features of each question text in the sample set, and construct a multimodal feature dataset using the visual area features, visual block features, and word-unit features; wherein the visual block features include block features and block description features;

[0013] a training module configured to train a pre-built traffic diagram question-answering model using the multimodal feature dataset to obtain a trained traffic diagram question-answering model;

[0014] The question-answering module is configured to screen the trained traffic diagram question-answering model. After the screening is completed, it is deployed on the electronic device to answer the traffic diagram questions to be answered and display them to the user.

[0015] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0016] (1) The present invention designs a contextual interactive question representation network and an enhanced fine-grained enhanced traffic diagram representation network. With the help of the block description generated by the traffic diagram itself, the semantic knowledge contained in the traffic diagram is enriched without the need for additional annotation costs. Through the fine-grained feature interaction between the traffic diagram and the question text, it helps to mine the key content in the traffic diagram and the question text while ignoring irrelevant background noise, thereby improving the accuracy of the traffic diagram question answering model.

[0017] (2) The present invention designs an adaptive question attention dispersion mechanism, which uses the word-element features of the question text as a guide to screen the parts of the traffic diagram blocks and block descriptions that are most relevant to the current word-element. Based on this, it is judged whether the attention of the question text on the traffic diagram should be concentrated or dispersed. This helps the model adaptively adjust the learning of difficult questions (attention dispersion) and simple questions (attention concentration), and ensures the accuracy of traffic diagram question answering in the presence of visual semantic confusion in the sample.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a small sample traffic diagram question-answering method based on multimodal context provided by the present invention;

[0020] Figure 2This is an example of the annotation of the traffic diagram sample set provided by the present invention;

[0021] Figure 3 This is a training diagram of a small sample traffic diagram question answering model based on multimodal context provided by the present invention;

[0022] Figure 4 This is a schematic diagram of a small sample traffic diagram question-answering device based on multimodal context provided by the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0024] First, as Figure 1 As shown, the present invention provides a small sample traffic diagram question answering method based on multimodal context, including:

[0025] S100, obtaining a sample set comprising a plurality of traffic diagrams and a plurality of question texts; each traffic diagram in the sample set corresponds to at least one question text;

[0026] For example, refer to Figure 2 First, we select any traffic diagram, mark the corresponding question text for each traffic diagram, and make a sample set of traffic diagrams. The sample set contains 1,100 traffic diagrams, and each traffic diagram corresponds to 1 to 3 question texts.

[0027] S200, extracting visual area features and visual block features of each traffic diagram in the sample set, and extracting word-unit features of each question text in the sample set, and constructing a multimodal feature dataset using the visual area features, visual block features, and word-unit features; wherein the visual block features include block features and block description features;

[0028] For example, the visual area feature F of the traffic diagram is extracted through the three-dimensional traffic diagram. R , block feature F P , block description feature F C ; Through the question text, extract the word element feature F of the question text W ; Then combine all features to build a multimodal feature dataset.

[0029] S300, using the multimodal feature dataset to train a pre-built traffic diagram question-answering model to obtain a trained traffic diagram question-answering model;

[0030] S400: Screening the trained traffic diagram question-answering model. After screening, the model is deployed on an electronic device to answer the traffic diagram question and present it to the user.

[0031] As a specific embodiment of the present invention, S200 includes:

[0032] S210, dividing each traffic diagram in the sample set into blocks of the same size, and obtaining block features of each block using a pre-trained convolutional network;

[0033] S220, uses the pre-trained Faster-rcnn model to obtain the visual area features of each traffic diagram;

[0034] S230, using a pre-trained description generation model to obtain a tile description corresponding to each tile, and for each tile description, using a pre-trained BERT model to obtain a tile description feature from the tile description;

[0035] S240, uses the pre-trained BERT model to obtain the word features of each question text.

[0036] refer to Figure 3 , Figure 3 FIG. 5 is a schematic diagram of a construction process of a traffic diagram question-answering model of the present invention. S300 of the present invention includes:

[0037] S310, define the contextual interactive question representation network, fine-grained enhanced traffic diagram representation network, multimodal question fusion network, answer selector and optimization loss function in the traffic diagram question answering model;

[0038] It is worth noting that the contextual interaction problem representation network, fine-grained enhanced traffic diagram representation network, and multimodal problem fusion network of the present invention can all be implemented based on existing network structures. For example, the contextual interaction problem representation network and the fine-grained enhanced traffic diagram representation network are implemented using the Transformer architecture, and the multimodal problem fusion network is implemented using the cross-attention network architecture.

[0039] S320, in the current iteration, selecting a visual block feature of a traffic diagram and a word-meta feature of a corresponding question text from the multimodal feature dataset as input, and inputting the inputs into the context-interactive question representation network to obtain a word-meta feature enhanced by context-interaction, a global question feature, and a word-meta enhanced visual block feature;

[0040] S330, inputting the context-interaction-enhanced word-unit features, the global question features, the word-unit-enhanced visual block features, and the visual area features of the traffic diagram in S320 into the fine-grained enhanced traffic diagram representation network to perform traffic diagram representation enhancement, thereby obtaining fine-grained enhanced visual area features and global traffic diagram features;

[0041] S340: Inputting the fine-grained enhanced visual area features, the global traffic diagram features, the context-interaction-enhanced word features, and the global question features into the multimodal question fusion network to obtain a fused multimodal traffic diagram question feature.

[0042] S350, inputting the fused multimodal traffic diagram question features into the answer selector to obtain a probability distribution of candidate answers;

[0043] S360, calculating a loss value of the input traffic diagram question and answer using the optimized loss function according to the probability distribution described in S350;

[0044] S370, using the loss value to adjust the weight parameters of the traffic diagram question answering model, and reselecting the visual area features, visual block features, and word features of the corresponding question text of a traffic diagram from the multimodal feature dataset as input;

[0045] S380, repeat S320 to S380 until the iteration end condition is reached, and obtain a trained traffic diagram question-answering model.

[0046] As a specific implementation of the present invention, S320 includes:

[0047] S321, in the current iteration, select a visual block feature of a traffic diagram and a word element feature of the corresponding question text from the multimodal feature dataset as input, and input them into the context interaction question representation network, so that the context interaction question representation network uses the word element feature as context, selects the block feature most relevant to the word element feature, and uses it as the word element enhanced block feature f WP ; Calculate the attention distribution Att between the word feature and the tile feature WP , and calculate the mean by column to obtain the first attention vector of the question on the tile

[0048]

[0049] Among them, W W1 and W P are two learnable parameter matrices, AvgPool is the average pooling operation, and d represents the mapping channel dimension.

[0050] S322, the context interaction problem representation network uses the word element feature as context, selects the tile description feature that is most relevant to the word element feature, and uses it as the word element enhanced tile description feature F WC Calculate the attention distribution Att between the word feature and the tile description feature WC , calculate the mean by column to obtain the second attention vector of the question's description of the tile

[0051]

[0052] Among them, W W2 and W C are two learnable parameter matrices, AvgPool is the average pooling operation, and d represents the mapping channel dimension;

[0053] S323, calculating the index i of the maximum value in the first attention vector and the index j of the maximum value in the second attention vector;

[0054] S324, if the index value i=j, it indicates that the question text focuses on a specific content in the traffic diagram, and the cascade features of the tile feature corresponding to the index value i and the tile description feature corresponding to the index value j are used as the key and value, and the word feature is used as the query to obtain the word feature enhanced by the context interaction; then the word feature enhanced by the context interaction is average pooled to obtain the global question feature F of the question text. Q ;

[0055] S325: If the index value i≠j, it indicates that the question text focuses on multiple contents in the traffic diagram. The cascade features of all tile features and all tile description features are used as keys and values, and the word features are used as queries to obtain word features enhanced by contextual interaction. Then, an average pooling operation is performed on the word features enhanced by contextual interaction to obtain the global question feature F of the question text. Q .

[0056] As a specific implementation of the present invention, S330 includes:

[0057] S331: Inputting the fine-grained enhanced visual area features, the global traffic diagram features, the context-interaction-enhanced word features, and the global question features into the multimodal question fusion network, so that the multimodal question fusion network uses the visual area features of the traffic diagram as a query, performs a self-attention operation, and filters out its own background noise, thereby obtaining self-attention-enhanced visual area features;

[0058] S332, using the self-attention enhanced visual area feature as a query and the context interaction enhanced word element feature as a key and value, obtain the word element feature enhanced visual area feature F WR ;

[0059] S333, using the word unit to enhance the block feature F WP and word-enhanced tile description features F WC As input, the semantics of the block features of the traffic diagram are refined through the dot product operation to obtain the multimodal block features F of the traffic diagram PC ;

[0060] S334, visual area feature F enhanced by the word element feature WR As the query, the multimodal patch feature F PC As keys and values, we obtain fine-grained patch-enhanced visual region features;

[0061] S335, performing an average pooling operation on the visual area features enhanced by the fine-grained image blocks to obtain a global traffic diagram feature F D .

[0062] As a specific implementation of the present invention, S340 includes:

[0063] S341, defining the number of layers L of the multimodal question fusion network and the number of multi-head attention heads H;

[0064] S342, the global problem feature F Q and the global traffic diagram feature F D Perform dot product operations to obtain the characteristics of the multimodal traffic diagram problem;

[0065] S343, using the multimodal traffic diagram problem feature as a query, the word feature enhanced by the context interaction and the visual area feature enhanced by the fine-grained tile in S334 as context, to obtain a fused multimodal traffic diagram problem feature.

[0066] As a specific implementation of the present invention, S350 includes:

[0067] The fused multimodal traffic diagram question features are input into the answer selector, so that the answer selector performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers.

[0068] As a specific implementation of the present invention, S360 includes:

[0069] S361: Input the probability distribution of the candidate answers into the optimization loss function to calculate the loss value of the input traffic diagram question and answer, which is expressed as:

[0070]

[0071] Among them, N represents the number of question texts corresponding to the input traffic diagram, C represents the number of candidate answers, and y ij represents the true answer distribution of the input question text, represents the predicted answer distribution.

[0072] As a specific implementation of the present invention, S400 includes:

[0073] S410, obtaining a test traffic diagram and a test question text, and answering the test traffic diagram and the test question text using the trained traffic diagram question-answering model to obtain answers corresponding to the test traffic diagram and the test question text;

[0074] S420, evaluating the performance of the traffic diagram question answering model using the test traffic diagram and the answers corresponding to the test question text;

[0075] S430, using the traffic diagram question-answering model whose performance meets the requirements as the selected traffic diagram question-answering model;

[0076] S440: deploying the filtered traffic diagram question-answering model on an electronic device to answer the traffic diagram question to be answered and displaying it to the user.

[0077] Second, reference Figure 4 The present invention provides a small sample traffic diagram question-answering device based on multimodal context, comprising:

[0078] An acquisition module is configured to acquire a sample set including a plurality of traffic diagrams and a plurality of question texts;

[0079] an extraction module configured to extract visual area features and visual block features of each traffic diagram in the sample set, and extract word-unit features of each question text in the sample set, and construct a multimodal feature dataset using the visual area features, visual block features, and word-unit features; wherein the visual block features include block features and block description features;

[0080] a training module configured to train a pre-built traffic diagram question-answering model using the multimodal feature dataset to obtain a trained traffic diagram question-answering model;

[0081] The question-answering module is configured to screen the trained traffic diagram question-answering model. After the screening is completed, it is deployed on the electronic device to answer the traffic diagram questions to be answered and display them to the user.

[0082] The present invention discloses a small-sample traffic diagram question-answering method and device based on multimodal context, which solves the problem of difficulty in representing traffic diagrams when the sample size is limited and semantic confusion exists. The method comprises: obtaining a sample set containing multiple traffic diagrams and multiple question texts; then extracting the visual area features, block features, block description features, and word features of the traffic diagrams, and then constructing a multimodal feature dataset; using the multimodal feature dataset to train a pre-constructed traffic diagram question-answering model to obtain a trained traffic diagram question-answering model, then screening the model, and deploying the model on an electronic device after screening to answer the traffic diagram questions to be answered and presenting them to the user. The present invention ensures the accuracy of traffic diagram question-answering in the presence of visual semantic confusion in the sample, and provides good technical support for applications such as multimodal retrieval and autonomous driving in intelligent traffic scenarios.

[0083] It is worth noting that the terms "first" and "second" in this disclosure are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0084] Although the present application is described herein with reference to various embodiments, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed application by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0085] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A small sample traffic diagram question answering method based on multimodal context, characterized by: include: S100, obtaining a sample set including multiple traffic diagrams and multiple question texts; Each traffic diagram in the sample set corresponds to at least one question text; S200, extracting visual area features and visual block features of each traffic diagram in the sample set, and extracting word-unit features of each question text in the sample set, and constructing a multimodal feature dataset using the visual area features, visual block features, and word-unit features; wherein the visual block features include block features and block description features; S300, using the multimodal feature dataset to train a pre-built traffic diagram question-answering model to obtain a trained traffic diagram question-answering model; S400: Screening the trained traffic diagram question-answering model. After screening, the model is deployed on an electronic device to answer the traffic diagram question and present it to the user. S300 includes: S310, define the contextual interactive question representation network, fine-grained enhanced traffic diagram representation network, multimodal question fusion network, answer selector and optimization loss function in the traffic diagram question answering model; S320, in the current iteration, selecting a visual block feature of a traffic diagram and a word-meta feature of a corresponding question text from the multimodal feature dataset as input, and inputting the inputs into the context-interactive question representation network to obtain a word-meta feature enhanced by context-interaction, a global question feature, and a word-meta enhanced visual block feature; S330, inputting the context-interaction-enhanced word-unit features, the global question features, the word-unit-enhanced visual block features, and the visual area features of the traffic diagram in S320 into the fine-grained enhanced traffic diagram representation network to perform traffic diagram representation enhancement, thereby obtaining fine-grained enhanced visual area features and global traffic diagram features; S340: Inputting the fine-grained enhanced visual area features, the global traffic diagram features, the context-interaction-enhanced word features, and the global question features into the multimodal question fusion network to obtain a fused multimodal traffic diagram question feature. S350, inputting the fused multimodal traffic diagram question features into the answer selector to obtain a probability distribution of candidate answers; S360, calculating a loss value of the input traffic diagram question and answer using the optimized loss function according to the probability distribution described in S350; S370, using the loss value to adjust the weight parameters of the traffic diagram question answering model, and reselecting the visual area features, visual block features, and word features of the corresponding question text of a traffic diagram from the multimodal feature dataset as input; S380, repeat S320 to S380 until the iteration end condition is reached, and obtain a trained traffic diagram question-answering model.

2. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: S200 includes: S210, dividing each traffic diagram in the sample set into blocks of the same size, and obtaining block features of each block using a pre-trained convolutional network; S220, uses the pre-trained Faster-rcnn model to obtain the visual area features of each traffic diagram; S230, using a pre-trained description generation model to obtain a tile description corresponding to each tile, and for each tile description, using a pre-trained BERT model to obtain a tile description feature from the tile description; S240, uses the pre-trained BERT model to obtain the word features of each question text.

3. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: The S320 includes: S321, in the current iteration, select a visual block feature of a traffic diagram and a word feature of the corresponding question text in the multimodal feature dataset as input, and input them into the context interaction question representation network, so that the context interaction question representation network uses the word feature as context, selects the block feature most relevant to the word feature, and uses it as the word-enhanced block feature ; Calculate the attention distribution between the word features and the tile features , and calculate the mean by column to obtain the first attention vector of the question on the tile; S322, the context interaction problem representation network uses the word element feature as context, selects the tile description feature that is most relevant to the word element feature, and uses it as the tile description feature enhanced by the word element Calculate the attention distribution between the word features and the tile description features , obtaining the second attention vector of the description of the question to the tile by taking the average value by column; S323, calculate the index of the maximum value in the first attention vector and the index of the maximum value in the second attention vector ; S324, if the index value , it indicates that the question text focuses on a specific content in the traffic diagram, with the index value Corresponding tile features and index values The cascade features of the corresponding tile description features are used as keys and values, and the word-unit features are used as queries to obtain the word-unit features enhanced by context interaction; then the word-unit features enhanced by context interaction are average pooled to obtain the global question features of the question text. ; S325, if the index value , it indicates that the question text focuses on multiple contents in the traffic diagram. The cascade features of all tile features and all tile description features are used as keys and values, and the word features are used as queries to obtain word features enhanced by contextual interaction. Then, the word features enhanced by contextual interaction are average pooled to obtain the global question features of the question text. .

4. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: The S330 includes: S331: Inputting the fine-grained enhanced visual area features, the global traffic diagram features, the context-interaction-enhanced word features, and the global question features into the multimodal question fusion network, so that the multimodal question fusion network uses the visual area features of the traffic diagram as a query, performs a self-attention operation, and filters out its own background noise, thereby obtaining self-attention-enhanced visual area features; S332, using the self-attention enhanced visual area features as a query and the context interaction enhanced word meta features as a key and a value, to obtain word meta feature enhanced visual area features ; S333, using the word element to enhance the block features and word-enhanced tile description features As input, the semantics of the block features of the traffic diagram are refined through the dot product operation to obtain the multimodal block features of the traffic diagram ; S334, visual area features enhanced by the word element features As a query, the multimodal patch features As keys and values, we obtain fine-grained patch-enhanced visual region features; S335, performing an average pooling operation on the visual area features enhanced by the fine-grained image blocks to obtain a global traffic diagram feature .

5. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: The S340 includes: S341, defining the number of layers L of the multimodal question fusion network and the number of multi-head attention heads H; S342, the global problem features and the global traffic schematic features Perform dot product operations to obtain the characteristics of the multimodal traffic diagram problem; S343, using the multimodal traffic diagram problem feature as a query, the word feature enhanced by the context interaction and the visual area feature enhanced by the fine-grained tile in S334 as context, to obtain a fused multimodal traffic diagram problem feature.

6. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: The S350 includes: The fused multimodal traffic diagram question features are input into the answer selector, so that the answer selector performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers. .

7. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: S360 includes: S361: Input the probability distribution of the candidate answers into the optimization loss function to calculate the loss value of the input traffic diagram question and answer, which is expressed as: Among them, N represents the number of question texts corresponding to the input traffic diagram, C represents the number of candidate answers, represents the true answer distribution of the input question text, represents the predicted answer distribution.

8. The small sample traffic diagram question answering method based on multimodal context according to claim 1 is characterized in that: S400 includes: S410, obtaining a test traffic diagram and a test question text, and answering the test traffic diagram and the test question text using the trained traffic diagram question-answering model to obtain answers corresponding to the test traffic diagram and the test question text; S420, evaluating the performance of the traffic diagram question answering model using the test traffic diagram and the answers corresponding to the test question text; S430, using the traffic diagram question-answering model whose performance meets the requirements as the selected traffic diagram question-answering model; S440: deploying the filtered traffic diagram question-answering model on an electronic device to answer the traffic diagram question to be answered and displaying it to the user.

9. A small sample traffic diagram question-answering device based on multimodal context, characterized in that: include: An acquisition module is configured to acquire a sample set including a plurality of traffic diagrams and a plurality of question texts; an extraction module configured to extract visual area features and visual block features of each traffic diagram in the sample set, and extract word-unit features of each question text in the sample set, and construct a multimodal feature dataset using the visual area features, visual block features, and word-unit features; wherein the visual block features include block features and block description features; a training module configured to train a pre-built traffic diagram question-answering model using the multimodal feature dataset to obtain a trained traffic diagram question-answering model; The question-answering module is configured to filter the trained traffic diagram question-answering model and, after the filtering is completed, is deployed on the electronic device to answer the traffic diagram question and present it to the user; The training module is configured as follows: S310, define the contextual interactive question representation network, fine-grained enhanced traffic diagram representation network, multimodal question fusion network, answer selector and optimization loss function in the traffic diagram question answering model; S320, in the current iteration, selecting a visual block feature of a traffic diagram and a word-meta feature of a corresponding question text from the multimodal feature dataset as input, and inputting the inputs into the context-interactive question representation network to obtain a word-meta feature enhanced by context-interaction, a global question feature, and a word-meta enhanced visual block feature; S330, inputting the context-interaction-enhanced word-unit features, the global question features, the word-unit-enhanced visual block features, and the visual area features of the traffic diagram in S320 into the fine-grained enhanced traffic diagram representation network to perform traffic diagram representation enhancement, thereby obtaining fine-grained enhanced visual area features and global traffic diagram features; S340: Inputting the fine-grained enhanced visual area features, the global traffic diagram features, the context-interaction-enhanced word features, and the global question features into the multimodal question fusion network to obtain a fused multimodal traffic diagram question feature. S350, inputting the fused multimodal traffic diagram question features into the answer selector to obtain a probability distribution of candidate answers; S360, calculating a loss value of the input traffic diagram question and answer using the optimized loss function according to the probability distribution described in S350; S370, using the loss value to adjust the weight parameters of the traffic diagram question answering model, and reselecting the visual area features, visual block features, and word features of the corresponding question text of a traffic diagram from the multimodal feature dataset as input; S380, repeat S320 to S380 until the iteration end condition is reached, and obtain a trained traffic diagram question-answering model.

Images