A scene text visual question answering method based on graph pooling inference network

Abstract

The application discloses a scene text visual question answering method based on a graph pooling reasoning network, and steps of the method comprise the following steps: 1. image and text coding and graph representation; 2. single-layer graph pooling learning; 3. fusion of double-path multi-layer graphs; 4. answer generation; and 5. model optimization. The application adopts an adaptive node deletion strategy, dynamically reduces the size of nodes with similar semantics, realizes evolution and updating of the graph, gradually aggregates the evolved object graph and the evolved scene text graph semantics into a graph vector to promote answer reasoning, and thus can realize effective and robust scene text-based visual question answering.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and involves technologies such as computer vision, affective computing, and natural language processing. Specifically, it is a scene-based text visual question answering method based on graph pooling inference networks. Background Technology

[0002] With the development of computer vision and natural language tasks, researchers are dedicated to proposing efficient and robust models to expand the capabilities of artificial intelligence. Visual question answering tasks have seen rapid development in the past few years. Recently, researchers have begun to focus on the ability of visual question answering models to understand scene text in images. Scene text expresses rich information about human activities, such as numerical symbols, advertising slogans, traffic signs, and price tags. Scene text-based visual question answering (TextVQA) has become an emerging and popular topic in the fields of vision and language. TextVQA models have a wide range of applications, such as assistive devices for the visually impaired, online education, online shopping, and driver assistance. This task aims to perceive and understand scene text to answer text-based visual questions. This requires models to have multiple functions, including object detection, text recognition, and multimodal reasoning. This invention focuses on improving the multimodal reasoning capabilities of TextVQA models.

[0003] To improve the multimodal reasoning ability of TextVQA models, existing research has attempted different methods, such as: attention-based methods that calculate the attention between different models to update features; representation-based methods that emphasize learning stronger feature representations; and graph network-based methods that perform fully connected or sparse relation reasoning on visual entities in images. This invention aims to improve upon current graph network methods. To utilize visual entities in images, most existing graph-based TextVQA models always use the fully connected or sparse relations of all visual entities for graph learning, which has the following problems: (1) accumulated redundant entities often cause attention bias; (2) sparse relation learning cannot effectively reduce the negative impact of entity redundancy, and strong or weak sparsity easily leads to insufficient interaction or insufficient redundancy removal. Therefore, there is still much room for improvement and development potential for scene-based text-based visual question answering tasks. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention proposes a scene-based text visual question answering method based on graph pooling inference networks. This method aims to adaptively reduce the size of redundant nodes, extract core semantics, and thus achieve effective and robust scene-based text-based visual question answering.

[0005] The present invention adopts the following technical solution to solve the technical problem:

[0006] The present invention provides a scene-based visual question answering method for text, which is characterized by following steps:

[0007] Step 1: Image and text encoding and graph representation;

[0008] Step 1.1: Extract visual object features from a given image using a visual feature encoder. And use an OCR system to extract OCR features from the image. ,in, This represents the feature of the m-th visual object. This represents the nth OCR feature; Indicates the number of features of a visual object. Indicates the number of OCR features. Indicates the channel dimension;

[0009] Extract question features from a given question sentence using a question encoder. ,in, The first sentence in the question Features of each word; Indicates the number of words in the question sentence;

[0010] Step 1.2: Utilize the features of the visual object and OCR features Constructing a dual-path graph representation ,in, Table of visual object diagrams, and , Indicate the features of each visual object in The set of visual object nodes corresponding to the text. yes Directed edges between visual object nodes in the image; Represents an OCR image, and , Indicates that each OCR feature is in The corresponding OCR node in the text, Represents the set of directed edges between OCR nodes;

[0011] Step 1.3: Correct the visual object nodes:

[0012] Calculate the problem characteristics using equation (1) and visual object node collection Cosine similarity matrix between Then, use equations (2) and (3) to calculate the set of visual object nodes. Characteristics of the problem Semantic differences between ,in, Represents the m-th visual object node and problem characteristics Semantic differences:

[0013] (1)

[0014] (2)

[0015] (3)

[0016] In equation (1), It is a cosine function; express function;

[0017] In equations (2) and (3), Represents the m-th visual object node Features of the i-th word Similarity between them; This represents the L2 regularization operation; There are two parameters to be learned; Indicates semantic differences Feature dimensions; The m-th visual feature represents the semantics of the aggregation problem;

[0018] Calculate using equation (4) Corresponding similarity difference And using equation (5) to Updated to the corrected visual object diagram ,in, Represents the m-th object node Attention weights This represents the corrected set of visual object nodes in the corrected visual object graph, and , Represents the corrected visual object diagram The nth corrected visual object node in the middle, It means yes Corrected edge features between visual object nodes:

[0019] (4)

[0020] (5)

[0021] In equation (5), express function, Indicates from Select the k visual object nodes with the largest attention weights and the edges connected to them;

[0022] Step 2, for and After performing single-layer graph pooling learning, visual object graphs are obtained. global vector and global vector of OCR image ;

[0023] Step 3: Fusion of dual-path multi-layer graphs:

[0024] Step 3.1: Define the total number of pooling learning iterations for a single-layer graph as L, and then construct an L-layer graph;

[0025] When l=1, the visual object graph will be... Input the l-th layer graph and perform single-layer graph pooling learning according to step 2 to obtain the visual object graph output by the l-th layer graph. and the global vector of the l-th layer graph ;

[0026] When l = 2, 3, ..., L; [the following will be] Input the l-th layer graph and perform single-layer graph pooling learning according to step 2 to obtain the visual object graph output by the l-th layer graph. and the global vector of the visual object graph Thus, the Lth layer graph outputs the global vector of the pooled-learned visual object graph. And form a global graph vector of L visual object graphs. ;

[0027] Step 3.2: Use equation (15) to obtain the global vector after fusing the semantics of the visual object. :

[0028] (15)

[0029] In equation (15), There are L parameters to be learned;

[0030] Step 3.3: Follow the process in steps 3.1-3.2 to process the OCR image. Multi-layer graph pooling learning is performed to obtain global vectors of L OCR images. and the global vector after integrating OCR semantics ,in, Represents the global vector of the l-th OCR image;

[0031] Step 3.4: Use equation (16) to obtain the joint bipath graph vector. :

[0032] (16)

[0033] In equation (16), Represents the global vector after incorporating the semantics of visual objects. The corresponding parameters to be learned, Represents the global vector after incorporating OCR semantics. The corresponding parameters to be learned;

[0034] Step 4: Build an answer decoder and use it to generate answers. :

[0035] Step 5: Model Optimization

[0036] Step 5.1: Construct the binary cross-entropy loss function using equations (21) and (22) respectively. And policy gradient loss based on ANLS,

[0037] (twenty one)

[0038] (twenty two)

[0039] In equations (21) and (22), This represents the true answer to a given question.

[0040] Step 5.2: Construct the overall training objective using equation (22). :

[0041] (twenty three)

[0042] In equation (23), It is a trade-off hyperparameter;

[0043] Step 5.3: Use stochastic gradient descent to... Perform optimization to solve the problem, so that A scene-based text visual question answering model based on graph pooling inference network is developed to minimize the optimal parameters and obtain the answer to the question. This model is used to infer the visual object features, OCR features, and question features in a given image.

[0044] The scene text visual question answering method based on graph pooling inference network described in this invention is also characterized in that step 2 includes:

[0045] Step 2.1: Calculate the global semantic vector of problem feature Q using equation (6). :

[0046] (6)

[0047] In equation (6), It is the feature of the i-th word in a single-layer graph. A corresponding parameter to be learned;

[0048] Step 2.2: In the corrected visual object diagram Learning through diagrams guided by questions:

[0049] The corrected visual object graph is calculated using equations (7) and (8). The relationship matrix between visual object nodes after upward correction Then, the updated visual object nodes are obtained using equations (9) and (10). This results in an updated object graph. ;

[0050] (7)

[0051] (8)

[0052] (9)

[0053] (10)

[0054] In equations (7) and (8), Represents the corrected visual object diagram The set of edges in the semantics of the fusion problem. Indicates the activation function; These represent directed edges in a single-layer graph. Global semantic vector and the edge set that integrates the semantics of the problem. The corresponding parameters to be learned; Represents the set of edges Attention weights;

[0055] In equations (9) and (10), Indicates the first A corrected visual object node and the One corrected visual object node Similarity weights between them; Represents the first edge after aggregating all edge features. One corrected visual object node feature; For nodes after aggregating all edge features The corresponding parameters to be learned Indicates the first visual object node The corresponding parameters to be learned, express The corresponding parameters to be learned; For the first A new visual object node;

[0056] Step 2.3: Calculate using equation (11) Attention weights for all new visual object nodes And use equation (12) to retain the top weights. Each node is used to obtain the visual object graph after single-layer graph pooling learning. ,in, express The collection of visual object nodes in the, and ,in, Represents the visual object graph after deleting nodes. The nth visual object node in the image. yes Directed edges between visual object nodes in the image:

[0057] (11)

[0058] (12)

[0059] In equation (11), It is a parameter to be learned;

[0060] In equation (12), This indicates the set percentage of nodes to be deleted;

[0061] Step 2.4: Use equation (13) to obtain the visual object graph after single-layer graph pooling learning. global vector :

[0062] (13)

[0063] Step 2.5: Update the OCR image following the process in Steps 2.1-2. Node characteristics;

[0064] Step 2.6: Obtain the OCR image after single-layer graph pooling learning following the process in Step 2.3. ;in, Indicates that each OCR feature is in The set of visual object nodes corresponding to the text. express Directed edges between visual object nodes in the image;

[0065] Step 2.7: Following the process in Step 2.4, obtain the global vector of the OCR image after single-layer graph pooling learning. .

[0066] Step 4 includes:

[0067] Step 4.1, the answer decoder includes: a Transformer block and an object classifier. and OCR tag classifier ;

[0068] Step 4.2: Define the total number of decoding steps for the decoder as follows. And obtain a length of The answer ;

[0069] Define the problem characteristics of decoding at step t-1 as follows: The visual object features decoded in step t-1 are: The OCR features decoded in step t-1 are: The global vector decoded in step t-1 is The hidden vector decoded in step t-1 is ;

[0070] When t=1, initialize , , , , ,in, Use random initial values;

[0071] Step 4.3: The Transformer block uses equation (17) to obtain the decoding result of step t. :

[0072] (17)

[0073] In equation (17), These are the five parameters to be learned in the Transformer block. This represents the problem characteristics being decoded at step t; This represents the visual object features decoded at step t; This represents the OCR feature decoded at step t; This represents the global vector decoded at step t. This represents the hidden vector decoded at step t;

[0074] Step 4.4, the classifier The probability score of the decoding step t is obtained using equation (18). :

[0075] (18)

[0076] In equation (18), express One parameter to be learned; This represents a deviation parameter;

[0077] Step 4.5, the object classifier Calculate using equation (19) and Correlation score between :

[0078] (19)

[0079] In equation (18), , Represents the OCR features decoded at step t. Two parameters to be learned; , This represents two deviation parameters;

[0080] Step 4.6: Calculate the predicted word decoded in step t using equation (20). :

[0081] (20)

[0082] In equations (18)-(20), Argmax represents the Argmax function;

[0083] Step 4.7: After assigning t+1 to t, if t> This indicates that the answer has been obtained. Otherwise, return to step 4.3 and execute sequentially.

[0084] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the scene-based text visual question-answering methods, and the processor is configured to execute the program stored in the memory.

[0085] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of any of the scene-based text visual question-answering methods.

[0086] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0087] 1. This invention introduces graph pooling learning into the TextVQA task for the first time, proposing a dynamically evolving graph learning model (GPIN), namely a graph pooling inference network. Unlike previous graph models that only focus on fully connected learning or relational learning, GPIN emphasizes node learning at its core. It employs an adaptive node deletion technique to enhance graph evolution, reducing interference from redundant visual entities during inference, thereby aiding in answer generation and significantly improving the accuracy of answer prediction.

[0088] 2. The GPIN proposed in this invention is a hierarchical graph structure that progressively extracts visual cues from objects and tags. A hierarchical object graph and OCR graph are designed to semantically cleanse the visual content. Through dual-path graph fusion, a global vector for answer reasoning is obtained. Feeding the obtained global vector into the answer generator helps with semantic alignment between different features, thereby enabling effective alignment of features between the question and the image. Attached Figure Description

[0089] Figure 1 This is a flowchart of the scene text visual question answering method based on graph pooling inference network of the present invention. Detailed Implementation

[0090] In this embodiment, as Figure 1 As shown, a scene-based text visual question answering method based on graph pooling inference networks differs from previous graph-based TextVQA models by constructing a dynamically evolving graph learning method, including: 1. Image and text encoding and graph representation; 2. Single-layer graph pooling learning; 3. Fusion of dual-path multi-layer graphs; 4. Answer generation; 5. Model optimization. Specifically, it proceeds in the following steps:

[0091] Step 1: Image and text encoding and graph representation;

[0092] Step 1.1: Use a pre-trained Faster R-CNN model as a visual feature encoder to extract visual object features from a given image. And use an OCR system to extract OCR features from the image. Each detected object is characterized by its appearance features. and bounding box features Each feature is encoded by two separate fully connected layers, and then summed to form the initial object features. For OCR detection, each OCR token continuously output by the OCR system consists of four features, namely FastText features. Pyramid histogram features Appearance features Bounding box features These features are encoded by four separate fully connected layers and then used as the initial OCR features. A uniform feature dimension is set to d = 768. This represents the feature of the m-th visual object. This represents the nth OCR feature; Indicates the number of features of a visual object. Indicates the number of OCR features. Indicates the channel dimension;

[0093] A pre-trained BERT model is used as a question encoder to extract question features from a given question sentence. ,in, The first sentence in the question Features of each word; Indicates the number of words in the question sentence;

[0094] Step 1.2: Utilizing the features of visual objects and OCR features Constructing a dual-path graph representation ,in, Table of visual object diagrams, and , Indicate the features of each visual object in The set of visual object nodes corresponding to the text. yes Directed edges between visual object nodes in the image; Represents an OCR image, and , Indicates that each OCR feature is in The corresponding OCR node in the text, This represents the set of directed edges between OCR nodes.

[0095] Let the nth OCR node To the OCR nodes edge features ,in These represent the nth OCR node. The top-left and bottom-right coordinates, These represent the nth OCR node. Width and height; They represent the first OCR nodes The center coordinates, width, and height of the bounding box.

[0096] Step 1.3: Correct the visual object nodes:

[0097] Calculate the problem characteristics using equation (1) and visual object node collection Cosine similarity matrix between Then, use equations (2) and (3) to calculate the set of visual object nodes. Characteristics of the problem Semantic differences between ,in, Represents the m-th visual object node and problem characteristics Semantic differences:

[0098] (1)

[0099] (2)

[0100] (3)

[0101] In equation (1), It is a cosine function; express function;

[0102] In equations (2) and (3), Represents the m-th visual object node Features of the i-th word Similarity between them; This represents the L2 regularization operation; There are two parameters to be learned; Indicates semantic differences Feature dimensions; The m-th visual feature represents the semantics of the aggregation problem; in this embodiment, the dimension of the semantic difference representation is set to p = 256.

[0103] Calculate using equation (4) Corresponding similarity difference And using equation (5) to Updated to the corrected visual object diagram ,in, Represents the m-th object node Attention weights This represents the corrected set of visual object nodes in the corrected visual object graph, and , Represents the corrected visual object diagram The nth corrected visual object node in the middle, It means yes Corrected edge features between visual object nodes:

[0104] (4)

[0105] (5)

[0106] In equation (5), express function, Indicates from The top k visual object nodes with the largest attention weights and their connected edges are selected. In this embodiment, the size of the visual object node set is reduced to N = 50.

[0107] Step 2, Single-layer graph pooling learning:

[0108] Step 2.1: Calculate the global semantic vector of problem feature Q using equation (6). :

[0109] (6)

[0110] In equation (6), It is the feature of the i-th word in a single-layer graph. The corresponding parameter to be learned.

[0111] Step 2.2: In the corrected visual object diagram Learning through diagrams guided by questions:

[0112] The corrected visual object graph is calculated using equations (7) and (8). The relationship matrix between visual object nodes after upward correction Then, the updated visual object nodes are obtained using equations (9) and (10). This results in an updated object graph. ;

[0113] (7)

[0114] (8)

[0115] (9)

[0116] (10)

[0117] In equations (7) and (8), Represents the corrected visual object diagram The set of edges in the semantics of the fusion problem. Indicates the activation function; These represent directed edges in a single-layer graph. Global semantic vector and the edge set that integrates the semantics of the problem. The corresponding parameters to be learned; Represents the set of edges Attention weights.

[0118] In equations (9) and (10), Indicates the first One corrected visual object node and the One corrected visual object node Similarity weights between them; The first edge feature obtained by aggregating all edge features One corrected visual object node feature; The nodes obtained by aggregating all edge features The corresponding parameters to be learned Indicates the first visual object node The corresponding parameters to be learned, express The corresponding parameters to be learned; For the first A new visual object node.

[0119] Step 2.3: Calculate using equation (11) Attention weights for all new visual object nodes And use equation (12) to retain the top weights. Each node is used to obtain the visual object graph after single-layer graph pooling learning. ,in, , yes Directed edges between visual object nodes in the image:

[0120] (11)

[0121] (12)

[0122] In equation (11), It is a parameter to be learned;

[0123] In equation (12), This indicates the set node deletion ratio. In this embodiment, the number of layers in the graph pool is set to L=3 and the node deletion ratio is set to... = 40%.

[0124] Step 2.4: Use equation (13) to obtain the visual object graph after single-layer graph pooling learning. global vector :

[0125] (13)

[0126] In equation (13), It is the visual object graph after deleting nodes. The nth visual object node in the array.

[0127] Step 2.5: Update the OCR image following the process in Steps 2.1-2. Node characteristics;

[0128] Step 2.6: Obtain the OCR image after single-layer graph pooling learning following the process in Step 2.3. ;in, Indicates that each OCR feature is in The set of visual object nodes corresponding to the text. express Directed edges between visual object nodes in the image;

[0129] Step 2.7: Following the process in Step 2.4, obtain the global vector of the OCR image after single-layer graph pooling learning. .

[0130] Step 3: Fusion of dual-path multi-layer graphs:

[0131] Step 3.1: Define the total number of pooling learning iterations for a single-layer graph as L, and then construct an L-layer graph;

[0132] When l=1, the visual object graph will be... The input is processed in the l-th layer, and single-layer graph pooling learning is performed according to step 2 to obtain the visual object graph output by the l-th layer. and the global vector of the l-th layer graph ;

[0133] When l = 2, 3, ..., L; [the following will be] The input is processed in the l-th layer, and single-layer graph pooling learning is performed according to step 2 to obtain the updated visual object graph. and the global vector of the visual object graph Thus, the global vector of the visual object graph is obtained through single-layer graph pooling learning in the Lth layer. And form a global graph vector of L visual object graphs. .

[0134] Step 3.2: Use equation (15) to obtain the global vector after fusing the semantics of the visual object. :

[0135] (15)

[0136] In equation (15), There are L parameters to be learned;

[0137] Step 3.3: Follow the process in steps 3.1-3.2 to process the OCR image. Multi-layer graph pooling learning is performed to obtain global vectors of L OCR images. and the global vector after integrating OCR semantics ,in, This represents the global vector of the l-th OCR image.

[0138] Step 3.4: Use equation (16) to obtain the joint bipath graph vector. :

[0139] (16)

[0140] In equation (16), Represents the global vector after incorporating the semantics of visual objects. The corresponding parameters to be learned, Represents the global vector after incorporating OCR semantics. The corresponding parameters to be learned.

[0141] Step 4: Answer Generation

[0142] Step 4.1: Construct the answer decoder, including: Transformer block and object classifier. and OCR tag classifier ;

[0143] Step 4.2, Defining the decoder, is executed in total. Decoding in steps yields an answer of length . ;

[0144] The input at step t-1 is the problem features. Visual object features OCR features Global vector of the graph latent vectors When t=1, initialize. , , , , ,in, It is obtained from random initialization.

[0145] Step 4.3: The Transformer block uses equation (17) to obtain the decoding result of step t. :

[0146] (17)

[0147] In equation (17), These are the five parameters to be learned in the Transformer block. This represents the problem features of the input at step t-1. This represents the problem features output after decoding at step t-1; This represents the visual object features input at step t-1. This represents the visual object features output after decoding at step t-1; This represents the OCR features of the input at step t-1. This represents the OCR feature output after decoding at step t-1; This represents the global vector of the input graph at step t-1. This represents the global vector of the graph output after decoding at step t-1; Let represent the latent vector of the input at step t-1. This represents the hidden vector output after decoding at step t-1.

[0148] Step 4.4, Classifier The probability score after decoding at step t is obtained using equation (18). :

[0149] (18)

[0150] In equation (18), , express One parameter to be learned; This represents a deviation parameter;

[0151] Step 4.5, Object Classifier Calculate using equation (19) and Correlation score between :

[0152] (19)

[0153] In equation (18), , Represents the OCR features after decoding at step t. Two parameters to be learned; , This represents two deviation parameters;

[0154] Step 4.6: Calculate the predicted word after decoding in step t using equation (20). :

[0155] (20)

[0156] In equations (18)-(20), Argmax represents the Argmax function;

[0157] Step 4.7: After assigning t+1 to t, if t> This indicates that the answer has been obtained. ;in, It is the length of the answer; in this embodiment, the number of steps for answer decoding is set to =12; otherwise, return to step 4.3 and execute sequentially.

[0158] Step 5: Model Optimization

[0159] Step 5.1: Construct the binary cross-entropy loss function using equations (21) and (22) respectively. And policy gradient loss based on ANLS,

[0160] (twenty one)

[0161] (twenty two)

[0162] In equations (21) and (22), This indicates the true answer;

[0163] Step 5.2: Construct the overall training objective using equation (22). :

[0164] (twenty three)

[0165] In equation (23), It is a trade-off hyperparameter.

[0166] Step 5.3: Use stochastic gradient descent to... Perform optimization to solve the problem, so that A scene-based text visual question answering model based on graph pooling inference network is developed to minimize the optimal parameters and obtain the answer to the question. This model is used to infer the visual object features, OCR features, and question features in a given image.

[0167] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0168] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0169] This invention was experimentally validated on two publicly available datasets, TextVQA and ST-VQA. The performance achieved by this invention surpasses that of existing graph network models, demonstrating its effectiveness. Significant performance improvements were achieved on the TextVQA dataset, and substantial performance improvements were observed across multiple metrics on the ST-VQA dataset.

Claims

1. A scene-based text visual question answering method based on graph pooling inference networks, characterized in that, The procedure is as follows: Step 1: Image and text encoding and graph representation; Step 1.1: Extract visual object features from a given image using a visual feature encoder. And use an OCR system to extract OCR features from the image. ,in, This represents the feature of the m-th visual object. This represents the nth OCR feature; Indicates the number of features of a visual object. Indicates the number of OCR features. Indicates the channel dimension; Extract question features from a given question sentence using a question encoder. ,in, The first sentence in the question Features of each word; Indicates the number of words in the question sentence; Step 1.2: Utilize the features of the visual object and OCR features Constructing a dual-path graph representation ,in, Table of visual object diagrams, and , Indicate the features of each visual object in The set of visual object nodes corresponding to the text. yes Directed edges between visual object nodes in the image; Represents an OCR image, and , Indicates that each OCR feature is in The corresponding OCR node in the text, Represents the set of directed edges between OCR nodes; Step 1.3: Correct the visual object nodes: Calculate the problem characteristics using equation (1) and visual object node collection Cosine similarity matrix between Then, use equations (2) and (3) to calculate the set of visual object nodes. Characteristics of the problem Semantic differences between ,in, Represents the m-th visual object node and problem characteristics Semantic differences: (1) (2) (3) In equation (1), It is a cosine function; express function; In equations (2) and (3), Represents the m-th visual object node Features of the i-th word Similarity between them; This represents the L2 regularization operation; There are two parameters to be learned; Indicates semantic differences Feature dimensions; The m-th visual feature represents the semantics of the aggregation problem; Calculate using equation (4) Corresponding similarity difference And using equation (5) to Updated to the corrected visual object diagram ,in, Represents the m-th object node Attention weights This represents the corrected set of visual object nodes in the corrected visual object graph, and , Represents the corrected visual object diagram The nth corrected visual object node in the middle, It means yes Corrected edge features between visual object nodes: (4) (5) In equation (5), express function, Indicates from Select the k visual object nodes with the largest attention weights and the edges connected to them; Step 2, for and After performing single-layer graph pooling learning, visual object graphs are obtained. global vector and global vector of OCR image ; Step 3: Fusion of dual-path multi-layer graphs: Step 3.1: Define the total number of pooling learning iterations for a single-layer graph as L, and then construct an L-layer graph; When l=1, the visual object graph will be... Input the l-th layer graph and perform single-layer graph pooling learning according to step 2 to obtain the visual object graph output by the l-th layer graph. and the global vector of the l-th layer graph ; When l = 2, 3, ..., L; [the following will be] Input the l-th layer graph and perform single-layer graph pooling learning according to step 2 to obtain the visual object graph output by the l-th layer graph. and the global vector of the visual object graph Thus, the Lth layer graph outputs the global vector of the pooled-learned visual object graph. And form a global graph vector of L visual object graphs. ; Step 3.2: Use equation (15) to obtain the global vector after fusing the semantics of the visual object. : (15) In equation (15), There are L parameters to be learned; Step 3.3: Follow the process in steps 3.1-3.2 to process the OCR image. Multi-layer graph pooling learning is performed to obtain global vectors of L OCR images. and the global vector after integrating OCR semantics ,in, Represents the global vector of the l-th OCR image; Step 3.4: Use equation (16) to obtain the joint bipath graph vector. : (16) In equation (16), Represents the global vector after incorporating the semantics of visual objects. The corresponding parameters to be learned, Represents the global vector after incorporating OCR semantics. The corresponding parameters to be learned; Step 4: Build an answer decoder and use it to generate answers. : Step 5: Model Optimization Step 5.1: Construct the binary cross-entropy loss function using equations (21) and (22) respectively. and policy gradient loss based on ANLS; (21) (22) In equations (21) and (22), This represents the true answer to a given question. Step 5.2: Construct the overall training objective using equation (22). : (23) In equation (23), It is a trade-off hyperparameter; Step 5.3: Use stochastic gradient descent to... Perform optimization to solve the problem, so that A scene-based text visual question answering model based on graph pooling inference network is developed to minimize the optimal parameters and obtain the answer to the question. This model is used to infer the visual object features, OCR features, and question features in a given image.

2. The scene text visual question answering method based on graph pooling inference network according to claim 1, characterized in that, Step 2 includes: Step 2.1: Calculate the global semantic vector of problem feature Q using equation (6). : (6) In equation (6), It is the feature of the i-th word in a single-layer graph. A corresponding parameter to be learned; Step 2.2: In the corrected visual object diagram Learning through diagrams guided by questions: The corrected visual object graph is calculated using equations (7) and (8). The relationship matrix between visual object nodes after upward correction Then, the updated visual object nodes are obtained using equations (9) and (10). This results in an updated object graph. ; (7) (8) (9) (10) In equations (7) and (8), Represents the corrected visual object diagram The set of edges in the semantics of the fusion problem. Indicates the activation function; These represent directed edges in a single-layer graph. Global semantic vector and the edge set that integrates the semantics of the problem. The corresponding parameters to be learned; Represents the set of edges Attention weights; In equations (9) and (10), Indicates the first One corrected visual object node and the One corrected visual object node Similarity weights between them; Represents the first edge after aggregating all edge features. One corrected visual object node feature; For nodes after aggregating all edge features The corresponding parameters to be learned Indicates the first visual object node The corresponding parameters to be learned, express The corresponding parameters to be learned; For the first A new visual object node; Step 2.3: Calculate using equation (11) Attention weights for all new visual object nodes And use equation (12) to retain the top weights. Each node is used to obtain the visual object graph after single-layer graph pooling learning. ,in, express The collection of visual object nodes in the, and ,in, Represents the visual object graph after deleting nodes. The nth visual object node in the image. yes Directed edges between visual object nodes in the image: (11) (12) In equation (11), It is a parameter to be learned; In equation (12), This indicates the set percentage of nodes to be deleted; Step 2.4: Use equation (13) to obtain the visual object graph after single-layer graph pooling learning. global vector : (13) Step 2.5: Update the OCR image following the process in Steps 2.1-2. Node characteristics; Step 2.6: Obtain the OCR image after single-layer graph pooling learning following the process in Step 2.

3. ;in, Indicates that each OCR feature is in The set of visual object nodes corresponding to the text. express Directed edges between visual object nodes in the image; Step 2.7: Following the process in Step 2.4, obtain the global vector of the OCR image after single-layer graph pooling learning. .

3. The scene text visual question answering method based on graph pooling inference network according to claim 2, characterized in that, Step 4 includes: Step 4.1, the answer decoder includes: a Transformer block and an object classifier. and OCR tag classifier ; Step 4.2: Define the total number of decoding steps for the decoder as follows. And obtain a length of The answer ; Define the problem characteristics of decoding at step t-1 as follows: The visual object features decoded in step t-1 are: The OCR features decoded in step t-1 are: The global vector decoded in step t-1 is The hidden vector decoded in step t-1 is ; When t=1, initialize , , , , ,in, Use random initial values; Step 4.3: The Transformer block uses equation (17) to obtain the decoding result of step t. : (17) In equation (17), These are the five parameters to be learned in the Transformer block. This represents the problem characteristics being decoded at step t; This represents the visual object features decoded at step t; This represents the OCR feature decoded at step t; This represents the global vector decoded at step t. This represents the hidden vector decoded at step t; Step 4.4, the classifier The probability score of the decoding step t is obtained using equation (18). : (18) In equation (18), express One parameter to be learned; This represents a deviation parameter; Step 4.5, the object classifier Calculate using equation (19) and Correlation score between : (19) In equation (18), , Represents the OCR features decoded at step t. Two parameters to be learned; , This represents two deviation parameters; Step 4.6: Calculate the predicted word decoded in step t using equation (20). : (20) In equations (18)-(20), Argmax represents the Argmax function; Step 4.7: After assigning t+1 to t, if t> This indicates that the answer has been obtained. Otherwise, return to step 4.3 and execute sequentially.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the scene text visual question answering method of any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of any of the scene text visual question answering methods described in claims 1-3.

Images