Scene text context alignment method based on multi-view separation and localization

Abstract

This paper proposes a scene text context alignment method based on multi-view separation and positioning. The method includes: using an OCR system to identify and extract the original image to obtain the final OCR features; obtaining the final fused representation vector of the object based on the original image; obtaining the input of the T5 encoder based on the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding of the object; obtaining a discretized distance based on the bounding box features of the text; obtaining an attention output result based on the discretized distance; calculating the KL divergence difference values of the three channels based on the original image, and obtaining the final visual association representation based on the KL divergence difference values; training a model using the processed text features and visual features; and obtaining a natural language answer using the trained model. The present invention analyzes the color distribution differences between texts in different scenes and evaluates their visual association, helping the model to more accurately judge the relationship between texts.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language processing, and in particular to a scene text context alignment method and system based on multi-view separation and positioning. Background Art

[0002] Text-based visual question answering (TextVQA) requires the model to be able to read, understand, and reason about textual information in natural scenes in order to answer questions related to the scene text. This task requires not only that the model be able to distinguish the visual, semantic, and spatial information of the scene text, but also that the model understand the interactions between these modalities in order to infer the correct answer. Previous research has mainly improved model performance by improving network structure, designing dedicated modules, and introducing pre-training tasks. These methods have greatly promoted the development of this field. However, most of these methods rely on traditional Transformer-based models to effectively integrate multimodal information. The Transformer architecture was originally designed for natural language processing tasks and is suitable for processing semantically coherent and grammatically complete natural language text.

[0003] However, in TextVQA, scene text in images must first be detected and recognized using an optical character recognition (OCR) system, and this text is distributed across different locations in the image. This distribution differs from natural language sentences, which inherently possess semantic coherence and connectivity, whereas scene text typically lacks this inherent semantic continuity. However, previous TextVQA methods have mostly ignored this, directly concatenating unrelated scene text into a single sentence. Therefore, understanding the semantic context between scene text on different media requires a dedicated mechanism.

[0004] Another significant difference is that natural language text has a fixed reading order, usually from left to right and from top to bottom; while scene text often does not follow this natural order. In natural language, words and sentences are naturally semantically connected, so they can be spliced into coherent sentences according to the reading order, and their sequence relationships can be represented using absolute or relative position encoding. However, text in natural scenes does not have a clear reading order and usually presents complex spatial relationships. If scene texts are only spliced in a left-to-right and top-to-bottom order, and their relationships are modeled using one-dimensional position encoding, modeling problems will arise. This approach will cause spatially adjacent texts to be encoded as being far apart and cannot accurately represent the true spatial relationship between texts. Therefore, developing more effective methods to model the spatial relationships of scene texts is a very necessary and worthy direction to explore. Summary of the Invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a scene text context alignment method based on multi-view separation and positioning to solve the above technical problems.

[0006] The present invention proposes a scene text context alignment method based on multi-view separation and positioning, the method comprising the following steps:

[0007] Step 1: Use the OCR system to identify and extract the original image to obtain the scene text;

[0008] Based on the scene text, the final OCR features are obtained;

[0009] Based on the original image, the final fused representation vector of the object is extracted through Faster R-CNN;

[0010] Step 2: The T5 Token embedding layer encodes the words in the question into feature embeddings to obtain question features;

[0011] Based on the final OCR features, the final fused representation vector of the object, and the question features, the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object are obtained respectively;

[0012] Use the concatenation function on the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object to obtain the input of the T5 encoder;

[0013] Step 3: Divide the original image into image blocks. Based on the bounding box features of the OCR text, map the bounding box coordinates of the OCR text to the coordinates of the image blocks, and further obtain the discretized distance.

[0014] Using the discretized distance, we can calculate the attention and get the attention output result.

[0015] Separate the original image into three RGB channels and calculate the KL divergence difference value of the three channels;

[0016] The KL divergence difference values of the three channels are used to obtain the red channel gating weight and the blue channel gating weight respectively;

[0017] The final visual association representation is obtained by multi-channel visual association fusion processing using the red channel gating weight and the blue channel gating weight.

[0018] Step 4: Using the final visual context representation, a pre-trained encoder is used to generate a joint multimodal context representation.

[0019] Based on the joint multimodal contextual representation, a pre-trained decoder is used for autoregressive generation to obtain natural language answers.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. Unlike methods that directly concatenate scene texts without clear semantic relationships into sentences, this method distinguishes texts that lack contextual semantic connections, thereby reducing noise in training. This effectively enhances the model's ability to learn text semantic associations and helps infer answers related to the text.

[0022] 2. This paper analyzes the color distribution differences between texts in different scenes and evaluates their visual relevance, helping the model to more accurately judge the relationship between texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flowchart of the scene text context alignment method based on multi-view separation and positioning proposed by the present invention. DETAILED DESCRIPTION

[0024] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0025] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0026] See also Figure 1 The embodiment of the present invention proposes a scene text context alignment method based on multi-view separation and positioning, which includes the following steps:

[0027] Step 1: Use the OCR system to identify and extract the original image to obtain the scene text;

[0028] Based on the scene text, the final OCR features are obtained;

[0029] Based on the original image, the final fused representation vector of the object is extracted through Faster R-CNN;

[0030] In step 1, the final OCR features are obtained based on the scene text. The specific steps are as follows:

[0031] Based on scene text, visual features are extracted by Faster R-CNN. The relationship between the corresponding process is:

[0032] ;

[0033] in, Indicates the OCR index, Bounding box representation of visual features, Indicates the horizontal coordinate of the upper left corner of the text box. Indicates the vertical coordinate of the upper left corner of the text box. Indicates the horizontal coordinate of the lower right corner of the text box. Indicates the vertical coordinate of the lower right corner of the text box; Indicates the width of the image, used for horizontal coordinate normalization; Indicates the height of the image, used for vertical coordinate normalization; Represents the numerical vector processing of the normalized coordinates of the OCR text box;

[0034] The visual features are processed using the T5 Token embedding layer to obtain the final OCR features. The corresponding relationship is:

[0035] ;

[0036] in, represents the final OCR feature, Indicates the layer normalization process through the T5 Token embedding layer. and Both represent learnable weight matrices, Represents the visual features of OCR, Represents the text bounding box features for OCR, represents the text embedding for OCR, Represents the learnable weight matrix of the T5 Token embedding layer;

[0037] Based on the original image, the final fusion representation vector of the object is extracted by Faster R-CNN. The relationship between the corresponding process is:

[0038] ;

[0039] in, Indicates the appearance features of the object corresponding to OCR, represents the bounding box feature corresponding to OCR, Indicates the T5 word embedding corresponding to the OCR corresponding object label, represents a learnable weight matrix that maps visual appearance features to uniform dimensions, represents a learnable weight matrix that maps bounding box features to uniform dimensions, represents a learnable weight matrix that maps text embeddings to uniform dimension.

[0040] Furthermore, in this step, based on the input original image, a pre-trained object detector is first used to perform object detection on the image, producing several "object" regions. At the same time, feature pooling is performed on each region to obtain a series of visual feature vectors. Then, OCR detection is performed on the text area in the image, and text fragments (OCR regions) are extracted. These are also sent to the pre-trained object detector to obtain the visual features of the text area.

[0041] Step 2: The T5 Token embedding layer encodes the words in the question into feature embeddings to obtain question features;

[0042] Based on the final OCR features, the final fused representation vector of the object, and the question features, the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object are calculated respectively;

[0043] Use the concatenation function to concatenate the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object to obtain the input of the T5 encoder;

[0044] In step 2, the T5 Token embedding layer is used to encode the words in the question into feature embeddings to obtain question features. The corresponding relationship is:

[0045] ;

[0046] in, represents the index of the question word, Indicates the characteristics of question words, Indicates the length of the question, Indicates the Embedding of question words;

[0047] The embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object are concatenated using the concatenation function to obtain the input of the T5 encoder. The corresponding relationship is:

[0048] ;

[0049] in, represents the input to the T5 encoder, which is the result of concatenating the object, the text extracted by OCR, and the question information; Indicates processing through the splicing function, The embedding matrix representing the problem, represents the embedding matrix of the OCR text, Embedding matrix representing the object.

[0050] Furthermore, in this step, the input of the T5 encoder It is used to participate in the subsequent encoder and decoder training and splicing operations, so that the T5 encoder can uniformly model the relationship between these input information.

[0051] Furthermore, in this step, the visual features of the "objects" and "OCR regions" output by the pre-trained object detector are projected separately to obtain a visual feature representation of unified dimension. The three types of text (question text, OCR detected text, and object label) are spliced into a long sequence and sent to the pre-trained text embedding layer, mapping each token to the corresponding vector representation.

[0052] Step 3: Divide the original image into image blocks. Based on the bounding box features of the OCR text, map the bounding box coordinates of the OCR text to the coordinates of the image blocks, and further obtain the discretized distance.

[0053] Using the discretized distance, we can calculate the attention and get the attention output result.

[0054] Separate the original image into three RGB channels and calculate the KL divergence difference value of the three channels;

[0055] The KL divergence difference values of the three channels are used to obtain the red channel gating weight and the blue channel gating weight respectively;

[0056] The final visual association representation is obtained by multi-channel visual association fusion processing using the red channel gating weight and the blue channel gating weight.

[0057] In step 3, the original image is divided into 11×11 image blocks. Based on the bounding box features of the OCR text, the bounding box coordinates of the OCR text are mapped to the coordinates of the image blocks, and the discretized distance is further obtained. The specific steps are as follows:

[0058] The original image is divided into image regions in the form of blocks. Based on the bounding box features of the OCR text, the bounding box coordinates of the OCR text are mapped to the coordinates of the image block to obtain the index of the text division into the image region. The corresponding relationship is:

[0059] ;

[0060] in, Indicates the The image region index to which the OCR text position is divided; Indicates that the spatial position of each OCR text is divided into a certain patch, that is, the image is divided into multiple image blocks of fixed size, and the spatial position of each OCR text is divided into a certain patch;

[0061] Based on the index of the text divided into image areas, the discretized distance is calculated and obtained by the Pythagorean theorem. The relationship between the corresponding process is:

[0062] ;

[0063] in, Indicates the OCR index, Represents the discretized distance between the image blocks where the two OCR texts are located, and Both represent text block coordinates. Indicates that the function processing is performed by calculating the Euclidean distance between two points. Represents an operation that converts the result to an integer;

[0064] Using the discretized distance, we can calculate the attention output through attention calculation. The specific steps are as follows:

[0065] The discretized distance is mapped to obtain the embedding vector of the spatial position. The relationship between the corresponding process is:

[0066] ;

[0067] in, Embedding vector representing the spatial position, Represents the operation of mapping discrete distance indices into dense vector representations;

[0068] Introducing the spatial bias term and calculating the attention score, the corresponding relationship is:

[0069] ;

[0070] in, represents the attention score, represents the learnable query matrix, represents the learnable bond matrix, Represents the input features of the OCR text, represents the learnable bias term; Represents the scaling factor to avoid the gradient disappearing due to excessive dot product value;

[0071] The main score and the embedding vector of the spatial position are integrated to obtain the attention weight. The relationship between the corresponding process is:

[0072] ;

[0073] in, represents the attention weight, represents spatial distance embedding;

[0074] According to the attention weight aggregation value vector, the attention output result is generated. The relationship between the corresponding process is:

[0075] ;

[0076] in, represents a matrix of learnable values; Represents the attention output result, that is, The OCR corresponding element is The attention output results generated by the corresponding OCR elements;

[0077] Separate the original image into three RGB channels and calculate the KL divergence difference value of the three channels. The corresponding relationship is:

[0078] ;

[0079] in, Represents the KL divergence difference value of the red channel, Indicates the calculation of KL divergence difference value, Indicates the The pixel value distribution of the red channel of the scene text area, Shidi The pixel value distribution of the red channel of the scene text area, represents the KL divergence difference value of the green channel, Indicates the The pixel value distribution of the green channel of the scene text area, Indicates the The pixel value distribution of the green channel of the scene text area, represents the KL divergence difference value of the blue channel, Indicates the The pixel value distribution of the blue channel of the scene text area, Indicates the The pixel value distribution of the blue channel of the scene text area;

[0080] The KL divergence difference values of the three channels are used to obtain the red channel gating weight and the blue channel gating weight respectively. The relationship between the corresponding processes is:

[0081] ;

[0082] in, represents the red channel gating weight, represents the blue channel gating weight;

[0083] Using the red channel gating weight and the blue channel gating weight, the final visual association representation is obtained through multi-channel visual association fusion processing. The relationship between the corresponding process is:

[0084] ;

[0085] in, Represents the final visual association representation.

[0086] Furthermore, in this step, the results are output by attention The auxiliary model understands the spatial relationship between scene texts.

[0087] Furthermore, in this step, It is a method of PyTorch tensor, which is used to convert the data type of the tensor to long integer. It is a parameterless method.

[0088] Furthermore, in this step, the query matrix that can be learned , learnable bond matrix and the learnable value matrix It is randomly generated when the model is initialized and continuously updated during the training process.

[0089] Furthermore, in this step, contextual tags are used to determine whether there is a semantic relationship between scene texts to prevent information from interfering with each other. The spacing between text blocks is then calculated, and spatial position information is encoded for all scene texts to help the model understand their relative layout in the image. Finally, the degree of association between scene texts is determined by comparing the differences in visual information distribution between them.

[0090] By inserting a ` <context>`Contextual tags.` <context>` is a learnable embedding representation from the T5 lookup table. Subsequently, we add the visual features and bounding box coordinates of each scene text to the corresponding ` <context>By designing a context-aware mask, each ` <context>` Tags can only be used with other ` <context>`Marks the attention calculation. Finally, each` <context>` Tags can determine whether there is semantic correlation between scene texts based on visual and positional differences through the attention mechanism.

[0091] Step 4: Using the final visual context representation, a pre-trained encoder is used to generate a joint multimodal context representation.

[0092] Based on the joint multimodal contextual representation, a pre-trained decoder is used for autoregressive generation to obtain natural language answers.

[0093] Furthermore, in this step, obtaining the joint multimodal context representation requires inputting the processed text features and the processed text features into the pre-trained encoder, wherein the processed text features include: embedding the text of the OCR in step 1 The text features obtained after context processing and the T5 word embedding corresponding to the object label in step 1 And the embedding of the question word obtained in step 2 ;

[0094] Among them, the processed visual features include: the visual features of OCR in step 1 , object appearance features corresponding to OCR and the final visual association representation in step 3 .

[0095] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0096] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0097] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.< / context> < / context> < / context> < / context> < / context> < / context>

Claims

1. A scene text context alignment method based on multi-view separation and positioning, characterized by: The method comprises the following steps: Step 1: Use the OCR system to identify and extract the original image to obtain the scene text; Based on the scene text, the final OCR features are obtained; Based on the original image, the final fused representation vector of the object is extracted through Faster R-CNN; Step 2: Use the T5 Token embedding layer to encode the words in the question into feature embeddings to obtain question features; Based on the final OCR features, the final fused representation vector of the object, and the question features, the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object are calculated respectively; Use the concatenation function to concatenate the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object to obtain the input of the T5 encoder; Step 3: Divide the original image into image blocks. Based on the bounding box features of the OCR text, map the bounding box coordinates of the OCR text to the coordinates of the image blocks, and further obtain the discretized distance. Using the discretized distance, we can calculate the attention and get the attention output result. Separate the original image into three RGB channels and calculate the KL divergence difference value of the three channels; The KL divergence difference values of the three channels are used to obtain the red channel gating weight and the blue channel gating weight respectively; The final visual association representation is obtained by multi-channel visual association fusion processing using the red channel gating weight and the blue channel gating weight. Step 4: Using the final visual context representation, a pre-trained encoder is used to generate a joint multimodal context representation. Based on the joint multimodal contextual representation, a pre-trained decoder is used for autoregressive generation to obtain natural language answers.

2. The scene text context alignment method based on multi-view separation and positioning according to claim 1 is characterized in that: In step 1, the final OCR features are obtained based on the scene text. The specific steps are as follows: Based on scene text, visual features are extracted by Faster R-CNN. The relationship between the corresponding process is: ； in, Indicates the OCR index, Bounding box representation of visual features, Indicates the horizontal coordinate of the upper left corner of the text box. Indicates the upper left corner vertical coordinate of the text box. Indicates the horizontal coordinate of the lower right corner of the text box. Indicates the vertical coordinate of the lower right corner of the text box. Indicates the width of the image, Indicates the height of the image, Represents the numerical vector processing of the normalized coordinates of the OCR text box; The visual features are processed using the T5 Token embedding layer to obtain the final OCR features. The corresponding relationship is: ； in, represents the final OCR feature, Indicates the layer normalization process through the T5 Token embedding layer. and Both represent learnable weight matrices, Represents the visual features of OCR, Represents the text bounding box features for OCR, represents the text embedding for OCR, Represents the learnable weight matrix of the T5 Token embedding layer.

3. The scene text context alignment method based on multi-view separation and positioning according to claim 2 is characterized in that: In step 1, based on the original image, the final fusion representation vector of the object is extracted by Faster R-CNN. The relationship between the corresponding process is: ； in, Indicates the appearance features of the object corresponding to OCR, represents the bounding box feature corresponding to OCR, Indicates the T5 word embedding corresponding to the OCR corresponding object label, Indicates the appearance features of the object corresponding to OCR, represents a learnable weight matrix that maps visual appearance features to uniform dimensions, represents a learnable weight matrix that maps bounding box features to uniform dimensions, represents a learnable weight matrix that maps text embeddings to uniform dimension.

4. The scene text context alignment method based on multi-view separation and positioning according to claim 3 is characterized in that: In step 2, the T5 Token embedding layer is used to encode the words in the question into feature embeddings to obtain question features. The corresponding relationship is: ； in, represents the index of the question word, Indicates the characteristics of the problem, Indicates the length of the question, Indicates the Embedding of question words.

5. The scene text context alignment method based on multi-view separation and positioning according to claim 4 is characterized in that: In step 2, the embedding matrix of the question, the embedding matrix of the OCR text, and the embedding matrix of the object are processed using a splicing function to obtain the input of the T5 encoder. The relationship between the corresponding process is: ； in, Represents the input of the T5 encoder, Indicates processing through the splicing function, The embedding matrix representing the problem, represents the embedding matrix of the OCR text, Embedding matrix representing the object.

6. The scene text context alignment method based on multi-view separation and positioning according to claim 5 is characterized in that: In step 3, the original image is divided into image blocks. Based on the bounding box features of the OCR text, the bounding box coordinates of the OCR text are mapped to the coordinates of the image blocks, and the discretized distance is further obtained. The specific steps are as follows: The original image is divided into image regions in the form of blocks. Based on the bounding box features of the OCR text, the bounding box coordinates of the OCR text are mapped to the coordinates of the image block to obtain the index of the text division into the image region. The corresponding relationship is: ； in, Indicates the The image region index into which the OCR text position is divided, Indicates that each OCR text space position is divided into a certain patch; Based on the index of the text divided into image areas, the discretized distance is calculated and obtained by the Pythagorean theorem. The relationship between the corresponding process is: ； in, Indicates the OCR index, Represents the discretized distance between the image blocks where the two OCR texts are located, and Both represent text block coordinates. Indicates that the function processing is performed by calculating the Euclidean distance between two points. Represents an operation that converts the result to an integer.

7. The scene text context alignment method based on multi-view separation and positioning according to claim 6 is characterized in that: In step 3, the attention output result is obtained by using the discretized distance and attention calculation. The specific steps are as follows: The discretized distance is mapped to obtain the embedding vector of the spatial position. The relationship between the corresponding process is: ； in, Embedding vector representing the spatial position, Represents the operation of mapping discrete distance indices into dense vector representations; Introducing the spatial bias term and calculating the attention score, the corresponding relationship is: ； in, represents the attention score, represents the learnable query matrix, represents the learnable bond matrix, Represents the input features of the OCR text, represents the learnable bias term, represents the scaling factor; The main score and the embedding vector of the spatial position are integrated to obtain the attention weight. The relationship between the corresponding process is: ； in, represents the attention weight, represents spatial distance embedding; According to the attention weight aggregation value vector, the attention output result is generated. The relationship between the corresponding process is: ； in, represents the learnable value matrix, Represents the attention output result.

8. The scene text context alignment method based on multi-view separation and positioning according to claim 7 is characterized in that: In step 3, the original image is separated into three channels, RGB, and the KL divergence difference values of the three channels are calculated. The corresponding relationship is: ； in, Represents the KL divergence difference value of the red channel, Indicates the calculation of KL divergence difference value, Indicates the The pixel value distribution of the red channel of the scene text area, Shidi The pixel value distribution of the red channel of the scene text area, represents the KL divergence difference value of the green channel, Indicates the The pixel value distribution of the green channel of the scene text area, Indicates the The pixel value distribution of the green channel of the scene text area, represents the KL divergence difference value of the blue channel, Indicates the The pixel value distribution of the blue channel of the scene text area, Indicates the Pixel value distribution of the blue channel of the scene text area.

9. The scene text context alignment method based on multi-view separation and positioning according to claim 8, characterized in that: In step 3, the KL divergence difference values of the three channels are used to obtain the red channel gating weight and the blue channel gating weight, respectively. The corresponding relationship is: ； in, represents the red channel gating weight, Represents the blue channel gating weight.

10. The scene text context alignment method based on multi-view separation and positioning according to claim 9, characterized in that: In step 3, the red channel gating weight and the blue channel gating weight are used to perform multi-channel visual association fusion processing to obtain the final visual association representation. The relationship between the corresponding process is: ； in, Represents the final visual association representation.

Images