OCR (Optical Character Recognition) image description generation method and system based on heterogeneous representation

Abstract

The invention discloses an OCR image description generation method and system based on heterogeneous representation, and relates to the technical field of OCR image description, and the method comprises the steps: extracting features based on a detected OCR region from the visual and text aspects, carrying out the text detection through Faster-RCNN, and carrying out the text recognition; the OCR visual embedding comprises a bounding box feature and a Faster-RCNN visual feature, and the OCR text embedding comprises a PHOC feature and a FastText feature. According to the method, the internal consistency of the OCR features can be explored, and alignment between vision and text information is enhanced through a heterogeneous modeling mechanism. And data features from different modalities are effectively integrated. More comprehensive understanding of the image content is ensured, and the performance of corresponding evaluation indexes is remarkably improved. The proposed heterogeneous attention module exhibits excellent ability in generating an accurate and contextually rich description of an image containing text elements.

Description

Technical Field

[0001] The present invention relates to the technical field of OCR image description, and more particularly to a method and system for generating OCR image description based on heterogeneous representation. Background Art

[0002] At present, computer vision and multimodal fusion technologies have been rapidly developed and widely used in human society. Due to the development of visual-language learning, multimodal tasks of images or videos have attracted more and more attention. In the field of multimodal tasks, traditional image description generation tasks aim to generate descriptive captions based on given images, usually giving priority to identifying and describing salient visual elements in the image, and often ignoring the large amount of textual information embedded in it. This may cause the image description provided by the model to only stay on the surface and fail to dig deep into its meaning, thus failing to fully capture the human semantic dimension in the image. One specific reason for this limitation is that existing image description datasets mainly focus on describing visual object elements, but pay less attention to the textual content embedded in the image.

[0003] Therefore, how to provide an OCR image description generation method and system based on heterogeneous representation, integrate visual and textual information, and capture the relationship between the two, so as to generate more accurate and semantically rich image description is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention provides a method and system for generating OCR image description based on heterogeneous representation, and constructs a new framework for generating text-based image description. This framework effectively solves the challenge of fully understanding the multimodal representation of scene text OCR tags by introducing a heterogeneous attention module. This framework can not only extract and fuse visual and textual information, but also capture the relationship between the two, thereby generating more accurate and semantically rich image descriptions.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: a method for generating OCR image description based on heterogeneous representation, comprising:

[0006] Get an image I containing M visual objects and N scene texts;

[0007] Encode all features of the image I based on the Faster-RCNN model pre-trained on the dataset to obtain object visual features and OCR marking features;

[0008] Project the object visual features to the same dimension as the LSTM hidden state through linear transformation to obtain the corresponding object-visual embedding;

[0009] Perform text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding;

[0010] Inputting the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, the heterogeneous attention module introduces heterogeneous similarity calculation, maps the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space, and obtains the OCR visual scene and the OCR text scene;

[0011] An image description model is constructed, and the OCR visual scene, OCR text scene and object-visual embedding are input into the image description model to obtain an image description based on visual objects and scene text.

[0012] Preferably, performing text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding includes:

[0013] Use the CTC model to perform text recognition on the OCR mark features to obtain spatial features, regional features, FastText features, and PHOC features;

[0014] The spatial features and the regional features are encoded to obtain OCR visual embedding; the FastText features and the PHOC features are encoded to obtain OCR text embedding.

[0015] Preferably, the object visual feature is represented as

[0016] The OCR mark feature is expressed as

[0017] Preferably, for the mth object, the object-visual embedding is represented as:

[0018]

[0019] Among them, W obj is a learnable weight matrix, l2 represents the normalization function, LN represents the ReLU activation function, and d represents the hidden layer size of the LSTM layer. Represents the visual feature of the mth object.

[0020] Preferably, encoding the spatial features and the regional features to obtain OCR visual embedding includes:

[0021]

[0022] Among them, W ocr_v is the weight matrix, [,] represents the concatenation operation, b1 represents the linear learning parameter, represents OCR visual embedding; Represents regional characteristics, Represents spatial features;

[0023] Encoding the FastText feature and the PHOC feature to obtain OCR text embedding includes:

[0024]

[0025] Among them, W ocr_t is the weight matrix, b2 represents the linear learning parameter, represents OCR text embedding, Represents FastText features, Indicates PHOC characteristics.

[0026] Preferably, embed OCR vision into and OCR text embedding After being input into the heterogeneous attention module, the ratio of heterogeneous fusion is controlled by the hyperparameter α, and the result is:

[0027]

[0028] Embed the heterogeneous fusion OCR vision and OCR text embedding The OCR images are input into the same first multi-layer perceptron respectively to complete the feature interaction between OCR text and OCR vision.

[0029] Preferably, in the first multi-layer perceptron, the OCR visual embedding and OCR text embedding Map the input features to hidden features through the linear layer;

[0030] Feature encoding through linear transformation;

[0031] The encoded features undergo another linear transformation and then are normalized by softmax to generate weighted feature representations;

[0032] These weighted features are concatenated to form the feature interaction result between OCR text and OCR vision.

[0033] Preferably, according to the feature interaction result between the formed OCR text and the OCR vision, a heterogeneous similarity calculation result is obtained by a second multi-layer perceptron;

[0034] The heterogeneous similarity calculation results are input into the third multi-layer perceptron after matrix multiplication and concatenation operations to obtain the final heterogeneous representation results.

[0035] Preferably, the OCR visual scene, the OCR text scene and the object-visual embedding are input into the image description model to obtain an image description based on visual objects and scene text, including:

[0036] Obtaining a scene graph based on the OCR visual scene, the OCR text scene, and the object-visual embedding;

[0037] The input image is represented by a scene graph G(V,E), where V is the node of all objects and OCR marks, and E is the edge;

[0038] Processing objects and OCR labeled entities separately based on multi-relation graph convolutional networks The relationship R between OCR and object is transformed nonlinearly by aggregating the weighted sum of its own features and neighbor features, and outputting the i-th enhanced visual and OCR text embedding in the l+1-th layer multi-relation graph convolutional network;

[0039] By minimizing the cross entropy loss L of the output description sentence CE to optimize the model and assign unique sources to words.

[0040] Preferably, the OCR image description generation system based on heterogeneous representation includes:

[0041] An image acquisition module, used to acquire an image I containing M visual objects and N scene texts;

[0042] A feature encoding module, used for encoding all features of the image I based on the Faster-RCNN model pre-trained on the data set to obtain object visual features and OCR marking features;

[0043] The linear transformation module is used to project the object visual features to the same dimension as the LSTM hidden state through linear transformation to obtain the corresponding object-visual embedding;

[0044] The text recognition module is used to perform text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding;

[0045] A mapping unification module is used to input the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, and the heterogeneous attention module introduces heterogeneous similarity calculation to map the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space to obtain OCR visual scenes and OCR text scenes;

[0046] The image description module is used to construct an image description model, input the OCR visual scene, OCR text scene and object-visual embedding into the image description model, and obtain an image description based on visual objects and scene text.

[0047] Through the above technical solution, it can be known that compared with the prior art, the present invention discloses a method and system for generating OCR image description based on heterogeneous representation, including: obtaining an image I containing M visual objects and N scene texts; encoding all features of the image I based on a Faster-RCNN model pre-trained on a data set to obtain object visual features and OCR mark features; projecting the object visual features to the same dimension as the LSTM hidden state through a linear transformation to obtain a corresponding object-visual embedding; performing text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding; inputting the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, the heterogeneous attention module introduces heterogeneous similarity calculation, and maps the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space to obtain OCR visual scenes and OCR text scenes; constructing an image description model, and inputting the OCR visual scene, the OCR text scene and the object-visual embedding into the image description model to obtain an image description based on visual objects and scene texts. The present invention extracts features based on the detected OCR area from the visual and textual aspects respectively, uses Faster-RCNN for text detection, and performs text recognition; OCR visual embedding includes bounding box features and Faster-RCNN visual features, and OCR text embedding includes PHOC features and FastText features. The present invention is able to explore the internal consistency of OCR features and enhance the alignment between visual and textual information through a heterogeneous modeling mechanism. It effectively integrates data features from different modalities. It not only ensures a more comprehensive understanding of the image content, but also significantly improves the performance on the corresponding evaluation indicators. The proposed heterogeneous attention module demonstrates excellent ability in generating accurate and context-rich descriptions of images containing text elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A schematic flow chart of a method for generating OCR image description based on heterogeneous representation provided in an embodiment of the present invention.

[0050] Figure 2 A schematic diagram of the structure of a heterogeneous attention module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The embodiment of the present invention discloses a method for generating OCR image description based on heterogeneous representation, such as Figure 1 As shown, including:

[0053] Get an image I containing M visual objects and N scene texts; given an image I containing M objects and N scene texts, the text-based image description model aims to automatically generate a natural language sentence C = {w1,…,W T}, where wi 是 The i-th word, T is the number of words.

[0054] Based on the Faster-RCNN model pre-trained on the dataset, all features of the image I are encoded to obtain object visual features and OCR marking features; for each visual object, all visual features are first encoded based on the Faster-RCNN model pre-trained on the Visual Genome dataset.

[0055] Specifically, the object visual feature is expressed as

[0056] The OCR mark feature is expressed as

[0057] Project the object visual features to the same dimension as the LSTM hidden state through linear transformation to obtain the corresponding object-visual embedding;

[0058] Specifically, before the object features are input into the image description model, they are projected to the same dimension as the LSTM hidden state through a linear transformation. For the mth object, the corresponding object-visual embedding representation is obtained as follows:

[0059]

[0060] Among them, W obj is a learnable weight matrix, l2 represents the normalization function, LN represents the ReLU activation function, and d represents the hidden layer size of the LSTM layer. Represents the visual feature of the mth object.

[0061] Text recognition is performed on the OCR mark features to obtain OCR visual embedding and OCR text embedding; unlike general visual objects, scene text naturally has features of both visual and text modes. For each OCR mark, the embodiment of the present invention extracts four features: spatial features (bounding box coordinates): Faster-RCNN features (regional features): FastText Features: PHOC characteristics:

[0062] The above OCR mark features are divided into two aspects: visual and text. and bounding box coordinates is regarded as the OCR visual embedding part and encoded as Similarly, OCR text embedding By PHOC Features and FastText features to encode.

[0063] Specifically, text recognition is performed on the OCR mark features to obtain OCR visual embedding and OCR text embedding, including:

[0064] Use the CTC model to perform text recognition on the OCR mark features to obtain spatial features, regional features, FastText features, and PHOC features;

[0065] The spatial features and the regional features are encoded to obtain OCR visual embedding; the FastText features and the PHOC features are encoded to obtain OCR text embedding.

[0066] Specifically, encoding the spatial features and the regional features to obtain OCR visual embedding includes:

[0067]

[0068] Among them, W ocr_v is the weight matrix, [,] represents the concatenation operation, b1 represents the linear learning parameter, represents OCR visual embedding; Represents regional characteristics, Represents spatial features;

[0069] Encoding the FastText feature and the PHOC feature to obtain OCR text embedding includes:

[0070]

[0071] Among them, Wocr_t is the weight matrix, b2 represents the linear learning parameter, represents OCR text embedding, Represents FastText features, Indicates PHOC characteristics.

[0072] In previous work, they embedded each OCR tag through simple l2 normalization and linear transformation to fuse multiple features, but ignored the correlation between different modalities. The HAM (Heterogeneous Attention Module) proposed in the embodiment of the present invention aims to capture the coherence between different modalities in OCR tags, interactively enhance the alignment between visual and text features, and provide context-aware scene text representation, so as to generate accurate text-based image descriptions.

[0073] Inputting the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, the heterogeneous attention module introduces heterogeneous similarity calculation, maps the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space, and obtains the OCR visual scene and the OCR text scene;

[0074] The embodiment of the present invention first introduces a hyperparameter α to control the ratio of heterogeneous fusion; Figure 2 As shown, the input of the HAM is and They first pass through the same first multilayer perceptron (MLP). In the first multilayer perceptron, and Initially passing through a linear layer, which maps the input features to hidden features, the OCR features are encoded through a linear transformation. The encoded OCR features are then subjected to another linear transformation and then normalized by softmax to generate weighted feature representations. These weighted feature representations are concatenated to form e t and e v , thereby completing the feature interaction between OCR text and OCR vision.

[0075] Specifically, embed OCR vision into and OCR text embedding After being input into the heterogeneous attention module, the ratio of heterogeneous fusion is controlled by the hyperparameter α, and the result is:

[0076]

[0077] Embed the heterogeneous fusion OCR vision and OCR text embedding The OCR images are input into the same first multi-layer perceptron respectively to complete the feature interaction between OCR text and OCR vision.

[0078] Specifically, in the first multi-layer perceptron, OCR visual embedding and OCR text embedding Map the input features to hidden features through the linear layer;

[0079] Feature encoding through linear transformation;

[0080] The encoded features undergo another linear transformation and then are normalized by softmax to generate weighted feature representations;

[0081] These weighted features are spliced ​​to form the feature interaction result between OCR text and OCR vision; t and e v Represents the feature interaction result between OCR text and OCR vision.

[0082] Once you have obtained e t and e v , they will pass through the second multi-layer perceptron to obtain ω v and ω t . The calculation of heterogeneous similarity is completed here.

[0083] Specifically, according to the feature interaction result between the formed OCR text and the OCR vision, the heterogeneous similarity calculation result is obtained through the second multi-layer perceptron;

[0084] After matrix multiplication and concatenation, the obtained heterogeneous similarity calculation results are input into the third multi-layer perceptron to obtain the final heterogeneous representation results. and Represents the final heterogeneous characterization results.

[0085] Specifically, through ω v and ω t The calculation results of heterogeneous similarity are expressed as follows:

[0086]

[0087] Among them, w vt ,w tv ,w v ,w t is the weight matrix, Γ vt ,Γ tv ,Γ v ,Γ t Represents different networks, which are simply designed as fully connected layers (FC); the calculation of heterogeneous similarity is completed here.

[0088] ω v and ω tAfter matrix multiplication and concatenation before inputting into the third multilayer perceptron, the final third multilayer perceptron MLP contains a residual connection layer, thus obtaining and As shown below:

[0089]

[0090] Represents different fully connected layer networks.

[0091] The embodiment of the present invention maps visual and text features that were originally located in different dimensional spaces into a unified feature space by introducing heterogeneous similarity calculation. This process ensures that the unique characteristics of each modality are preserved, while being able to more effectively fuse these heterogeneous representations and optimize the key step in multimodal tasks: feature alignment. In addition, the inclusion of trainable matrices in the similarity calculation provides a dynamic and flexible mechanism for the module to adjust and optimize feature alignment. These matrices are learned during the training process, enabling the module to adapt to the specific characteristics and nuances of the data. It can be seen that the heterogeneous similarity calculation consists of multiple linear layers and trainable matrices. By using multiple linear layers in the similarity calculation process, HAM ensures that the module can learn complex patterns and interactions in the data. By stacking these layers, the module gains the ability to perform deep and hierarchical feature extraction, gradually improving the understanding of the data at each step. Therefore, the embodiment of the present invention can seamlessly integrate and associate data information from vision and text, enabling the model to better understand and analyze complex scenes. This approach not only enhances the interpretability of multimodal data, but also significantly improves the accuracy of the model in generating meaningful descriptions, ultimately making the understanding of the presented information more in-depth and comprehensive.

[0092] An image description model is constructed, and the OCR visual scene, OCR text scene and object-visual embedding are input into the image description model to obtain an image description based on visual objects and scene text.

[0093] In a specific embodiment of the present invention, following the previous process, the embodiment of the present invention extracts features based on the detected OCR area from visual and textual aspects respectively. The embodiment of the present invention uses Rosetta as the OCR system. Rosetta is a two-step text detection model architecture that uses Faster-RCNN for text detection and uses the CTC model for text recognition. OCR-visual embedding includes bounding box features and Faster-RCNN visual features, while OCR-textual embedding includes PHOC features and FastText features. The main contribution of the embodiment of the present invention is to explore the internal consistency of OCR features and enhance the alignment between visual and textual information through a heterogeneous modeling mechanism. By utilizing heterogeneous modeling, the embodiment of the present invention effectively integrates data features from different modalities in one model. This approach not only ensures a more comprehensive understanding of the image content, but also significantly improves the performance on the corresponding evaluation indicators. The framework overview of the embodiment of the present invention is as follows. Figure 1 shown.

[0094] Among them, visual and text embeddings come from heterogeneous encoders. Heterogeneous data provides multiple sources of information, but also brings difficulties in interaction and fusion. In order to extract and integrate context-aware representations from OCR-visual embedding and OCR-textual embedding, an embodiment of the present invention proposes a heterogeneous attention module (HeterogeneousAttentionModule, HAM) to solve the complexity of heterogeneous data interaction and fusion by establishing robust relationships between different modalities. The core concept of HAM is to use the complementarity of visual and textual information to achieve a more comprehensive understanding of the input data. In addition, HAM uses a specific similarity measure to calculate the heterogeneous similarity between visual and text embeddings. The architecture of HAM is as follows Figure 2 shown.

[0095] The heterogeneous attention module aims to capture the consistency of OCR tags between different modalities, provide context-aware scene text representation for text-based image description tasks, and establish heterogeneous relationships between visual and text features of OCR tags by calculating heterogeneous similarities. The method of the embodiment of the present invention is based on the premise that effective text-based image description requires a detailed understanding and integration of multiple features. To this end, the embodiment of the present invention comprehensively analyzes the importance and necessity of using various heterogeneous strategies for input feature modeling from theoretical and experimental perspectives. The experiments of the embodiment of the present invention on the TextCaps validation set and test set show that the proposed framework has a significant effect in improving the accuracy of description. In addition, the embodiment of the present invention emphasizes the potential and importance of inter-modal alignment in generating accurate descriptions. This further highlights the value of exploring more complex heterogeneous relationships in OCR tags in future research, especially in the case of higher data quality. Such progress may bring greater improvements in the authenticity and contextual accuracy of generated descriptions, paving the way for more robust and versatile text-based image description systems.

[0096] The embodiment of the present invention represents the input image with a scene graph G(V,E). V is the node of all objects and OCR tags, and E is the edge. Since not all relationships between objects and OCR tags are useful for text-based image description, the embodiment of the present invention does not build a graph on all objects and OCR tags in the image, but only retains the edges between OCR tags and their attached objects, while deleting the edges with other irrelevant objects. The embodiment of the present invention processes objects and OCR tag entities by using a multi-relational graph convolutional network (MR-GCN). As well as the relationship R between OCR and objects to enrich the visual embedding. This information is input into this single-layer network, and the next layer l+1 outputs enhanced features.

[0097] Specifically, the OCR visual scene, the OCR text scene, and the object-visual embedding are input into the image description model to obtain an image description based on visual objects and scene text, including:

[0098] Obtaining a scene graph based on the OCR visual scene, the OCR text scene, and the object-visual embedding;

[0099] The input image is represented by a scene graph G(V,E), where V is the node of all objects and OCR marks, and E is the edge;

[0100] Processing objects and OCR labeled entities separately based on multi-relation graph convolutional networks The relationship R between OCR and object is transformed nonlinearly by aggregating the weighted sum of its own features and neighbor features, and outputting the i-th enhanced visual and OCR text embedding in the l+1-th layer multi-relation graph convolutional network;

[0101] By minimizing the cross entropy loss L of the output description sentence CE to optimize the model and assign unique sources to words.

[0102] The update rules for node features in the multi-relational graph convolutional network are as follows:

[0103]

[0104] Where σ is the ReLU activation function; The weight matrix representing the features of node i in layer 1; represents the i-th enhanced visual and OCR text embedding in the l+1-th layer GCN; R represents the set of relations and different types of edges; represents the set of neighbor embedding nodes of node i under relation r; is the neighbor embedding of node x i the number of is the weight matrix of the neighbor embedding node features of relation r in the first layer; is the j-th enhanced visual and OCR text embedding in the first layer of GCN.

[0105] In GCN, the embodiment of the present invention uses this formula to update the node feature representation by aggregating the weighted sum of its own features and neighbor features, and then performing a nonlinear transformation.

[0106] Since the words in a sentence come from a fixed vocabulary and OCR tags, the embodiment of the present invention assigns a unique source to the words. Therefore, the embodiment of the present invention minimizes the cross entropy loss L of the output description sentence. CE To optimize the model, specifically by word y t Sequence to optimize:

[0107]

[0108] For the input image, the embodiment of the present invention sets the number of detected target areas to 100. For each area, the embodiment of the present invention extracts 2048-dimensional visual features and corresponding bounding boxes. For OCR tags, the embodiment of the present invention uses the results of Rosetta OCR detection and extracts corresponding appearance features. In addition, the embodiment of the present invention also uses corresponding tools to extract 604-dimensional PHOC features and 300-dimensional FastText features. The experiment was conducted using the MMF framework. In order to construct a fixed vocabulary, the embodiment of the present invention retains words that appear more than 10 times, and the rest are regarded as unknown words and marked as <unk>. The embodiment of the present invention sets the dimension of word embedding, the hidden layer size of the LSTM layer, and the output size of GCN to 1000. The model was trained for 30 cycles with a batch size of 50. During the training process, the embodiment of the present invention used an initial learning rate of 2e-4, which was reduced by 0.8 times every 3 cycles. In the inference stage, the embodiment of the present invention selected the model with the highest CIDEr score for evaluation to ensure the best performance. The performance of the embodiment of the present invention was tested on the TextCaps dataset, using BLEU-4, METOR, Rouge-L, and CIDEr as measurement indicators, and achieved significant results.

[0109] In a specific embodiment of the present invention, a heterogeneous representation-based OCR image description generation system includes:

[0110] An image acquisition module, used to acquire an image I containing M visual objects and N scene texts;

[0111] A feature encoding module, used for encoding all features of the image I based on the Faster-RCNN model pre-trained on the data set to obtain object visual features and OCR marking features;

[0112] The linear transformation module is used to project the object visual features to the same dimension as the LSTM hidden state through linear transformation to obtain the corresponding object-visual embedding;

[0113] The text recognition module is used to perform text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding;

[0114] A mapping unification module is used to input the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, and the heterogeneous attention module introduces heterogeneous similarity calculation to map the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space to obtain OCR visual scenes and OCR text scenes;

[0115] The image description module is used to construct an image description model, input the OCR visual scene, OCR text scene and object-visual embedding into the image description model, and obtain an image description based on visual objects and scene text.

[0116] At the same time, the embodiments of the present invention have conducted extensive experiments on the TextCaps dataset, and the results show that the proposed framework performs well on the validation set and the test set, effectively improving the accuracy of image description and the ability to understand context. This shows that the embodiments of the present invention have significant advantages in integrating visual and textual information to generate higher quality descriptions, providing a new direction for future multimodal research.

[0117] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0118] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / unk>

Claims

1. An OCR image description generation method based on heterogeneous representation, characterized in that: include: Get an image I containing M visual objects and N scene texts; Encode all features of the image I based on the Faster-RCNN model pre-trained on the dataset to obtain object visual features and OCR marking features; Project the object visual features to the same dimension as the LSTM hidden state through linear transformation to obtain the corresponding object-visual embedding; Perform text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding; Inputting the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, the heterogeneous attention module introduces heterogeneous similarity calculation, maps the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space, and obtains the OCR visual scene and the OCR text scene; An image description model is constructed, and the OCR visual scene, OCR text scene and object-visual embedding are input into the image description model to obtain an image description based on visual objects and scene text.

2. The method for generating OCR image description based on heterogeneous representation according to claim 1, characterized in that: Perform text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding, including: Use the CTC model to perform text recognition on the OCR mark features to obtain spatial features, regional features, FastText features, and PHOC features; The spatial features and the regional features are encoded to obtain OCR visual embedding; the FastText features and the PHOC features are encoded to obtain OCR text embedding.

3. The method for generating OCR image description based on heterogeneous representation according to claim 1, characterized in that: The object visual feature is expressed as The OCR mark feature is expressed as 4. The method for generating OCR image description based on heterogeneous representation according to claim 1, characterized in that: For the mth object, the object-visual embedding is represented as: Among them, W obj is a learnable weight matrix, l2 represents the normalization function, LN represents the ReLU activation function, and d represents the hidden layer size of the LSTM layer. Represents the visual feature of the mth object.

5. The method for generating OCR image description based on heterogeneous representation according to claim 2, characterized in that: Encoding the spatial features and the regional features to obtain OCR visual embedding includes: Among them, W ocr_v is the weight matrix, [,] represents the concatenation operation, b1 represents the linear learning parameter, represents OCR visual embedding; Represents regional characteristics, Represents spatial features; Encoding the FastText feature and the PHOC feature to obtain OCR text embedding includes: Among them, W ocr_t is the weight matrix, b2 represents the linear learning parameter, represents OCR text embedding, Represents FastText features, Indicates PHOC characteristics.

6. The method for generating OCR image description based on heterogeneous representation according to claim 1, characterized in that: Embed OCR vision and OCR text embedding After being input into the heterogeneous attention module, the ratio of heterogeneous fusion is controlled by the hyperparameter α, and the result is: Embed the heterogeneous fusion OCR vision and OCR text embedding The OCR images are input into the same first multi-layer perceptron respectively to complete the feature interaction between OCR text and OCR vision.

7. The method for generating OCR image description based on heterogeneous representation according to claim 6, characterized in that: In the first multi-layer perceptron, OCR visual embedding and OCR text embedding Map the input features to hidden features through the linear layer; Feature encoding through linear transformation; The encoded features undergo another linear transformation and then are normalized by softmax to generate weighted feature representations; These weighted features are concatenated to form the feature interaction result between OCR text and OCR vision.

8. The method for generating OCR image description based on heterogeneous representation according to claim 7, characterized in that: According to the feature interaction results between the formed OCR text and the OCR vision, the heterogeneous similarity calculation results are obtained through the second multi-layer perceptron; The heterogeneous similarity calculation results are input into the third multi-layer perceptron after matrix multiplication and concatenation operations to obtain the final heterogeneous representation results.

9. The method for generating OCR image description based on heterogeneous representation according to claim 8, characterized in that: Inputting the OCR visual scene, OCR text scene and object-visual embedding into the image description model to obtain an image description based on visual objects and scene text, including: Obtaining a scene graph based on the OCR visual scene, the OCR text scene, and the object-visual embedding; The input image is represented by a scene graph G(V,E), where V is the node of all objects and OCR marks, and E is the edge; Processing objects and OCR labeled entities separately based on multi-relation graph convolutional networks The relationship R between OCR and object is transformed nonlinearly by aggregating the weighted sum of its own features and neighbor features, and outputting the i-th enhanced visual and OCR text embedding in the l+1-th layer multi-relation graph convolutional network; By minimizing the cross entropy loss L of the output description sentence CE to optimize the model and assign unique sources to words.

10. An OCR image description generation system based on heterogeneous representation, characterized in that: include: An image acquisition module, used to acquire an image I containing M visual objects and N scene texts; A feature encoding module, used for encoding all features of the image I based on the Faster-RCNN model pre-trained on the data set to obtain object visual features and OCR marking features; The linear transformation module is used to project the object visual features to the same dimension as the LSTM hidden state through linear transformation to obtain the corresponding object-visual embedding; The text recognition module is used to perform text recognition on the OCR mark features to obtain OCR visual embedding and OCR text embedding; A mapping unification module is used to input the OCR visual embedding and the OCR text embedding into a heterogeneous attention module, and the heterogeneous attention module introduces heterogeneous similarity calculation to map the OCR visual embedding and the OCR text embedding originally located in different dimensional spaces into a unified feature space to obtain OCR visual scenes and OCR text scenes; The image description module is used to construct an image description model, input the OCR visual scene, OCR text scene and object-visual embedding into the image description model, and obtain an image description based on visual objects and scene text.

Images