A method for generating a text subtitle based on a main target scene

Abstract

The application discloses a kind of based on main target scene-oriented text subtitle generation method, it is related to image description generation technical field technical field.Processing to obtain the sample data including about the object region frame coordinates of each object in M objects, object visual feature, object representation vector and about the text region frame coordinates of each text in N texts, text visual feature, text representation vector of image;Establish the graph relationship between object region frame coordinates and text region frame coordinates;Graph convolution network model is enhanced according to graph relationship to object representation vector, text representation vector and obtains enhanced feature;Enhanced feature is sent to decoding model and is decoded to obtain the sentence expressing picture content.Introduce the main target concept of scene text, remove the redundancy and interference information in scene graph.Pay attention to object in image, also focus on the relationship between text and object in image, and the generated sentence is more accurate for the description of scene text subtitle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image description generation, and more particularly to a scene-oriented text subtitle generation method based on a main target. BACKGROUND

[0002] Image description generation is a research hotspot in cross-media understanding, which spans computer vision and natural language processing. However, the existing visual-linguistic cross-media research ignores the important semantic information and reasoning clues of scene text. Cross-modal understanding of natural scene text has important academic significance and application value for image description generation.

[0003] The existing scene text-oriented image caption generation method is mostly based on a traditional image caption generation model, which adds features extracted according to scene text detection results to the input end of the model, and then fuses the encoding. On the basis of this framework, subsequent research adds a global scene graph between the text and the target, which partially improves the effect. However, this method ignores the interaction between scene text and image targets. In reality, scene text is mainly dependent on a main target in the scene, and has little relevance to other targets. The global scene graph introduces noise information of the text and other non-main targets, which affects the accuracy of image caption generation.

[0004] Starting from this problem, the present application proposes a new scene text-oriented caption generation model based on a main target, which optimizes the scene graph centered on scene text to improve the accuracy of image caption generation. In practical applications, scene text information and target information in each image in the encapsulated data set are detected, and the main target of the scene text is found according to the spatial position relationship. At the same time, the detection results of the scene text are clustered to ensure the integrity of the text. The detected text and target feature representation are used as graph nodes, and then a graph convolution network is used for feature enhancement. At the same time, based on the LSTM network, an adaptive decoding part with a gating mechanism is designed, and the weights of the visual features and the scene text visual features are adjusted. First, the concept of the main target of the scene text is defined, thereby solving the problem of redundant and interfering information in the scene graph. Secondly, not only the objects in the image are focused on, but also the relationship between the objects constructed by the text related to the image is focused on, and the sentences generated by the caption generation model are more accurate in describing the scene text. SUMMARY

[0005] Therefore, the present application provides a scene-oriented text subtitle generation method based on a main target, which achieves the purpose of removing redundant and interfering information in the scene graph.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] A text subtitle generation method based on a main target scene, specifically comprising the following steps:

[0008] Preprocessing the image to obtain sample data, wherein the sample data includes object region frame coordinates, object visual features, and object representation vectors for each of the M objects, and the sample data further includes text region frame coordinates, text visual features, and text representation vectors for each of the N texts;

[0009] Extracting the object region frame coordinates and the text region frame coordinates, and establishing a graph relationship therebetween;

[0010] The graph convolution network model enhances the object representation vectors and the text representation vectors according to the graph relationship to obtain enhanced features;

[0011] Sending the enhanced features to a decoding model for decoding to obtain sentences expressing the content of the picture.

[0012] Further, M is a positive integer greater than zero in the M objects, and N is a positive integer greater than zero in the N texts.

[0013] Optionally, the preprocessing step comprises:

[0014] Using a Faster-RCNN network to extract region features from the image to obtain the object region frame coordinates and the object visual features of the M objects, and processing the object visual features through linear transformation, layer standardization regularization, and ReLU activation function to obtain the object representation vectors;

[0015] Extracting features from the image through an optical character recognition tool to obtain the text region frame coordinates of the N texts, using a pre-trained ResNet101 network to extract features from the text region to obtain corresponding N text visual features, using a FastText tool to obtain a semantic text vector for each text, using a Phoc method to process the text characters at varying pyramid level scales to obtain a text vector, and processing the text vector through linear transformation, layer standard regularization LN, and ReLU activation function to obtain the text representation vectors.

[0016] Optionally, the method for establishing the graph relationship comprises:

[0017] Taking the text representation vectors as vertices V, performing edge building on each of the N texts to obtain an edge set E and a main target in the object visual features, and establishing a graph relationship G(V, E) through the vertices V and the edge set E; wherein the established graph relationship is a sparse graph relationship;

[0018] The method for performing edge building on each of the N texts comprises:

[0019] If the region of the character is completely contained in the region of the object, the character is attached to the corresponding object, and an edge e is established between the character and the object ij ∈ E.

[0020] If the region of the character is contained in the regions of multiple objects, the object region with the smallest area is selected from the multiple object regions as the adjacent node of the character, and an edge is established between the character and the object corresponding to the object region with the smallest area

[0021] Optionally, the method for processing the relationship graph by using the graph convolution network model comprises the following steps.

[0022] The node features input by the graph convolution network are composed of the object representation vector and the character representation vector.

[0023] The node features are calculated by one layer of the graph convolution network MR-GCN to obtain enhanced features.

[0024] Optionally, the decoding model is iterated multiple times until the end-of-sentence mark is predicted.

[0025] After the multiple iterations are completed, the sentence describing the content of the picture is obtained according to the index and the word-number correspondence table.

[0026] Each of the multiple iterations comprises the following processes.

[0027] The decoding vector obtained in the last prediction, the word vector obtained in the last prediction, the average value of the enhanced features input into the decoding model, and the weighted features obtained in the last prediction are spliced to obtain the decoding vector for this prediction, wherein the weighted features are obtained by attention mechanism calculation.

[0028] The weighted decoding vector for this prediction is obtained after the proportion of the character visual features in the decoding vector is adjusted by using a gating mechanism, and the weighted decoding vector for this prediction is calculated by a linear classification network to obtain the prediction score for this prediction.

[0029] The index of the maximum prediction score is obtained in the prediction score.

[0030] Optionally, in the tth iteration of the loop iteration, the formula for splicing the decoding vector obtained in the last prediction, the word vector obtained in the last prediction, the average value of the enhanced features input into the decoding model, and the weighted features obtained in the last prediction for this iteration comprises:

[0031] The vector splicing calculation formula is:

[0032]

[0033] The vector splicing calculation formula is:

[0034]

[0035] In the formula: x t represents the decoding vector input this time, h t represents the decoding vector of the tth prediction, y t-1 represents the word vector of the (t-1)th prediction, represents the average value of the enhanced features input into the decoding model, represents the weighted features of the (t-1)th prediction;

[0036] The average value of the enhanced features input into the decoding model is calculated according to the following formula:

[0037]

[0038] In the formula: represents the average value of the enhanced features, i represents the sequence index of the enhanced features, represents the ith enhanced feature in the total number M+N of enhanced features;

[0039] h t ,c t =LSTM(h t-1 ,c t-1 ,x t )

[0040] In the formula: c t represents the cell state of the current tth step, h t-1 represents the decoding vector of the last time, c t-1 represents the cell state of the last time;

[0041] The attention mechanism is calculated according to the following formula:

[0042]

[0043] In the formula: represents the weighted features of the tth step, and X represents the set of enhanced features.

[0044] In the tth iteration of the loop, the method for obtaining the weighted decoding vector and the prediction score comprises:

[0045] Using a logistic regression model to obtain the gating weight and the word weight corresponding to the word visual feature;

[0046] Using a decoding vector weighting formula to obtain the weighted decoding vector, wherein the decoding vector weighting formula adjusts the proportion of the word visual feature in the decoding vector, and captures the decoding vector that needs to be retained through the gating weight;

[0047] After linear classification of the weighted decoding vector by a linear classification network, a prediction score of the current prediction is obtained.

[0048] The calculation formula of the logistic regression model is as follows:

[0049]

[0050] In the formula, a1 represents a gating weight, a N+1 represents a character weight corresponding to the Nth character visual feature, represents transforming the intermediate process vector into a row vector, W vis represents linearly transforming the decoding vector, h t represents the decoding vector of this time, W txt represents linearly transforming the character representation vector;

[0051] The calculation formula of the decoding vector weighting is as follows:

[0052]

[0053] In the formula, represents the weighted decoding vector of this time, a1 represents a gating weight, a k represents a character weight corresponding to the k-1th character visual feature, represents the k-1th character visual feature;

[0054] The calculation formula of the linear classification network is as follows:

[0055]

[0056] The calculation formula of the linear classification network is as follows:

[0057]

[0058] In the formula, s t represents a prediction score, W fc represents linearly dimensionally transforming the weighted decoding vector, b fc represents an offset term of linear transformation.

[0059] Optionally, in the tth iteration of the loop iteration, the formula for obtaining the index of the maximum prediction score in the prediction score is as follows:

[0060] index = argmax(s t )

[0061] In the formula, index represents an index, s t represents a prediction score.

[0062] Optionally, the method further includes a training method of the decoding network:

[0063] The probability value of the classification to which the prediction score of the tth cycle belongs is obtained by using a logistic regression function for probability conversion;

[0064] Based on the probability value, the decoding model parameters are trained and optimized using a loss function;

[0065] Loss function:

[0066]

[0067] In the formula, L represents the loss function, T represents the maximum length of the predicted description sentence, t represents the cycle number, C represents the number of classifications, idx represents the number of the idxth classification, y represents the score of the true value of the data itself, represents the probability value of the idxth classification of the tth cycle;

[0068]

[0069] In the formula: represents the probability value of the 1st classification of the tth cycle, represents the probability value of the Cth classification of the tth cycle, s t represents the prediction score.

[0070] The process of training and optimizing the decoding model parameters by using the loss function further includes: judging whether the loss function converges, and if the result of the judgment is yes, obtaining the decoding network model of this time, and if the result of the judgment is no, sending the enhanced features into the model decoding network again.

[0071] According to the technical solution, compared with the prior art, the application provides a scene-oriented text subtitle generation method based on a main target, so that the following beneficial effects can be obtained:

[0072] 1. The concept of the main target of scene text is defined, so that the redundant and interfering information in the scene graph is removed.

[0073] 2. Not only the objects in the image are focused on, but also the relationship between the objects constructed by the text related to the image is focused on, and from the generated result, it can be directly seen that the generated sentence is more accurate in describing the scene text. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0075] Figure 1The accompanying drawings are schematic diagrams of the flow of the present application.

[0076] Figure 2 The accompanying drawings are schematic diagrams of the decoding model structure of the present application. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of the present application.

[0078] As Figure 1 , Figure 2 The embodiment of the present application discloses a text subtitle generation method based on a main target scene, and the specific steps include:

[0079] The image is preprocessed to obtain sample data, wherein the sample data includes object region frame coordinates, object visual features and object representation vectors of each object in M objects, and the sample data also includes text region frame coordinates, text visual features and text representation vectors of each text in N texts;

[0080] The object region frame coordinates and the text region frame coordinates are extracted, and a graph relationship between them is established;

[0081] The graph convolution network model enhances the object representation vectors and the text representation vectors according to the graph relationship to obtain enhanced features;

[0082] The enhanced features are sent to a decoding model for decoding to obtain sentences expressing the content of the picture.

[0083] Further, the sample data includes a plurality of object region frame coordinates, a plurality of object visual features, a plurality of object representation vectors, a plurality of text region frame coordinates, a plurality of text visual features and a plurality of text representation vectors, and the object region frame coordinates, the object visual features and the object representation vectors are one-to-one corresponding, and the text region frame coordinates, the text visual features and the text representation vectors are one-to-one corresponding.

[0084] Optionally, the preprocessing step includes:

[0085] S1: using a Faster-RCNN network to extract region features of the image to obtain object region frame coordinates and object visual features of M objects;

[0086] A set is established for the object region frame coordinates:

[0087]

[0088] B = {B1, B2, …, Bn} (1) obj denotes a set of object region frame coordinates, denotes the first object region frame coordinate, denotes the mth object region frame coordinate, denotes the Mth object region frame coordinate.

[0089] A set of object visual features is established:

[0090]

[0091] V = {V1, V2, …, Vn} (2) obj denotes a set of text region frame coordinates, denotes the first text region frame coordinate, denotes the mth text region frame coordinate, denotes the Mth text region frame coordinate.

[0092] S2: Feature extraction is performed on the image by an optical character recognition tool to obtain text region frame coordinates of N texts;

[0093] A set of text region frame coordinates is established:

[0094]

[0095] B = {B1, B2, …, Bn} (1) ocr denotes a set of text region frame coordinates, denotes the first text region frame coordinate, denotes the n th text region frame coordinate, denotes the Nth text region frame coordinate.

[0096] S3: Feature extraction is performed on the image using a ResNet101 network to obtain text visual features of N texts;

[0097] A set of text visual features is established:

[0098]

[0099] V = {V1, V2, …, Vn} (2) ocr denotes a set of text visual features, denotes the first text visual feature, denotes the n th text visual feature, denotes the Nth text visual feature.

[0100] S4: Visual features are processed by linear transformation, layer standardization regularization, and ReLU activation function to obtain object representation vectors;

[0101]

[0102] wherein: denotes the mth object representation vector, σ denotes the ReLU activation function, LN denotes the layer normalization regularization, W obj denotes a linear transformation on each object representation vector;

[0103] A set of object visual features is established:

[0104]

[0105] wherein: X obj denotes a set of object representation vectors, denotes the mth object representation vector, denotes the Mth object representation vector.

[0106] S5: using the FastText tool to extract features of the image, obtaining the character vector about the nth character; using the Phoc method to process the text characters at varying pyramid level scales, obtaining the text vector; through linear transformation, layer standard regularization LN and ReLU activation function processing, obtaining the character representation vector;

[0107] The character representation vector calculation formula is:

[0108]

[0109] wherein: denotes the nth character representation vector, W m denotes a linear dimensional transformation on the overall vector, denotes the text vector, denotes the semantic text vector, W bb denotes a linear dimensional transformation on the coordinates corresponding to the character region;

[0110] Further, the character visual feature, the text vector and the text semantic vector are spliced into an overall vector in the second dimension.

[0111] A set of character representation vectors is established:

[0112]

[0113] wherein: X ocr denotes a set of nth character representation vectors, denotes the 1st character representation vector, denotes the Nth character representation vector.

[0114] Further, step one, there are objects (such as people, cars, books) and possible text identification information (such as road signs, advertising slogans, book titles, etc.) in the image. In order to enable the model to describe the objects and text in the image, it is necessary to extract these information from the image and convert it into feature representation:

[0115] (1) Use the pre-trained Faster-RCNN network to extract the region features of the image I, and get the bounding box coordinates of M objects And the corresponding M visual features Wherein, the bounding box can also be called the region frame. The visual features are processed by linear transformation, layer normalization regularization LN and ReLU activation function σ to obtain the representation of the image object:

[0116]

[0117] (2) Get the bounding box coordinates of N texts in the image I through the optical character recognition tool Use the pre-trained ResNet101 network to extract features of the image I in the bounding box coordinates B ocr Region, get the corresponding N visual feature sets For each text, use the FastText tool to obtain the semantic text vector Use the Phoc method to process the text characters at varying pyramid level scales to obtain the text vector Use the method similar to (1) to process through linear transformation, layer normalization regularization LN and ReLU activation function to obtain the text representation feature.

[0118]

[0119] Through the above steps, the feature representation set of the object in the image is established And the feature representation set of the text

[0120] Optionally, the method for establishing the graph relationship comprises:

[0121] The text representation vector is taken as the vertex V; each of the N texts is edge-processed to obtain the edge set E and the main target in the object visual feature; the graph relationship G(V, E) is established through the vertex V and the edge set E; wherein, the established graph relationship is a sparse graph relationship;

[0122] The method for edge processing each of the N texts comprises:

[0123] If the region completely contained in the region of the object that is, the rectangular region j completely contains the rectangular region i, it is considered that the text i is attached to the corresponding object j, and an edge e is established between the text i and the object j ij ∈E.

[0124] If the region of the text i is simultaneously contained in multiple object regions j1,..., j T , select the object region j t with the smallest area from the multiple object regions as the adjacent node of the text i, and establish an edge T between the text i and the object j t .

[0125] Further, step two, the characteristic vectors of the object and the text obtained in step one and the bounding boxes B obj and B ocr corresponding to the object and the text are taken as vertices V in the graph relationship G(V, E), and the steps of establishing the graph relationship are as follows:

[0126] (1) If the region coordinates of the text i are completely contained in the region coordinates of the object, that is, the rectangular region j completely contains the rectangular region i, it is considered that the text i is attached to the corresponding object j, and an edge e is established between the text i and the object j ij ∈E.

[0127] Further, for the region completely contained in the region, the region is embodied as a region box, and the region box is described using two vertex coordinates (x1, y1) and (x2, y2) of the upper left corner and the lower right corner. Therefore, the vertex coordinates of the two boxes determine whether the two boxes are in a containing relationship.

[0128] (2) If the region of the text i is simultaneously contained in multiple object regions j1,..., j T , select the object region j t with the smallest area from the multiple object regions as the adjacent node of the text i, and establish an edge T between the text i and the object j t .

[0129] (3) For the N texts in step one, perform steps (1) and (2) to find the main target in the object visual feature, and simultaneously establish a relatively sparse graph G(V, E).

[0130] Optionally, the method for processing the relationship graph by using the graph convolution network model comprises the following steps:

[0131] The node feature inputted by the graph convolution network is composed of the object representation vector and the character representation vector.

[0132] The node feature is calculated by the MR-GCN of one layer of the graph convolution network to obtain the enhanced feature.

[0133] Further, the node feature inputted by the graph convolution network is composed of the object representation vector and the character representation vector. According to the existing or established node adjacent relationship R in the graph relationship G(V, E), the MR-GCN graph convolution network makes each node feature obtain the feature information of the adjacent nodes in the relationship R. After linear transformation, the average value of these adjacent nodes is calculated, and the original node after linear transformation is added, and finally the final result is obtained after processing by the activation function.

[0134] The above processing can be performed l times according to the number of layers l of the graph convolution network. In actual cases, 1-3 layers are mostly used, and only one layer is used in this embodiment of the application. The input of the graph network is Through network linear transformation, more abundant adjacent relationship node information under the relationship R is obtained, so that the enhanced feature is realized

[0135] Enhanced feature formula:

[0136]

[0137] In the formula, X represents the enhanced feature set of the object and the character.

[0138] Further, the graph convolution network can have multiple layers, and the superscript l represents that the vector feature belongs to the calculation result of the lth layer of the graph convolution network, and l+1 represents the result after the calculation result of the lth layer of the graph convolution network is calculated by the (l+1)th layer of the network. When the feature superscript l = 0 or is not marked with the superscript l, it represents the feature before enhancement. In the embodiment of the application, the convolution network used in step one is one layer, so l+1 represents the feature after enhancement by the graph convolution network, wherein the first M consecutive objects + the last N consecutive characters are numbered, so the whole is M+N objects, and the numbering starts from 1 and increases until M+N, represents the enhanced feature of the object, represents the enhanced feature of the character, and R (M+N)×1000 The overall feature geometry is M+N 1000-dimensional vectors, that is, the overall dimension of the feature is 2-dimensional (M+N) x 1000.

[0139] Further, the calculation process of the MR-GCN is as follows:

[0140]

[0141] where σ is the ReLU activation function, the whole in the bracket can be divided into two parts, the former is the node feature , the latter is the information of the adjacent node of the node ; represents each feature before enhancement, represents the feature after the enhancement of the lth layer of the graph convolution network, for the lth layer network: the original information part, represents the linear transformation of the original feature to preserve the original information; the adjacent node information part, represents the linear transformation of all adjacent nodes of the feature, represents the number of all adjacent nodes of node i in the relationship , the average value of the feature is calculated as the denominator (when the number is 0, this part is no longer calculated to avoid the case that the denominator divisor is 0). There can be more than one graph relationship between nodes, represents one of the graph relationships, here R represents the relationship between the text feature and the object feature, and the relationship between the object feature and the text feature, the relationship between the text feature and the object feature and the relationship between the object feature and the text feature are two different direction relationships. In essence, the graph convolution network uses the adjacent nodes of each node as additional information according to the adjacent relationship in the graph relationship G(V, E), so that the original node has more relationship information.

[0142] The graph relationship G(V, E) of the sparse graph records all the edges between the vertices, and the vertex-edge relationship can be represented by a matrix, where 0 represents no relationship and 1 represents edge relationship. The graph convolution network calculates the enhanced result of the feature vector corresponding to the vertex according to the relationship between the points and edges in the graph relationship G(V, E). The specific calculation process is as follows:

[0143] 1) For a set of features X = {x1, x2, x3...x i ,...,x N} input into the network, each node is x i . According to the vertex V in the graph relationship G(V, E), it can be obtained which vertex in the graph corresponds to x i ; according to the relationship in the edge E, if e {ij} = 1, it can be obtained that the feature i and the feature j are adjacent and related features.

[0144] 2) For x i , all nodes {x j} adjacent to it are linearly transformed by the parameter W r and averaged.

[0145] 3) The graph relationship G(V, E) can be represented by a 0-1 matrix Wr indicates a relationship, and the value at the i-th row and j-th column of the matrix is 1, indicating that there is a correlation between the i-th feature and the j-th feature. Therefore indicates that a linear transformation is performed on all adjacent nodes of each node of the feature.

[0146] Further, step three, using the method of graph convolution to process the relationship graph to obtain a new image representation. The node feature input into the graph convolution network is the representation vector of the object and the text in step 1 The graph relationship is the relationship G established in step two. After one layer of graph convolution network MR-GCN calculation, the enhanced feature

[0147] Optionally, the decoding model adopts multiple loop iterations until the end of sentence is predicted.

[0148] After the multiple loop iterations end, the sentence describing the content of the picture is obtained according to the index and the word-number correspondence table.

[0149] Each of the multiple loop iterations includes the following processes:

[0150] The decoding vector obtained in the last prediction, the word vector obtained in the last prediction, the average value of the enhanced feature input into the decoding model, and the weighted feature obtained in the last prediction are spliced to obtain the decoding vector of this prediction, wherein the weighted feature is obtained through attention mechanism calculation.

[0151] The weighted decoding vector of this prediction is obtained after adjusting the proportion of the text visual feature in the decoding vector through the gating mechanism, and the prediction score of this prediction is obtained after the weighted decoding vector of this prediction is calculated through the linear classification network.

[0152] The index of the maximum prediction score in the prediction score is obtained.

[0153] Optionally, in the t-th loop iteration, the formula for splicing the decoding vector obtained in the last prediction, the word vector obtained in the last prediction, the average value of the enhanced feature input into the decoding model, and the weighted feature obtained in the last prediction of this time includes:

[0154] The vector splicing calculation formula is:

[0155]

[0156] In the formula: x t indicates the input decoding vector of this time, h t indicates the decoding vector of the t-th prediction, y t-1 indicates the word vector of the t-1th prediction, an average value of the enhanced features of the input decoding model, a weighted feature of the t-1th time;

[0157] an average value of the enhanced features of the input decoding model,

[0158]

[0159] wherein: an average value of the enhanced features, i represents a sequence index of the enhanced features, the i th enhanced feature in the total number M+N of enhanced features;

[0160] h t ,c t = LSTM(c t-1 ,h t-1 ,x t )

[0161] wherein: c t ,c t-1 represent the current time and the previous time state cell vectors respectively, h t-1 represents the previous decoding vector;

[0162] Further, h and c are both inputs and outputs of the LSTM network, and are updated with the iteration of the time, but only h is used as the decoding vector, and c is only used for the update calculation in the LSTM internal network and is not used elsewhere.

[0163] an attention mechanism calculation formula:

[0164]

[0165] wherein: a weighted feature of the t th step, X represents a set of enhanced features.

[0166] Optionally, in the t th iteration of the loop, the method for obtaining the weighted decoding vector and the prediction score comprises:

[0167] using a logistic regression model to obtain a gating weight and a word weight corresponding to a word visual feature;

[0168] using a decoding vector weighting formula to obtain the weighted decoding vector, wherein the decoding vector weighting formula adjusts the proportion of the word visual feature in the decoding vector, and captures the decoding vector to be retained through the gating weight;

[0169] obtaining a prediction score of the current prediction by performing linear classification on the weighted decoding vector through a linear classification network;

[0170] The logistic regression model calculation formula is specifically as follows:

[0171]

[0172] wherein: a1 represents a gating weight, a N+1 represents a character weight corresponding to the Nth character visual feature, represents transforming the intermediate process vector into a row vector, W vis represents performing linear transformation on the decoding vector, h t represents the decoding vector of this time, W txt represents performing linear transformation on the character representation vector;

[0173] wherein, transforming the attention matrix into an N+1-dimensional row vector, W vis represents performing linear dimension transformation on the decoding vector, W txt represents performing linear dimension transformation on the result of splicing the average of the enhanced features and all character features, and adding the result of the dimension transformation to the decoding vector to obtain the attention matrix of the decoding vector on the character features.

[0174] The decoding vector weighting formula is specifically as follows:

[0175]

[0176] wherein: represents the weighted decoding vector of this time, a1 represents a gating weight, a k represents a character weight corresponding to the k-1th character visual feature, represents the k-1th character visual feature;

[0177] The linear classification network calculation formula is specifically as follows:

[0178]

[0179] wherein: s t represents a prediction score, W fc represents performing linear dimension transformation on the weighted decoding vector, b fc represents an offset term of the linear transformation.

[0180] In the tth iteration of the loop iteration, the formula for obtaining the index of the maximum prediction score in the prediction score is specifically as follows:

[0181] index=argmax(s t )

[0182] wherein: index represents an index, s t represents a prediction score.

[0183] Further, the index corresponding word is the prediction result of this time.

[0184] Step four, enhanced features The input model decoding part. The decoding part consists of long-short term memory (LSTM) with attention mechanism, and the decoding part adopts a cyclic iteration method.

[0185] (1) In the t step, the previously predicted decoding vector h t-1 , the word vector y t-1 predicted in the previous step, the average value of the input feature The weighted feature after attention mechanism calculation The above four vectors are spliced. Since the predicted sentence in this task has words from the image (such as the identification words of a billboard in the image), and the word-number table established by the model itself does not have these words. Therefore, when the word w t-1 comes from a fixed word table, y t-1 is expressed as a vector converted by the embedding network corresponding to the number; when the word w t-1 comes from the text in the image, y t-1 is expressed as a corresponding text representation vector The vector splicing method and the attention mechanism calculation process are as follows:

[0186]

[0187] h t ,c t = LSTM (h t-1 ,x t )

[0188]

[0189] (2) The model is decoded by LSTM to obtain the decoding vector h t . When the predicted word is the scene text in the image, the decoding vector should have a high correlation with the visual feature of the text. Therefore, the model adopts a gating mechanism to adjust the proportion of the visual feature of the text in the decoding vector, so as to pay attention to different aspects of information. Through the following calculation steps, the gating weight α1 represents the reserved decoding vector, and the remaining weight represents the proportion of the visual feature of the text. W vis ,W txt are all learning parameters, so the weight is dynamically adjusted in each step of prediction.

[0190]

[0191]

[0192] at last The current predicted score s is obtained through the linear classification network fc. t .

[0193]

[0194] (3) In the predicted scores, obtain the index of the highest score: index = argmax(s) t The word at the index position is used as the prediction result for the current step. Simultaneously, the vector y corresponding to the predicted word is... t The predicted decoding vector h t As input for the next prediction, proceed to step (1). This process is repeated until the end marker of the predicted sentence is reached, at which point prediction stops. After prediction, the sentence describing the image content is obtained based on the index and word-number lookup table.

[0195] Optionally, training methods for the decoding network may also be included:

[0196] The probability value of the predicted score belonging to the category is obtained by performing probability transformation using the logistic regression function in the t-th iteration.

[0197] Based on probability values, the parameters of the decoding model are trained and optimized using a loss function.

[0198] Loss function:

[0199]

[0200] In the formula: L represents the loss function, T represents the maximum length of the predicted descriptive sentence, t represents the number of iterations, C represents the number of categories, idx represents the number of the idx-th category, and y represents the score of the true value of the data. This represents the probability value of the idx-th category in the t-th iteration;

[0201]

[0202] In the formula: Let represent the probability value of the first category in the t-th iteration. Let s represent the probability value of the C-th category in the t-th iteration. t This represents the predicted score.

[0203] Optionally, the process of training and optimizing the decoding model parameters using the loss function also includes: determining whether the loss function has converged; if the result is yes, obtaining the current decoding network model; if the result is no, re-feeding the enhanced features into the model decoding network.

[0204] Further, step five, using the cross-entropy function as follows, the model parameters are trained and optimized, where C is the number of classifications, that is, the number of all commonly used words in the word table and the total number of categories of the scene text detected in the image, and T is the maximum length of the predicted description sentence.

[0205]

[0206]

[0207] Where y represents the fraction of the true value of the data itself, and the specific form is one-hot encoding [0, 0,..., 1, 0, 0,..., 0], that is, the position corresponding to the word number is 1, and the rest are 0. t The score predicted by the model is converted into probability using the softmax function.

[0208] The application will be further described below in conjunction with specific embodiments:

[0209] The given image data set TextCaps contains about 28,000 pictures. Each picture has 5 subtitles, and the data is divided into training, validation and test sets. The data is trained and validated in batches of 50.

[0210] The object and object region coordinates of the picture are extracted by using the Faster-RCNN target detection model based on the convolutional neural network. Given a picture, the detection model detects 100 object regions from the picture, and obtains object visual feature dimension 2048 and 100 region coordinates with dimension 4.

[0211] We use the optical character recognition tool Rosetta to extract the character text in the picture, and obtain up to 50 text characters and 50 region coordinates. The pre-trained ResNet101 network is used to extract visual features of the 50 regions in the picture, and obtain object visual feature dimension 2048 and 100 region coordinates with dimension 4.

[0212] In the data construction stage, the object visual features and the text visual features are linearly changed and standardized to become 1024-dimensional vectors.

[0213] A word table is established for the words in the image caption. First, the symbols for the beginning, end and length padding of the sentence in the TexCaps data set are selected, and all the words with an occurrence frequency greater than 10 are selected to establish a corresponding relationship in order to obtain 0-6735 numbers as a fixed word table. Secondly, there are different scene texts in each image, and the results of the optical character recognition tool Rosetta are numbered after the fixed word table. For each picture, there are up to 50 scene text numbers 6736-6785.

[0214] The graph convolution model module adopts a multi-relation graph convolution network. The node edge relation in the graph relation G(V, E) is divided into object-scene text and scene text-object. In the convolution stage of the graph, a graph convolution layer with a dimension of 768 is used.

[0215] In the training process, the maximum length T=30 of the cyclic iteration prediction sentence is set, the dropout method is used to prevent overfitting, and the Adam optimizer is used. The model is trained for about 30 epochs, and the gradient descent optimization cross-entropy loss function is used.

[0216] In the test stage, the beam search is set to 5 to generate subtitles, and common indicators BLEU, METEOR, CIDEr and ROUGE-L are used to evaluate the overall subtitle generation model. All evaluation indicators are obtained by calculating the indicators between the generated results and the true values of the TexCaps dataset.

[0217] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0218] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text subtitle generation method based on a primary target and scene orientation, characterized in that, Specifically, the following steps are included: The image is preprocessed to obtain sample data, which includes the object region bounding coordinates, object visual features, and object representation vectors for each of the M objects. The sample data also includes the text region bounding coordinates, text visual features, and text representation vectors for each of the N texts. Extract the coordinates of the body region bounding box and the text region bounding box, and establish a graph relationship between the two; Graph convolutional network models enhance object representation vectors and text representation vectors based on graph relationships to obtain enhanced features; The enhanced features are sent to the decoding model for decoding to obtain sentences that describe the content of the image; The decoding model uses multiple loop iterations until the end marker of the sentence is predicted; After multiple iterations, sentences describing the content of the image are obtained based on the index and word-number lookup table; Each of the multiple iterations includes the following process: The decoding vector obtained from the previous prediction, the word vector obtained from the previous prediction, the average value of the enhanced features of the input decoding model, and the weighted features obtained from the previous prediction are concatenated to obtain the current input decoding vector. The weighted features are calculated through an attention mechanism. After adjusting the weight of the visual features of the characters in the decoding vector using a gating mechanism, the weighted decoding vector for this prediction is obtained. The weighted decoding vector for this prediction is then processed by a linear classification network to obtain the prediction score for this prediction. Get the index of the highest predicted score from the predicted scores; In the t-th iteration of the loop, the formula for concatenating the decoded vector obtained from the previous prediction, the word vector obtained from the previous prediction, the average value of the enhanced features input to the decoding model, and the weighted features obtained from the previous prediction includes: Vector concatenation calculation formula: In the formula: x t h represents the decoded vector of this input. t Indicates the first t The decoded vector of the next prediction, y t-1 This represents the word vector predicted in the (t-1)th prediction. This represents the average value of the augmented features of the input decoding model. This represents the weighted feature at the (t-1)th iteration; The formula for calculating the average value of the augmented features of the input decoding model is as follows: In the formula: This represents the average value of the enhanced features, where i represents the index of the enhanced feature. Let i represent the i-th enhanced feature among the total number of enhanced features M+N; h t ,c t =LSTM(h t-1 ,c t-1 ,x t ) In the formula: c t h represents the current cell state at step t. t-1 c represents the previous decoding vector. t-1 Indicates the previous cell state; The formula for calculating attention mechanism: In the formula: Let X represent the weighted features at step t, and let X represent the enhanced feature set.

2. A text subtitle generation method based on a primary target and scene orientation according to claim 1, characterized in that, The preprocessing steps include: The Faster-RCNN network is used to extract region features from the image, obtaining the object region bounding box coordinates and object visual features for M objects. The object visual features are then processed by linear transformation, layer normalization regularization, and ReLU activation function to obtain object representation vectors. Image features are extracted using optical character recognition tools to obtain the coordinates of text region boxes for N characters. A pre-trained ResNet101 network is used to extract features from the text regions, resulting in the corresponding N character visual features. For each character, the FastText tool is used to obtain a semantic text vector. The Phoc method is used to process the text characters at varying pyramid hierarchical scales to obtain a text vector. The text vector is then processed through linear transformation, layer normalization (LN), and ReLU activation functions to obtain the character representation vector.

3. A text subtitle generation method based on a primary target and scene orientation according to claim 1, characterized in that, Methods for establishing graph relationships include: The text representation vector is used as vertex V; edges are constructed for each of the N texts to obtain the edge set E and the main target in the visual features of the object; a graph relation G(V,E) is established through vertex V and edge set E; the established graph relation is a sparse graph relation. The specific methods for constructing edges for each of the N characters include: If the text area is completely contained within the object area, then the text is attached to the corresponding object, and an edge e is established between the text and the object. ij ∈E; If a text region is contained within multiple object regions, select the object region with the smallest area as the adjacent node of the text, and establish an edge between the text and the object corresponding to the smallest object region. .

4. A text subtitle generation method based on a primary target and scene orientation according to claim 1, characterized in that, Methods for processing relational graphs using graph convolutional network models include: The node features input to the graph convolutional network consist of object representation vectors and text representation vectors; The node features are processed by a single-layer graph convolutional network (MR-GCN) to obtain enhanced features.

5. A text subtitle generation method based on a primary target and scene orientation according to claim 1, characterized in that, In the t-th iteration of the loop, the methods for obtaining the weighted decoded vector and the predicted score include: Use a logistic regression model to obtain the gating weights and the text weights corresponding to the visual features of the text; A weighted decoding vector is obtained by using a decoding vector weighting formula. The decoding vector weighting formula adjusts the proportion of visual features of characters in the decoding vector and captures the decoding vector that needs to be retained through gating weights. The prediction score for this prediction is obtained by performing linear classification on the weighted decoded vector using a linear classification network. The specific formula for calculating the logistic regression model is as follows: In the formula: α1 represents the gating weight, α N+1 This represents the text weight corresponding to the visual feature of the Nth text. W represents the transformation of the intermediate process vector into a row vector. vis This indicates that the decoded vector will undergo a linear transformation, h t Let W represent the decoding vector for this operation. txt This indicates a linear transformation of the text representation vector; The specific formula for weighting the decoded vectors is as follows: In the formula: This represents the weighted decoding vector for this operation, where α1 represents the gating weight, and α... k This represents the text weight corresponding to the (k-1)th text visual feature. This represents the visual feature of the (k-1)th character; The specific formula for calculating linear classification networks is as follows: In the formula: s t W represents the predicted score. fc This represents a linear dimensionality transformation of the weighted decoding vector, b fc This represents the offset term of the linear transformation.

6. A text subtitle generation method based on a primary target and scene orientation according to claim 1, characterized in that, The formula for obtaining the index of the maximum predicted score in the predicted scores during the t-th iteration of the loop is as follows: index=argmax(s t ) In the formula: index represents the index, s t This indicates the predicted score.

7. A text subtitle generation method based on a primary target and scene orientation according to claim 1, characterized in that, It also includes training methods for decoding networks: The probability value of the predicted score belonging to the category is obtained by performing probability transformation using the logistic regression function in the t-th iteration. Based on probability values, the parameters of the decoding model are trained and optimized using a loss function. Loss function: In the formula: L represents the loss function, T represents the maximum length of the predicted descriptive sentence, t represents the number of iterations, C represents the number of categories, idx represents the number of the idx-th category, and y represents the score of the true value of the data. This represents the probability value of the idx-th category in the t-th iteration; In the formula: Let represent the probability value of the first category in the t-th iteration. Let s represent the probability value of the C-th category in the t-th iteration. t This indicates the predicted score.

8. A text subtitle generation method based on a primary target and scene orientation according to claim 7, characterized in that, The process of training and optimizing the decoding model parameters using the loss function also includes: determining whether the loss function has converged; if the result is yes, the current decoding network model is obtained; if the result is no, the enhanced features are re-input into the model decoding network.

Images