Method for identifying text appearing in video and related hardware
By segmenting and fusing the features of image frames in a video, and using the Transformer model to perform local and global feature fusion, the problem of low accuracy in image text recognition under extreme scenarios is solved, achieving higher recognition accuracy and robustness.
Patent Information
- Application Number
- CN202510958545.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2025-11-18
AI Technical Summary
Existing image text recognition technologies have low accuracy in extreme and special scenarios, especially when the text to be recognized is obscured by objects or is incomplete in shape, making it difficult to recognize effectively.
By identifying the target text image region in the video, dividing it into multiple sub-image regions, extracting image features from each frame, and using the Transformer model for feature fusion, including local and global feature fusion, the text recognition process is optimized.
It improves the accuracy of text recognition in videos by utilizing high-quality features and discarding low-quality features through cross-frame analysis, thus achieving better text recognition results and robustness.
Smart Images
Figure CN120976904A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image text recognition, and in particular to a method for recognizing text appearing in a video and related hardware. BACKGROUND
[0002] Image text recognition technology is a technology for converting text in an image into editable and searchable text, and is widely used in identification of certificates and bills, identification of equipment numbers, identification of license plates, etc. The existing image text recognition technology has a general recognition accuracy in some extreme special scenarios (e.g., the image to be recognized has complex content, and the shape of the text to be recognized in the image to be recognized is blocked by an object, resulting in an incomplete shape), and needs to be further improved. SUMMARY
[0003] The embodiments of the present application provide a method for recognizing text appearing in a video and related hardware, to improve the recognition accuracy of text appearing in a video frame.
[0004] The embodiments of the present application provide a method for recognizing text appearing in a video, comprising:
[0005] determining a target text image region to be recognized in a plurality of target image frames of a video to be recognized, and dividing any target text image region into a plurality of target sub-image regions; wherein the target text image region is an image region where target text content is located;
[0006] extracting at least one target image feature of the target text image region in each target image frame;
[0007] grouping target image features of target sub-image regions having the same relative position with respect to the target text image region in different target image frames into a target feature sequence;
[0008] fusing target image features in any target feature sequence to obtain a first fused feature;
[0009] fusing each first fused feature to obtain a second fused feature;
[0010] determining target text based on the second fused feature.
[0011] Optionally, the fusing of target image features in any target feature sequence to obtain a first fused feature comprises:
[0012] inputting any target feature sequence into a first Transformer encoder to obtain a first fused feature corresponding to the target feature sequence output by the first Transformer encoder.
[0013] The second fusion features are obtained by fusing the first fusion features.
[0014] The first fusion features are input into a Transformer second encoder in sequence to obtain a second fusion feature sequence output by the Transformer second encoder.
[0015] The target text is determined based on the second fusion features.
[0016] The second fusion feature sequence is input into a Transformer decoder to obtain a target text output by the Transformer decoder.
[0017] Further optionally, the target image features of the target sub-image regions with the same relative position with respect to the target text image region in different target image frames are grouped into a target feature sequence.
[0018] The target image features of the target sub-image regions with the same relative position with respect to the target text image region in different target image frames are grouped into a target feature sequence.
[0019] The target feature sequence is input into a Transformer first encoder to obtain first fusion features corresponding to the target feature sequence output by the Transformer first encoder.
[0020] The target feature sequence is input into a Transformer first encoder to obtain a final output feature sequence corresponding to the target feature sequence output by the Transformer first encoder, and a feature at a starting position of the final output feature sequence is taken as the first fusion features.
[0021] Further optionally, the method further comprises:
[0022] For any target image frame, a recommendation score of a target text image region of the target image frame is determined according to attention weights of the target sub-image regions calculated by a CLS token of a last encoding layer of the Transformer first encoder.
[0023] A target text image region with the maximum recommendation score is determined as a recommended target text image region, and / or a target image frame corresponding to the target text image region with the maximum recommendation score is determined as a recommended target image frame.
[0024] Further optionally, the recommendation score Attn(t) satisfies the following relationship:
[0025]
[0026] t is the serial number of the target image frame, n is the serial number of the target sub-image region in the target image, N is the total number of target sub-image regions in a target image frame, m is the serial number of the attention head, M is the total number of attention heads, is the query vector corresponding to the mth attention head of the last encoding layer of the first encoder of the Transformer for the CLS token of the nth final input feature sequence, is the key vector corresponding to the mth self-attention mechanism for the tth frame corresponding feature of the nth final input feature sequence.
[0027] For any target feature sequence, the final input feature sequence is the input feature sequence of the last encoding layer of the first encoder of the Transformer in the process of inputting the target feature sequence into the first encoder of the Transformer for multi-level encoding layer by layer processing to obtain the first fusion feature.
[0028] Optionally, the method further comprises:
[0029] performing text target detection on the plurality of target image frames in the video to be recognized to determine text image regions;
[0030] classifying the text image regions to determine text image regions belonging to the same text content, wherein the text image regions corresponding to the target text content are determined as the target text image regions.
[0031] Based on the same inventive concept, the embodiments of the present application also provide a device for recognizing text appearing in a video, comprising:
[0032] a target detection module configured to determine target text image regions to be recognized in a plurality of target image frames of a video to be recognized, and for any target text image region, divide the target text image region into a plurality of target sub-image regions; wherein the target text image region is an image region where the target text content is located;
[0033] a feature extraction module configured to extract at least one target image feature of the target text image region in each target image frame; and group the target image features of the target sub-image regions having the same relative position with respect to the target text image region in different target image frames into a target feature sequence;
[0034] The local feature fusion module is configured to fuse each target image feature in any target feature sequence to obtain a first fused feature;
[0035] The global feature fusion module is configured to fuse each first fused feature to obtain a second fused feature.
[0036] The text recognition module is configured to determine a target text based on the second fused feature.
[0037] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, comprising a processor and a memory for storing instructions executable by the processor;
[0038] The processor is configured to execute the instructions to implement the method for recognizing text appearing in a video.
[0039] Based on the same inventive concept, the embodiments of the present application also provide a computer-readable storage medium storing computer program codes, which, when executed on a computer, cause the computer to execute the method for recognizing text appearing in a video.
[0040] Based on the same inventive concept, the embodiments of the present application also provide a computer program product, comprising computer program codes, which, when executed on a computer, cause the computer to execute the method for recognizing text appearing in a video.
[0041] The present application has the following advantages:
[0042] The method for recognizing text appearing in a video and related hardware provided by the embodiments of the present application first fuse image features of different image frames corresponding to target text content according to positions, then fuse image features according to all positions after the fusion of image features according to positions, and finally recognize text based on fused features. The method can realize cross-image frame analysis of features in the process of fusing image features according to positions, fully utilize the advantages of image features of positions with better recognition effects in each image frame for text recognition, discard the disadvantages of image features of positions with worse recognition effects in each image frame for text recognition, realize the advantages of each image frame in the process of text recognition, and improve the text recognition effect and the robustness of text recognition compared with the prior art. BRIEF DESCRIPTION OF DRAWINGS
[0043] Figure 1 One of flowcharts of the method for recognizing text appearing in a video provided by the embodiments of the present application;
[0044] Figure 2Flowchart two of the method for recognizing text appearing in a video provided by the embodiment of the present application;
[0045] Figure 3 Flowchart three of the method for recognizing text appearing in a video provided by the embodiment of the present application;
[0046] Figure 4 Flowchart four of the method for recognizing text appearing in a video provided by the embodiment of the present application;
[0047] Figure 5 Partial flowchart of the method for recognizing text appearing in a video provided by the embodiment of the present application;
[0048] Figure 6 Schematic diagram of the data processing process in the method for recognizing text appearing in a video provided by the embodiment of the present application;
[0049] Figure 7 Schematic diagram of the attention parameter calculation of the CLS token in the embodiment of the present application;
[0050] Figure 8 Schematic diagram of the structure of the device for recognizing text in a video provided by the embodiment of the present application;
[0051] Figure 9 Schematic diagram of the structure of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the above objectives, features and advantages of the present application more apparent, comprehensible and easier to understand, the present application will be further described below with reference to the drawings and embodiments. However, the example embodiments can be implemented in various forms, and should not be understood as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided so that the present application is more comprehensive and complete, and the ideas of the example embodiments are fully conveyed to those skilled in the art. The same reference numerals in the drawings represent the same or similar structures, and thus repeated description thereof will be omitted. The words expressing position and direction described in the present application are described with reference to the drawings, but can be changed as needed, and the changes made are included in the protection scope of the present application. The drawings of the present application are only used to illustrate the relative positional relationship and do not represent the true proportions.
[0053] It is to be understood that the embodiments described herein are merely exemplary of the application made and that numerous other embodiments can be implemented which are still within the scope and spirit of the application as described. Accordingly, the particular embodiments described in detail herein are illustrative only and not restrictive.
[0054] The method for recognizing text appearing in a video and related hardware provided by the embodiments of the present application are specifically described below with reference to the accompanying drawings.
[0055] The embodiments of the present application provide a method for recognizing text appearing in a video, as shown in the following formula (1), comprising the following steps. Figure 1
[0056] S111, determining a target text image region to be recognized in a plurality of target image frames of a video to be recognized.
[0057] The target text image region is an image region where target text content is located, and the target image frame is an image frame where the target text content is located in the video to be recognized. For example, the method provided by the embodiments of the present application is applied to an application scenario of recognizing a license plate number of a vehicle appearing in a traffic monitoring video. The license plate of a target vehicle appears in a plurality of image frames in the video to be recognized. The image frame where the license plate of the target vehicle appears is the target image frame, and the image region corresponding to the license plate of the target vehicle in each target image frame is the target text image region.
[0058] As an optional implementation, the target text image region can be determined manually by manually marking the image frames in the video to be recognized. For example, a text recognition frame control is provided on a video playing interface, and a user uses the text recognition frame control to mark an image region where target text content to be recognized is located on a picture of an image frame of the video to be recognized, so as to determine the target text image region to be recognized.
[0059] As another optional implementation, the text image region is determined by automatically performing text target detection on a plurality of image frames in the video to be recognized, and the target text image region corresponding to the target text content is determined based on the text image region of each image frame.
[0060] In specific implementation processes, the You Only Look Once (YOLO) algorithm, the Differentiable Binarization Network (DBNet) algorithm, the Faster Region-based Convolutional Neural Network (Faster R-CNN) algorithm, or the like can be used to detect text targets in multiple image frames in the video to be recognized, and determine text image regions.
[0061] In specific implementation processes, if only one text content (i.e., target text content) appears in the video to be recognized, each text image region obtained by detecting text targets in each image frame in the video to be recognized can be directly determined as a target text image region; if multiple text contents appear in the video to be recognized, after detecting text targets in multiple image frames in the video to be recognized to determine text image regions, each text image region needs to be classified to determine text image regions belonging to the same text content, and the text image region corresponding to the target text content is determined as the target text image region. The classification of each text image region to determine text image regions belonging to the same text content can be implemented by using an Object Tracking algorithm, a feature matching algorithm, or the like. For example, a Deep Learning-based Simple Online and Realtime Tracking (Deep SORT) algorithm, a Kalman filter algorithm, or the like can be used to classify each text image region to determine text image regions belonging to the same text content, and further determine the target text content and the target text image region corresponding to the target text content in each target image frame.
[0062] S112, for any target text image region, the target text image region is divided into multiple target sub-image regions.
[0063] In specific implementation processes, the target text image region can be divided into multiple target sub-image regions according to a preset number or a preset size. For example, for a target image frame t, the target text image region can be divided into wh target sub-image regions according to a manner of dividing the target text image region into w regions in a horizontal direction and h regions in a vertical direction, and the target sub-image regions are respectively denoted as target sub-image regions (1, 1, t), (1, 2, t), …, (w, h, t).
[0064] S120, at least one target image feature of the target text image region is extracted in each target image frame.
[0065] In the implementation process, a machine learning algorithm such as a Convolutional Neural Network (CNN) can be used to extract features from the target text image region. The features extracted from the target text image region can be explicit features (such as pixel color, resolution, pixel density, pixel color difference, etc.) of a feature category pre-set according to expert experience, or implicit features learned by implementing a feature extraction machine learning algorithm based on a machine learning training process.
[0066] S130, each target image feature of the target sub-image region with the same relative position to the target text image region in different target image frames is combined to form a target feature sequence.
[0067] Each target image feature of the target sub-image region corresponding to the same relative position to the target text image region in different target image frames is combined to form a target feature sequence, and then a target feature sequence corresponding to each relative position to the target text image region is obtained. For example, for the relative position (i, j), the target sub-image regions (i, j, 1), (i, j, 2), …, (i, j, T) corresponding to the target image frame 1, the target image frame 2, …, the target image frame T, the target image features The target feature sequence S i,j is combined to form a target feature sequence S i,j The target image features For example, for any target text image region, the target text image region is divided into hw target sub-image regions in the manner of dividing w regions horizontally and h regions vertically, respectively denoted as relative positions (1, 1), (1, 2), …, (h, w). After the above steps, hw target feature sequences {S i,j |i = 1, 2, …, w; j = 1, 2, …, h} are obtained.
[0068] S140, for any target feature sequence, the target image features in the target feature sequence are fused to obtain a first fused feature.
[0069] In the implementation process, for the same relative position, the influence of each target image feature corresponding to the relative position in different target image frames on the text recognition effect can be considered, and in the target features corresponding to the relative position, the target image feature that is more conducive to text recognition has a greater influence on the value of the first fusion feature corresponding to the target feature sequence. Thus, the first fusion feature obtained by fusing each target image feature of the target feature sequence can focus more on the target image feature that is conducive to text recognition.
[0070] S150, fusing each first fusion feature to obtain a second fusion feature.
[0071] S160, determining the target text based on the second fusion feature.
[0072] In the step S140, the first fusion feature corresponding to each relative position is obtained by fusing the target image features of different target image frames at the same relative position. In the step S150, the second fusion feature is obtained by further fusing the first fusion features corresponding to each relative position, which is the fusion feature of the target text image region in all target image frames as a whole. Then, the text recognition is performed based on the second fusion feature. Thus, the present application first fuses the image features of different image frames corresponding to the target text content by position, then performs the image feature fusion in all positions after the image feature fusion by position is completed, and finally performs the text recognition based on the fusion feature. In the process of fusing the image features by position, the feature cross-image frame analysis is realized, the advantages of the image features of the positions with better recognition effect in each image frame on the text recognition are fully utilized, the disadvantages of the image features of the positions with poor recognition effect in each image frame on the text recognition are abandoned, the strengths of each image frame in the text recognition process are taken, and the text recognition effect and the robustness of the text recognition are better than those of the prior art. For example, for the case that the pictures of a small number of image frames of the target text content in all target image frames are clear and easy to identify, the scheme provided by the present application can fully utilize the effects of different positions of different image frames on the text recognition, mainly consider the image features of the positions with better text recognition effect in the target image frames with better text recognition effect to perform the text recognition, and obtain the target text. For another example, for the case that the pictures of the target text content in all target image frames are all incomplete due to being blocked by an obstacle, but the character incomplete positions of the target text content in different target image frames are not the same, the scheme provided by the present application can recognize the complete characters of the target text content by using the different incomplete character shapes of the target text content in each target image frame.
[0073] Several possible embodiments are given below:
[0074] Embodiment 1:
[0075] As shown in Figure 2 , specifically comprising the following steps:
[0076] S111, determining a target text image region to be identified in each of a plurality of target image frames of a video to be identified.
[0077] S112, for any target text image region, dividing the target text image region into a plurality of target sub-image regions.
[0078] S120, extracting at least one target image feature of the target text image region in each target image frame.
[0079] S130, for the target text image region, the target sub-image regions with the same relative position in different target image frames are combined into a target feature sequence.
[0080] S141A, for any target feature sequence, determining a first fusion weight corresponding to each target image feature in the target feature sequence based on the feature value of each target image feature in the first evaluation feature dimension.
[0081] S142A, for any target feature sequence, calculating a weighted average value of each target image feature in the target feature sequence based on the corresponding first fusion weight to obtain a first fusion feature.
[0082] For example, the target image feature contains multiple feature dimensions, including the clarity of the sub-image region corresponding to the target image feature, the pixel color difference, etc., which can be used to measure the preset first evaluation feature dimension whether the target image feature is conducive to text recognition, then the first fusion weight corresponding to each target image feature can be determined according to the feature value of each target image feature in the first evaluation feature dimension in the target feature sequence, and then the weighted average value of each target image feature in the target feature sequence is calculated based on the first fusion weight, and the feature fusion is realized. Specifically, the higher the clarity of the sub-image region corresponding to the target image feature, the greater the first fusion weight corresponding to the target image feature. The greater the pixel color difference corresponding to the target image feature, the greater the first fusion weight corresponding to the target image feature. The configuration principle of the first fusion weight can be set based on the characteristics of the first evaluation feature dimension, which is not limited herein.
[0083] S151A, determining a second fusion weight corresponding to each first fusion feature based on the feature value of each first fusion feature in the second evaluation feature dimension.
[0084] S152A, calculating a weighted sum of each first fusion feature based on the corresponding second fusion weight to obtain a second fusion feature.
[0085] For example, the target image feature contains multiple feature dimensions, including the relative position of the sub-image region corresponding to the target image feature relative to the target image region. Then, the relative position of the sub-image region corresponding to the first fusion feature relative to the target image region can be used as a second evaluation feature dimension, the second fusion weight is determined based on the relative position corresponding to each first fusion feature, and then each first fusion feature is weighted and summed based on the second fusion weight, realizing secondary fusion of features. Specifically, the closer the relative position of the sub-image region corresponding to the first fusion feature relative to the edge of the target image region, the smaller the second fusion weight corresponding to the first fusion feature. The configuration principle of the second fusion weight can be set based on the characteristics of the second evaluation feature dimension, and the embodiments of the present application do not make too many limitations here.
[0086] S160, determining the target text based on the second fusion feature.
[0087] In the specific implementation process, the second fusion feature can be input into a pre-trained text recognition model to obtain the target text input by the text recognition model. The text recognition model can be implemented by using a convolutional neural network (CNN), a support vector machine (SVM), a decision tree, a random forest, etc., and the embodiments of the present application do not make too many limitations here.
[0088] Embodiment 2:
[0089] Embodiment 2 is an embodiment implemented by using a transformer (Transformer) model algorithm. As shown in Figures 3-4 , the specific steps include:
[0090] S111, determining a target text image region to be recognized in a plurality of target image frames of a video to be recognized.
[0091] S112, for any target text image region, dividing the target text image region into a plurality of target sub-image regions.
[0092] S120, extracting at least one target image feature of the target text image region in each target image frame.
[0093] S130, grouping the target image features of the target sub-image regions with the same relative position relative to the target text image region in different target image frames into a target feature sequence.
[0094] S140B, for any target feature sequence, input the target feature sequence into a first Transformer encoder (Encoder) to obtain a first fusion feature corresponding to the target feature sequence output by the first Transformer encoder.
[0095] S150B, input each first fusion feature in sequence order into a second Transformer encoder to obtain a second fusion feature sequence output by the second Transformer encoder.
[0096] In a specific implementation process, each first fusion feature is grouped into a first fusion feature sequence, and then the first fusion feature sequence is input into the second Transformer encoder to obtain a second fusion feature sequence output by the second Transformer encoder.
[0097] S160B, input the second fusion feature sequence into a Transformer decoder (Decoder) to obtain a target text output by the Transformer decoder.
[0098] That is, in the scheme provided in Embodiment 2, the number of encoders of the Transformer model is changed from 1 to 2 compared with the existing standard Transformer model. In this embodiment, for any relative position (i, j), the target image features corresponding to different target image frames can be regarded as input tokens of the Transformer model. The first encoder is used to fuse the target image features of different target image frames at each same relative position in step S140B, and the first fusion features corresponding to each relative position are obtained by feature encoding. For example, for the first encoder, for the relative position (i, j), by inputting the target feature sequence S i,j into the first encoder, the first fusion feature G i,j output by the first encoder is obtained. Let the length of the first fusion feature be e. Then, the first fusion features G 1,1 , G 1,2 , …, G h,w corresponding to the relative positions (1, 1), (1, 2), …, (h, w) respectively are obtained. The second encoder is used to fuse the first fusion features corresponding to each relative position again in step S150B, and the first fusion features G 1,1 , G 1,2 , …, G h,w corresponding to the relative positions (1, 1), (1, 2), …, (h, w) respectively are grouped into a sequence G = Concat(G 1,1 , G 1,2 , …, G h,w) e R h·w·e The second encoder obtains a second fusion feature sequence corresponding to the target text image region in all target image frames as a whole through feature encoding.
[0099] For the step S140B and the step S150B, the features are input into the Transformer encoder in sequence, and the self-attention mechanism of the encoder is used to embed and encode the features.
[0100] Specifically, as shown in Figure 5 For the encoder (including the first encoder and the second encoder), the process mainly includes the following steps:
[0101] S210, sequentially determine the current encoding layer according to the order of the encoding layers in the encoder.
[0102] In the specific implementation process, the encoder in the Transformer model includes multiple encoding layers with the same structure but different model parameters, for example, the encoder in the standard Transformer model includes 6 encoding layers. The number of encoding layers in the encoder (the first encoder and the second encoder) in the embodiment can be set as needed, and the embodiment does not make too many limitations.
[0103] If there is an encoding layer in the encoder that has not been selected, step S220 is performed; if all the encoding layers in the encoder have been selected, step S260 is performed.
[0104] S220, the multi-head self-attention mechanism of the current encoding layer calculates the attention weights of the features in the input feature sequence, and outputs an attention feature sequence based on the attention weights of the features.
[0105] For the first encoder, the input feature sequence of the first encoding layer is the target feature sequence, and the input feature sequence of the non-first encoding layer is the output feature sequence of the upper encoding layer. For the second encoder, the input feature sequence of the first encoding layer is the sequence composed of the first fusion features, and the input feature sequence of the non-first encoding layer is the output feature sequence of the upper encoding layer.
[0106] S230, add the attention feature sequence and the target feature sequence and perform normalization processing to obtain a first sequence.
[0107] S240, input the first sequence into the fully connected feedforward neural network of the current encoding layer to obtain a second sequence output by the fully connected feedforward neural network.
[0108] S250, normalize the first sequence after adding the second sequence to obtain an output feature sequence corresponding to the current encoding layer. Return to step S210.
[0109] S260, take the output feature sequence of the last encoding layer as the final output feature sequence of the encoder.
[0110] Among them, for the first encoder, the first fusion feature is determined according to the output feature sequence of the last encoding layer (referred to as the final output feature sequence in this embodiment). For the second encoder, the output feature sequence of the last encoding layer (i.e. the final output feature sequence) is taken as the second fusion feature in the form of a sequence.
[0111] As an optional implementation, the step S140B, for any target feature sequence, inputting the target feature sequence into the Transformer first encoder to obtain a first fusion feature corresponding to the target feature sequence output by the Transformer first encoder, specifically comprising:
[0112] For any target feature sequence, input the target feature sequence into the Transformer first encoder to obtain a final output feature sequence corresponding to the target feature sequence output by the Transformer first encoder, and calculate the average value of part of the final output features in the final output feature sequence as the first fusion feature.
[0113] For example, the target feature sequence includes target image features of 16 target image frames, that is, the target feature sequence contains 16 tokens. Input the target feature sequence into the Transformer first encoder, and the Transformer first encoder outputs a final output feature sequence, which contains 16 final output features (also referred to as tokens). Then the average value of part of the tokens (for example, taking the maximum 3 tokens on the third evaluation feature dimension, and for example, taking the 3 tokens with the maximum sum of feature values on all feature dimensions) can be calculated, and the average value of the 3 tokens is taken as the first fusion feature.
[0114] As another optional implementation, as shown in Figure 4 The step S130 specifically comprises:
[0115] S131B, group the target image features of the target sub-image regions with the same relative position with respect to the target text image region in different target image frames into an original feature sequence.
[0116] S132B, for any original feature sequence, a classification token (CLS token) is added to the starting position of the original feature sequence to obtain a target feature sequence.
[0117] Correspondingly, the step S140B specifically comprises:
[0118] S140B', for any target feature sequence, the target feature sequence is input into the first Transformer encoder to obtain a final output feature sequence corresponding to the target feature sequence output by the first Transformer encoder, and a feature at a starting position in the final output feature sequence is taken as a first fusion feature.
[0119] The CLS embedding is a special mechanism introduced by the Transformer model when processing downstream tasks such as classification. In the Transformer model, a special embedding vector (i.e., CLS token) is added to the starting position of the feature sequence as a representative vector of the entire feature sequence. The CLS token then gradually obtains global information of the entire feature sequence during the model processing process, so as to facilitate subsequent classification prediction.
[0120] Let the length of each token be the length c of the target image feature. After obtaining the original feature sequence composed of each target image feature, a CLS token is additionally added to the starting position of the original sequence to obtain a target feature sequence Where T is the total number of target image frames. Then, based on the self-attention mechanism of the Transformer model, the CLS token will be fused with the target image features in different image frames. After the fusion is completed, the feature at the starting position in the final output sequence (i.e., the output token corresponding to the CLS token) can be taken as the first fusion feature of each image frame after fusion at the corresponding relative position.
[0121] Compared with the technical solution of embodiment 1, the self-attention mechanism of the Transformer model is used in embodiment 2, in the process of feature fusion, for any relative position, the influence of each target image feature corresponding to the relative position in different target image frames on the text recognition effect can be considered, so that in each target image feature of the target feature sequence corresponding to the relative position, the target image feature more conducive to text recognition has a greater influence on the value of the first fusion feature corresponding to the target feature sequence through the self-attention mechanism, so as to realize the effect that the first fusion feature obtained by fusing each target image feature of the target feature sequence more focuses on the target image feature conducive to text recognition; and when the first fusion features corresponding to each relative position are fused again, the relative position more conducive to text recognition has a greater influence on the value of the second fusion feature through the self-attention mechanism, so as to realize the effect that the second fusion feature corresponding to the whole target text image region obtained by fusing the first fusion features corresponding to each relative position again more focuses on the target image feature conducive to text recognition, and finally the target text recognition result is obtained through the Transformer decoder, and the recognition effect and robustness of the overall process are more optimal.
[0122] For example Figure 6 As shown, the text recognition of the “STOP” target text content in the video to be recognized is performed, first, the target image features of each target image frame are input into the CNN model to extract the target image features, then each target image feature of a target sub-image region with the same relative position with respect to the target text image region in different target image frames is combined to form a feature sequence, and a CLS token is added at the beginning of the sequence to obtain a target feature sequence. The target feature sequences corresponding to each relative position are input into the Transformer Encoder 1, and the feature at the beginning position output by the Transformer Encoder 1 is taken as the first fusion feature. Then, the first fusion features corresponding to each relative position are combined to form a sequence and input into the Transformer Encoder 2, and the output result of the Transformer Encoder 2 is input into the Transformer Decoder for decoding to obtain the final target text “STOP”.
[0123] Further, after the text recognition of the target text content is completed through the above steps, the identified target text can be configured to provide the recommended target text image area and / or the recommended target image frame to the user, so that the user can determine whether the target text is correct by watching the recommended target text image area and / or the recommended target image frame. In the specific implementation process, a target text image area / target image frame can be randomly selected as the recommended target text image area / recommended target image frame; or the target text image area / target image frame with the best image clarity can be selected as the recommended target text image area / recommended target image frame, or other ways to determine the recommended target text image area / recommended target image frame.
[0124] In Embodiment 2, considering that in the self-attention mechanism of the Transformer encoder, the query (query) vector q a of a token (for example, token a) is calculated as b the key (key) vector k a of another token (for example, token b), the dot product represents the contribution of token b to the output of token a, q b ·k b The greater the value is, the more token a pays attention to the information of token b. In the first encoder, as shown in Figure 7 , the attention parameters of the CLS token and other tokens after fusing the information of each target image frame can reflect the importance of the corresponding frame features. Based on this, as shown in Figure 4 , another possible implementation of determining the recommended target text image area / recommended target image frame is:
[0125] S171B, for any target image frame, determining the recommendation score of the target text image area of the target image frame according to the attention weight of the CLS token of the last encoding layer of the first Transformer encoder to each target sub-image area.
[0126] Optionally, the recommendation score Attn(t) satisfies the following relationship:
[0127]
[0128] t is the sequence number of the target image frame, n is the sequence number of the target sub-image area, N is the total number of target sub-image areas in a target image frame, m is the sequence number of the attention head, M is the total number of attention heads, is the query (query) vector of the CLS token of the nth last input feature sequence in the last encoding layer of the first Transformer encoder corresponding to the mth attention head. is the key vector corresponding to the mth self-attention mechanism for the feature corresponding to the nth final input feature sequence and the tth frame of the target text image region.
[0129] S172B, determining the target text image region with the maximum recommendation score as the recommended target text image region, and / or determining the target image frame corresponding to the target text image region with the maximum recommendation score as the recommended target image frame.
[0130] In this way, by using the attention parameters of the CLS token and the features corresponding to each target image frame in the self-attention mechanism as the reference factors of the recommended target text image region and the recommended target image frame, since the attention parameters of the CLS token and the features corresponding to each target image frame can reflect the contribution of the features corresponding to each target image frame to the final recognition result, the greater the attention parameter, the greater the contribution of the target image feature corresponding to the target image frame to the target text recognition process, and the target image frame corresponding to the target image feature with the greater attention parameter is generally cleaner and clearer or not damaged and obstructed, so that the recommended target text image region and the target image frame are more referential.
[0131] Based on the same inventive concept, the embodiments of the present application also provide a device for recognizing text appearing in a video, as shown in the accompanying drawings, comprising: Figure 8
[0132] a target detection module M1, configured to determine a target text image region to be recognized in a plurality of target image frames of a video to be recognized, and for any target text image region, divide the target text image region into a plurality of target sub-image regions; wherein the target text image region is an image region where the target text content is located;
[0133] a feature extraction module M2, configured to extract at least one target image feature of the target text image region in each target image frame; and group the target image features of the target sub-image regions with the same relative position with respect to the target text image region in different target image frames into a target feature sequence;
[0134] a local feature fusion module M3, configured to, for any target feature sequence, fuse the target image features in the target feature sequence to obtain a first fusion feature;
[0135] a global feature fusion module M4, configured to fuse the first fusion features to obtain a second fusion feature;
[0136] The text recognition module M5 is configured to determine a target text based on the second fusion feature.
[0137] As an optional implementation, the local feature fusion module M3 is specifically configured to:
[0138] For any target feature sequence, the local feature fusion module M3 is configured to determine a first fusion weight corresponding to each target image feature in the target feature sequence based on a feature value of the target image feature in the target feature sequence in a first evaluation feature dimension, and obtain a first fusion feature by calculating a weighted average value of each target image feature in the target feature sequence based on the corresponding first fusion weight.
[0139] The global feature fusion module M4 is specifically configured to:
[0140] The global feature fusion module M4 is configured to determine a second fusion weight corresponding to each first fusion feature based on a feature value of the first fusion feature in a second evaluation feature dimension, and obtain a second fusion feature by calculating a weighted sum of each first fusion feature based on the corresponding second fusion weight.
[0141] As another optional implementation, the local feature fusion module M3 is specifically configured to:
[0142] For any target feature sequence, the local feature fusion module M3 is configured to input the target feature sequence into a first Transformer encoder to obtain a first fusion feature corresponding to the target feature sequence output by the first Transformer encoder;
[0143] The global feature fusion module M4 is specifically configured to:
[0144] The global feature fusion module M4 is configured to input each first fusion feature in sequence order into a second Transformer encoder to obtain a second fusion feature sequence output by the second Transformer encoder.
[0145] The text recognition module M5 is specifically configured to:
[0146] The text recognition module M5 is configured to input the second fusion feature sequence into a Transformer decoder to obtain a target text output by the Transformer decoder.
[0147] Further optionally, the step of grouping each target image feature in a target sub-image region having the same relative position with respect to a target text image region in different target image frames into a target feature sequence includes:
[0148] each target image feature of a target sub-image region in different target image frames with a same relative position to a target text image region is composed into an original feature sequence, and a classification mark CLS token is added to a start position of the original feature sequence to obtain a target feature sequence;
[0149] The target feature sequence is input into the Transformer first encoder to obtain a first fusion feature corresponding to the target feature sequence output by the Transformer first encoder.
[0150] The target feature sequence is input into the Transformer first encoder to obtain a first fusion feature corresponding to the target feature sequence output by the Transformer first encoder.
[0151] Optionally, the apparatus further comprises:
[0152] An image recommendation module M6 is configured to determine a recommended target text image region and / or a recommended target image frame.
[0153] Further optionally, the image recommendation module M6 is specifically configured to:
[0154] For any target image frame, a recommendation score of a target text image region of the target image frame is determined according to attention weights of each target sub-image region calculated by the CLS token of the last encoding layer of the Transformer first encoder.
[0155] A target text image region with a maximum recommendation score is determined as a recommended target text image region, and / or a target image frame corresponding to the target text image region with the maximum recommendation score is determined as a recommended target image frame.
[0156] Further optionally, the recommendation score Attn(t) satisfies the following relationship:
[0157]
[0158] t is a serial number of a target image frame, n is a serial number of a target sub-image region in a target image, N is a total number of target sub-image regions in a target image frame, m is a serial number of an attention head, and M is a total number of attention heads, is a query vector corresponding to an mth attention head of a last encoding layer of the Transformer first encoder for a CLS token of an nth final input feature sequence, The key vector corresponding to the mth self-attention mechanism corresponds to the feature of the nth final-stage input feature sequence in the tth frame of the target feature sequence;
[0159] For any target feature sequence, the final-stage input feature sequence is the input feature sequence of the last encoding layer of the first Transformer encoder in the process of inputting the target feature sequence into the first Transformer encoder for layer-by-layer processing of multi-level encoding layers to obtain the first fusion feature.
[0160] Optionally, the method further includes:
[0161] Text target detection is performed on the plurality of target image frames in the video to be recognized to determine the text image regions.
[0162] Each text image region is classified to determine the text image regions belonging to the same text content, wherein the text image regions corresponding to the target text content are determined as the target text image regions.
[0163] It should be understood that the above-described device embodiments are merely illustrative, for example, the division of the modules is only a logical functional division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection between the modules through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0164] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, i.e., they can be located in one place or distributed on a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.
[0165] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. If the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can be stored in a computer readable storage medium.
[0166] Since the specific manner in which each module of the apparatus for recognizing text appearing in a video performs operations has been described in detail in the content of the corresponding method for recognizing text appearing in a video, no further elaboration is provided here.
[0167] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, comprising: a processor 110 and a memory 120 for storing instructions executable by the processor 110; wherein the processor 110 is configured to execute the instructions to implement the method for recognizing text appearing in a video. Figure 9
[0168] In the implementation process, the device can have great differences due to different configurations or performances, and can include one or more processors 110, memories 120, computer readable storage media 130, one or more application programs 131 or data 132 included in the memory 120 and / or computer readable storage medium 130. The memory 120 and / or computer readable storage medium 130 can also include one or more operating systems 133, such as Windows, Mac OS, Linux, IOS, Android, Unix, FreeBSD, etc. Among them, the memory 120 and the computer readable storage medium 130 can be temporary storage or persistent storage. The application program 131 can include one or more modules (not shown in the foregoing description), and each module can include a series of instruction operations. Further, the processor 110 can be configured to communicate with the computer readable storage medium 130 and execute a series of instruction operations in the computer readable storage medium 130 on the device. The device can also include one or more power supplies (not shown in the foregoing description), one or more network interfaces 140, the network interface 140 including a wired network interface 141 and / or a wireless network interface 142, and one or more input / output interfaces 143. Figure 9 Figure 9
[0169] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium, which stores computer program code, when the computer program code is run on a computer, so that the computer implements the method for recognizing text appearing in a video.
[0170] The computer readable storage medium can be any available medium or data storage that can be accessed by a computer and can be integrated into a server, data center, etc. data storage device containing one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a U disk, a magnetic tape, a Read-Only Memory (ROM), a Random Access Memory (RAM)), an optical medium (such as a high-density digital video disc (Digital Video Disc, DVD), a video compact disc (Video Compact Disc, VCD)), or a semiconductor medium (such as a Solid State Disk (SSD)) and the like.
[0171] Since the above-mentioned computer readable storage medium solves the problem by the same principle as the method of recognizing the text appearing in the video described above, the implementation of the above-mentioned computer readable storage medium can refer to the implementation of the method, and the repeated parts will not be described.
[0172] Based on the same inventive concept, the embodiments of the present application also provide a computer program product, which comprises computer program code, when the computer program code runs on a computer, so that the computer implements the method of recognizing the text appearing in the video.
[0173] The computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode.
[0174] Since the above-mentioned computer program product solves the problem by the same principle as the method of recognizing the text appearing in the video described above, the implementation of the above-mentioned computer program product can refer to the implementation of the method, and the repeated parts will not be described.
[0175] The method for recognizing the text appearing in the video and the related hardware provided by the embodiment of the application first fuse the image features of different image frames corresponding to the target text content according to positions, then fuse the image features according to positions, and finally perform text recognition based on the fused features, so that the feature cross-image frame analysis can be realized in the process of fusing the image features according to positions, the advantages of the image features of positions with better recognition effects in each image frame for text recognition are fully utilized, the disadvantages of the image features of positions with poor recognition effects in each image frame for text recognition are abandoned, the advantages and disadvantages of each image frame in the text recognition process are complemented, and the text recognition effect and the robustness of text recognition are better than those of the prior art.
[0176] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.
[0177] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks
[0178] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks
[0179] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flowchart
[0180] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.
Claims
1. A method for recognizing text appearing in a video, characterized in that, include: Determine the target text image region to be identified in multiple target image frames of the video to be identified, and for any target text image region, divide the target text image region into multiple target sub-image regions; The target text image region is the image region where the target text content is located; Extract at least one target image feature from the target text image region in each target image frame; A target feature sequence is formed by combining the target image features of the target sub-image regions that are in the same relative position to the target text image region in different target image frames. For any target feature sequence, the target image features in the target feature sequence are fused to obtain the first fused feature; The first fusion features are fused together to obtain the second fusion feature; The target text is determined based on the second fusion feature.
2. The method as described in claim 1, characterized in that, For any target feature sequence, fusing the target image features in the target feature sequence to obtain a first fused feature includes: For any target feature sequence, the target feature sequence is input to the first encoder of the Transformer to obtain the first fused feature corresponding to the target feature sequence output by the first encoder of the Transformer; The process of fusing the first fusion features to obtain the second fusion feature includes: Each first fused feature is input into the Transformer second encoder in sequence to obtain the second fused feature sequence output by the Transformer second encoder; The step of determining the target text based on the second fusion feature includes: The second fused feature sequence is input into the Transformer decoder to obtain the target text output by the Transformer decoder.
3. The method as described in claim 2, characterized in that, The step of forming a target feature sequence by combining the target image features of target sub-image regions with the same relative position to the target text image region in different target image frames includes: In different target image frames, the target image features of the target sub-image regions that are in the same relative position to the target text image region are combined into an original feature sequence; for any original feature sequence, the classification tag CLS token is added to the beginning position of the original feature sequence to obtain the target feature sequence; The step of inputting the target feature sequence into the Transformer first encoder to obtain the first fused feature corresponding to the target feature sequence output by the Transformer first encoder includes: The target feature sequence is input into the first encoder of the Transformer to obtain the final output feature sequence corresponding to the target feature sequence output by the first encoder of the Transformer. The feature at the starting position in the final output feature sequence is used as the first fusion feature.
4. The method as described in claim 3, characterized in that, The method further includes: For any target image frame, the recommended score of the target text image region of the target image frame is determined based on the attention weight of each target sub-image region by the CLS token calculated by the last coding layer of the Transformer first encoder; The target text image region with the highest recommendation score is determined as the recommended target text image region, and / or the target image frame corresponding to the target text image region with the highest recommendation score is determined as the recommended target image frame.
5. The method as described in claim 4, characterized in that, The recommended score Attn(t) satisfies the following relationship: t is the sequence number of the target image frame, n is the sequence number of the target sub-image region in the target image, N is the total number of target sub-image regions in a frame of the target image, m is the sequence number of the attention head, and M is the total number of attention heads. This is the query vector corresponding to the m-th attention head of the last encoding layer of the first encoder of the Transformer, representing the CLS token of the nth final-level input feature sequence. The key vector corresponding to the feature of the nth final-level input feature sequence in frame t at the mth self-attention mechanism; For any target feature sequence, the final input feature sequence is the input feature sequence of the last coding layer of the Transformer first encoder during the process of inputting the target feature sequence into the Transformer first encoder for multi-level coding layer processing to obtain the first fused feature.
6. The method as described in claim 1, characterized in that, The process of determining the target text image region to be identified in multiple target image frames of the video to be identified includes: Perform text target detection on multiple target image frames in the video to be identified, and determine the text image region; Each text image region is classified to identify text image regions belonging to the same text content, with the text image region corresponding to the target text content being identified as the target text image region.
7. A device for recognizing text appearing in a video, characterized in that, include: The target detection module is used to determine the target text image region to be identified in multiple target image frames of the video to be identified, and to divide any target text image region into multiple target sub-image regions. The target text image region is the image region where the target text content is located; The feature extraction module is used to extract at least one target image feature of the target text image region in each target image frame; A target feature sequence is formed by combining the target image features of the target sub-image regions that are in the same relative position to the target text image region in different target image frames. The local feature fusion module is used to fuse the target image features in any target feature sequence to obtain a first fused feature; The global feature fusion module is used to fuse the first fusion features to obtain the second fusion feature; The text recognition module determines the target text based on the second fusion feature.
8. An electronic device, characterized in that, include: A processor and a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the method for recognizing text appearing in a video as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code that, when executed on a computer, causes the computer to perform the method for recognizing text appearing in a video as described in any one of claims 1-6.
10. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method for recognizing text appearing in a video as described in any one of claims 1-6.