Scene text detection method, device and model and training method and training device thereof

Abstract

The present invention discloses a scene text detection method, device and model and a training method and a training device thereof, which relate to the field of computer vision. By using the simple relational features between text candidate frames, by distinguishing whether the candidate frames belong to the same text instance for comparative learning, the effective local relational features between the candidate frames belonging to the same text instance are strengthened, the discrimination ability for text instances is supplemented, the positioning and combination of text frames are organically integrated into feature learning, and subsequent text frame classification and regression tasks are more easily distinguished whether the text frame contains complete text, thereby improving the text detection performance of the model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a scene text detection model and a training method and a training device thereof, and a scene text detection method and a detection device. Background Art

[0002] OCR (Optical Character Recognition) traditionally refers to analyzing and processing input scanned document images to identify text information in the image. Scene Text Recognition (STR) refers to identifying text information in natural scene images. Text recognition in natural scene images is much more difficult than text recognition in scanned document images because its text display form is extremely rich. For example, it allows the mixing of texts in multiple languages, characters can have different sizes, fonts, colors, brightness, contrast, etc., text lines may have horizontal, vertical, curved, rotated, twisted and other styles, text can appear on flat, curved or wrinkled surfaces, and there are complex interfering textures near the text area.

[0003] A scene text detection technology based on candidate boxes, which first locates text candidate boxes by laying out anchor boxes, and then combines text candidate boxes belonging to the same text instance through appearance semantic relationships or geometric position relationships.

[0004] Research has found that the positioning task and combination task of the above detection technology are independent and will affect the detection results. Summary of the invention

[0005] The disclosed embodiment utilizes the naive relational features between text candidate boxes, and performs comparative learning by distinguishing whether the candidate boxes belong to the same text instance, thereby strengthening the effective local relational features between the candidate boxes belonging to the same text instance, supplementing the ability to distinguish text instances, and organically integrating the positioning and combination of text boxes into feature learning, helping to perform subsequent text box classification and regression tasks more easily to distinguish whether the text box contains complete text, thereby improving the text detection performance of the model.

[0006] Some embodiments of the present disclosure provide a method for training a scene text detection model, including:

[0007] Generate multiple text candidate boxes based on images of natural scenes;

[0008] By using the simple relational features between text candidate boxes and identifying whether the text candidate boxes belong to the same text instance, the relationship between the text candidate boxes is compared and learned.

[0009] Generate text candidate box features of identifiable text instances of each text candidate box according to the comparative learning results;

[0010] According to the text candidate box features of the identifiable text instances of each text candidate box, the text detection results are obtained through classification and regression of the text candidate boxes and segmentation of the text regions within the text candidate boxes, including the detected text target boxes and texts;

[0011] According to the difference between the detected text target box and text and the annotated text target box and text, the total loss is calculated, and the parameters of the scene text detection model are updated according to the total loss.

[0012] In some embodiments, a method for determining a simple relationship feature between text candidate boxes includes:

[0013] Based on the appearance semantics and geometric position of each text candidate box, generating a first text candidate box feature of each text candidate box;

[0014] According to the first text candidate box features of each text candidate box, a first relation feature between each text candidate box and other text candidate boxes is generated as a naive relation feature.

[0015] In some embodiments, the comparative learning of the relationship between text candidate boxes includes: based on the supervision of the comparative learning loss function at the text instance level, comparative learning of the first relationship features of each text candidate box is performed, the association of text candidate boxes belonging to the same text instance is enhanced, and the association of text candidate boxes belonging to different text instances is weakened, so as to generate a second relationship feature of a recognizable text instance between each text candidate box and other text candidate boxes.

[0016] In some embodiments, generating text candidate box features of the recognizable text instance of each text candidate box includes: fusing the first text candidate box features and the second relationship features of each text candidate box to obtain the second text candidate box features of the recognizable text instance of each text candidate box.

[0017] In some embodiments, calculating the total loss includes:

[0018] The total loss is calculated based on the set total loss function, combining the gap between the detected text target box and text and the annotated text target box and text.

[0019] Among them, the total loss function includes the generation loss term, classification loss term and regression loss term of the text candidate box, the text area segmentation loss term within the text candidate box, and the contrastive learning loss term.

[0020] In some embodiments, the contrastive learning loss function includes: contrastive learning loss, an associated representation of positive sample pairs, and an associated representation of negative sample pairs, wherein the contrastive learning loss is negatively correlated with the associated representation of the positive sample pairs, and the contrastive learning loss is positively correlated with the associated representation of the negative sample pairs, wherein, for each annotated text target box, a first text candidate box that meets the proximity condition and a second text candidate box that does not meet the proximity condition are determined relative to the annotated text target box, the first relationship feature of the annotated text target box and the first relationship feature of each first text candidate box constitute a positive sample pair, and the first relationship feature of the annotated text target box and the first relationship feature of each second text candidate box constitute a negative sample pair.

[0021] In some embodiments, the proximity condition is that the ratio of the intersection and union between the text candidate box and the text target box is within a preset low threshold range, and the ratio of the intersection between the text candidate box and the text target box to the text candidate box is within a preset high threshold range.

[0022] In some embodiments, generating the first relationship feature between each text candidate box and other text candidate boxes based on the first text candidate box features of each text candidate box includes: for each text candidate box, using the similarity between the first text candidate box feature of each text candidate box and the first text candidate box features of other text candidate boxes as a weight, and calculating the first relationship feature between each text candidate box and other text candidate boxes by weighted summation.

[0023] In some embodiments, the similarity is obtained by performing a dot multiplication of the first text candidate box feature of each text candidate box and the first text candidate box features of other text candidate boxes and performing normalization.

[0024] Some embodiments of the present disclosure provide a scene text detection method, including:

[0025] Generate multiple text candidate boxes based on images of natural scenes;

[0026] Generating text candidate box features of identifiable text instances of each text candidate box;

[0027] According to the text candidate box features of the recognizable text instances of each text candidate box, through the classification and regression of the text candidate boxes and the segmentation of the text regions within the text candidate boxes, the text detection results are obtained, including the detected text target boxes and text.

[0028] In some embodiments, generating text candidate box features of the recognizable text instance of each text candidate box includes:

[0029] Based on the appearance semantics and geometric position of each text candidate box, generating a first text candidate box feature of each text candidate box;

[0030] Using the comparative learning result, according to the first text candidate frame features of each text candidate frame, a second relation feature of identifiable text instances between each text candidate frame and other text candidate frames is generated;

[0031] The first text candidate box feature and the second relationship feature of each text candidate box are fused to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

[0032] In some embodiments, generating the first text candidate box feature of each text candidate box includes: using candidate box region pooling to extract the first text candidate box feature of each text candidate box, including appearance semantic features and geometric position features.

[0033] In some embodiments, generating multiple text candidate boxes based on an image of a natural scene includes: extracting a multi-scale feature map of the image; and generating multiple text candidate boxes through a region generation network for each feature map.

[0034] In some embodiments, a feature pyramid network is used to extract multi-scale feature maps of an image.

[0035] In some embodiments, for each feature map, generating multiple text candidate boxes through a region generation network includes:

[0036] Lay s anchor boxes of fixed size and aspect ratio at each position of the feature map of size H×W, and each feature map corresponds to H×W×s anchor boxes;

[0037] Use a fully convolutional neural network to classify each anchor box into foreground and background, and correct the position and size of each anchor box;

[0038] A preset number N of anchor boxes with the highest classification scores are retained and output as text candidate boxes.

[0039] In some embodiments, the classification and regression of text candidate boxes includes:

[0040] Use the classifier to classify each text candidate box into foreground and background.

[0041] The regressor is used to correct the position and size of each text candidate box.

[0042] In some embodiments, segmenting the text region within the text candidate box includes:

[0043] Select a preset number of k text candidate boxes with the highest classification scores of the classifier as the input of the candidate box area pooling to obtain k feature maps of fixed size;

[0044] Input each feature map into the fully convolutional neural network to obtain the text separation region of each feature map;

[0045] According to the transformation relationship between the feature map and the text candidate box, the text separation area of ​​the feature map is mapped to the text separation area of ​​the text candidate box;

[0046] According to the text separation area of ​​the text candidate box and the position of the text candidate box in the image, the text separation area and the corresponding text in the image are obtained as the text detection result.

[0047] Some embodiments of the present disclosure provide a scene text detection model, including:

[0048] A text candidate box generation module is configured to generate multiple text candidate boxes based on an image of a natural scene;

[0049] A text candidate box feature generation module is configured to generate text candidate box features of a recognizable text instance of each text candidate box;

[0050] The text detection module is configured to obtain text detection results, including detected text target boxes and text, through classification and regression of text candidate boxes and text area segmentation within the text candidate boxes based on text candidate box features of recognizable text instances of each text candidate box.

[0051] In some embodiments, the text candidate box feature generation module includes:

[0052] A first text candidate box feature generating unit is configured to generate a first text candidate box feature of each text candidate box based on the appearance semantics and geometric position of each text candidate box;

[0053] A second relationship feature generating unit is configured to generate a second relationship feature of a recognizable text instance between each text candidate box and other text candidate boxes according to the first text candidate box features of each text candidate box by using the comparative learning result;

[0054] The second text candidate box feature generating unit is configured to fuse the first text candidate box feature and the second relationship feature of each text candidate box to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

[0055] In some embodiments, the text candidate box generation module includes:

[0056] Feature pyramid network is used to extract multi-scale feature maps of images.

[0057] The region generation network is used to generate multiple text candidate boxes for each feature map.

[0058] In some embodiments, the text detection module includes:

[0059] The classifier is used to classify each text candidate box into foreground and background.

[0060] Regressor, used to correct the position and size of each text candidate box;

[0061] The text segmentation unit is used to segment the text area within the text candidate box.

[0062] Some embodiments of the present disclosure provide a training device for a scene text detection model, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a training method for the scene text detection model based on instructions stored in the memory.

[0063] Some embodiments of the present disclosure provide a scene text detection device, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a scene text detection method based on instructions stored in the memory.

[0064] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a scene text detection model or steps of a scene text detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0066] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0067] Figure 1 A flowchart illustrating a method for training a scene text detection model according to some embodiments of the present disclosure is shown.

[0068] Figure 2 A schematic diagram showing the process of a method for training a scene text detection model according to some embodiments of the present disclosure.

[0069] Figure 3 A schematic diagram showing a scene text detection model according to some embodiments of the present disclosure.

[0070] Figure 4 A schematic diagram showing a scene text detection method according to some embodiments of the present disclosure.

[0071] Figure 5A schematic diagram of the structure of a training device for a scene text detection model according to some embodiments of the present disclosure.

[0072] Figure 6 The present invention is a schematic diagram of the structure of a scene text detection device according to some embodiments of the present invention. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

[0074] Unless otherwise specified, descriptions such as “first” and “second” in the present disclosure are used to distinguish different objects and are not used to indicate meanings such as size or time sequence.

[0075] Figure 1 A flowchart illustrating a method for training a scene text detection model according to some embodiments of the present disclosure is shown.

[0076] like Figure 1 As shown, the training method of the scene text detection model of this embodiment includes: steps 110-150.

[0077] In step 110 , a plurality of text candidate boxes are generated based on the image of the natural scene.

[0078] In model training, the image of the natural scene may be, for example, an image sample of the natural scene, which is annotated with corresponding text target boxes and text.

[0079] In some embodiments, generating multiple text candidate boxes based on an image of a natural scene includes: steps 111 - 112 .

[0080] In step 111, a multi-scale feature map of the image is extracted, for example, by using Feature Pyramid Networks (FPN for short) to extract a multi-scale feature map of the image.

[0081] In step 112, for each feature map, a plurality of text candidate boxes are generated by a region proposal network (RPN for short).

[0082] RPN is used to generate a large number of candidate boxes containing foreground objects for each image. In the scenario of text detection, the foreground is text, and RPN can generate multiple text candidate boxes for each image.

[0083] RPN generates N text candidate boxes for each image, assuming N = 2000, and N can also take other values. In order to generate text candidate boxes, the multi-scale feature map output by FPN is used as the input of RPN, and s anchor boxes of fixed size and aspect ratio are laid at each position of the feature map of size H×W, and each feature map corresponds to H×W×s anchor boxes; a lightweight fully convolutional neural network (FCN) is used to classify each anchor box into foreground and background, and to correct the position and size of each anchor box; the preset number N anchor boxes with the highest classification scores are retained as the text candidate box output.

[0084] The loss function L of RPN during training rpn Includes classification item L rpn_cls and the regression term L rpn_reg , that is, L rpn =L rpn_cls +L rpn_reg Among them, the classification item L rpn_cls The loss function is, for example, softmax, and the regression term L rpn_reg The loss function is, for example, Smooth L1.

[0085] Subsequently, according to steps 120 and 130, the naive relational features (first relational features) between the text candidate boxes are used to perform comparative learning on the relationship between the text candidate boxes by identifying whether the text candidate boxes belong to the same text instance; based on the comparative learning results, text candidate box features (second text candidate box features) of the identifiable text instance of each text candidate box are generated, which is marked as step 123.

[0086] In step 120 , a first text candidate box feature of each text candidate box is generated based on the appearance semantics and geometric position of each text candidate box.

[0087] Generating the first text candidate box features of each text candidate box includes: using the candidate box region pooling (Region of Interest Pooling, referred to as RoI Pooling), extracting the first text candidate box features of each text candidate box, including appearance semantic features and geometric position features. For example, N first text candidate box features corresponding to N text candidate boxes are extracted respectively, and are recorded as in, represents the appearance semantic features of the i-th text candidate box, Represents the geometric position features of the i-th text candidate box.

[0088] In step 130, based on the first text candidate box features of each text candidate box, through comparative learning, the relevance of the text candidate boxes belonging to the same text instance is enhanced, while the relevance of the text candidate boxes belonging to different text instances is weakened, so as to generate a second text candidate box feature of a recognizable text instance for each text candidate box.

[0089] In some embodiments, step 130 includes steps 131 - 133 .

[0090] In step 131, a first relationship feature between each text candidate box and other text candidate boxes is generated according to the first text candidate box feature of each text candidate box.

[0091] In some embodiments, generating the first relationship feature includes: for each text candidate box, using the similarity between the first text candidate box feature of each text candidate box and the first text candidate box features of other text candidate boxes as a weight, and calculating the first relationship feature between each text candidate box and other text candidate boxes by weighted summation. Wherein, the similarity is obtained by dot multiplication of the first text candidate box feature of each text candidate box and the first text candidate box feature of other text candidate boxes and normalization.

[0092] The first relation feature is also called the naive relation feature. This naive relation feature does not have enough prior knowledge to guide it, which may lead to ignoring the relationship between some important text candidate frames, or paying too much attention to some unnecessary relationships. However, candidate frames belonging to the same text instance should establish a more weighted relationship to help the candidate frames locate the complete text more accurately; correspondingly, candidate frames that do not belong to the same text instance should minimize the relationship to prevent adhesion between different text instances.

[0093] In step 132, based on the supervision of the contrastive learning loss function at the text instance level, the first relational features of each text candidate box are contrastively learned to enhance the relevance of text candidate boxes belonging to the same text instance, while weakening the relevance of text candidate boxes belonging to different text instances, so as to generate a second relational feature of identifiable text instances between each text candidate box and other text candidate boxes, thereby helping subsequent text box classification and regression to make more accurate predictions.

[0094] The contrastive learning loss function at the text instance level includes: contrastive learning loss, association representation of positive sample pairs and association representation of negative sample pairs, wherein the contrastive learning loss is negatively correlated with the association representation of positive sample pairs, and the contrastive learning loss is positively correlated with the association representation of negative sample pairs. For each annotated text target frame, a first text candidate frame that meets the proximity condition and a second text candidate frame that does not meet the proximity condition are determined relative to the annotated text target frame, the first relational feature of the annotated text target frame and the first relational feature of each first text candidate frame constitute a positive sample pair, and the first relational feature of the annotated text target frame and the first relational feature of each second text candidate frame constitute a negative sample pair.

[0095] According to the task scenario and motivation, the first relation feature (referred to as “target relation feature”, denoted as q) corresponding to N′ annotated text target boxes (real target labels) is taken. i ) as For each annotated text target box, there are M adjacent first text candidate boxes, and their corresponding first relationship features As the target relation feature q i M positive samples; for the first relational features corresponding to K non-adjacent second text candidate boxes Considered as target relation feature q i There are K negative samples. Constitute a positive sample pair, constitute a negative sample pair.

[0096] The contrastive learning loss function formula at the text instance level is expressed as follows:

[0097]

[0098] Among them, exp represents the exponential function with the natural constant e as the base, and τ is the temperature hyperparameter.

[0099] The proximity condition is: the ratio of the intersection and the union between the text candidate box and the text target box is within a preset low threshold range, for example, 0.1 to 1.0, for example, 0.1 to 0.5; and the ratio of the intersection between the text candidate box and the text target box to the text candidate box is within a preset high threshold range, for example, 0.7 to 1.0. The high threshold range makes the proportion of the text candidate box falling in the text target box very high. The low threshold range makes the number of positive sample pairs as much as possible.

[0100] In step 133, the first text candidate box feature and the second relationship feature of each text candidate box are fused to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

[0101] The fusion method is, for example, to add the first text candidate frame feature and the second relationship feature of each text candidate frame to obtain the second text candidate frame feature of the identifiable text instance of each text candidate frame.

[0102] Therefore, by utilizing the naive relational features between text candidate boxes and performing comparative learning by identifying whether the candidate boxes belong to the same text instance, the effective local relational features between the candidate boxes belonging to the same text instance are strengthened, the ability to discriminate text instances is supplemented, and the positioning and combination of text boxes are organically integrated into feature learning, which helps to make it easier to distinguish whether the text box contains complete text in subsequent text box classification and regression tasks, thereby improving the text detection performance of the model.

[0103] In step 140 , based on the second text candidate frame features of each text candidate frame, text detection results are obtained through classification and regression of the text candidate frames and segmentation of the text regions within the text candidate frames, including detected text target frames and text.

[0104] The classification of the text candidate box includes: using a classifier to perform binary classification of foreground (ie, text) and background on each text candidate box, and inputting the second text candidate box feature of the text candidate box into the classifier to obtain the category of the text candidate box.

[0105] The regression of the text candidate box includes: using a regressor to correct the position and size of each text candidate box, predicting the offset of the center point and aspect ratio of the text candidate box and the text target box, and inputting the second text candidate box feature of the text candidate box into the regressor to obtain the position and size of the corrected text candidate box.

[0106] During the training phase, the loss function of the candidate box classification and regression branch includes the classification loss term L cls and the regression loss term L reg Among them, the classification loss term L cls The loss function is, for example, softmax, and the regression loss term L reg The loss function is, for example, Smooth L1.

[0107] In some embodiments, text area segmentation within a text candidate box includes: selecting a preset number (k, such as k=100) of text candidate boxes with the highest classification scores of the classifier as inputs for candidate box area pooling to obtain k feature maps of a fixed size (such as 14×14); inputting each feature map into a four-layer fully convolutional neural network to obtain the text separation area of ​​each feature map; mapping the text separation area of ​​the feature map to the text separation area of ​​the text candidate box based on the transformation relationship between the feature map and the text candidate box; and obtaining the text separation area and the corresponding text in the image as a text detection result based on the text separation area of ​​the text candidate box and the position of the text candidate box in the image.

[0108] During the training phase, the loss function of the text region segmentation branch within the text candidate box is set to L mask , for example, a binary segmentation loss function.

[0109] In step 150, the total loss is calculated based on the set total loss function combined with the gap between the detected text target box and text and the annotated text target box and text, and the parameters of the scene text detection model are updated according to the total loss.

[0110] The total loss function includes the generation loss, classification loss, and regression loss of the text candidate box, the text region segmentation loss within the text candidate box, and the contrastive learning loss. The formula is as follows:

[0111] L=L rpn +L cls +L reg +L mask +λL InsCL

[0112] The meanings of the symbols in the formula refer to the above.

[0113] According to the training samples, the parameters of the scene text detection model are iteratively updated according to the total loss until the training termination condition is met, for example, the total loss is less than a preset value, or a certain number of iterations is reached.

[0114] The above embodiment utilizes the naive relational features between text candidate boxes, and performs comparative learning by distinguishing whether the candidate boxes belong to the same text instance, thereby strengthening the effective local relational features between the candidate boxes belonging to the same text instance, supplementing the ability to distinguish text instances, and organically integrating the positioning and combination of text boxes into feature learning, helping to perform subsequent text box classification and regression tasks more easily to distinguish whether the text box contains complete text, thereby improving the text detection performance of the model.

[0115] Figure 2 A schematic diagram showing the process of a method for training a scene text detection model according to some embodiments of the present disclosure.

[0116] like Figure 2As shown, an image sample of a natural scene contains characters such as "RYA" and "BAGGAGE", and these characters have different sizes. The feature pyramid network FPN is used to extract the multi-scale feature map of the image sample, and multiple text candidate boxes of different scales are generated based on the feature map through the region generation network RPN. For example, the box around "Y", the box around "RYA", the box around "BA", the box around "GE", the box around "BAGGAGE", etc. The first text candidate box feature of each text candidate box is extracted using candidate box region pooling. According to the first text candidate box feature of each text candidate box, the first relationship feature between each text candidate box and other text candidate boxes is generated. Based on the supervision of the contrastive learning loss function at the text instance level, the first relationship feature of each text candidate box is contrastively learned to enhance the relevance of the text candidate boxes belonging to the same text instance, while weakening the relevance of the text candidate boxes belonging to different text instances, so as to generate the second relationship feature of the identifiable text instance between each text candidate box and other text candidate boxes. The first text candidate box feature and the second relationship feature of each text candidate box are fused to obtain the second text candidate box feature of the identifiable text instance of each text candidate box. According to the second text candidate frame features of each text candidate frame, text detection results such as the "RYA" target frame and the "BAGGAGE" target frame and the text in the target frame are obtained through classification and regression of the text candidate frame and segmentation of the text area in the text candidate frame.

[0117] Based on the training method of the above scene text detection model, after the training is completed, a scene text detection model is obtained.

[0118] Figure 3 A schematic diagram showing a scene text detection model according to some embodiments of the present disclosure.

[0119] like Figure 3 As shown, the scene text detection model includes: modules 310-330.

[0120] The text candidate box generation module 310 is configured to generate a plurality of text candidate boxes based on an image of a natural scene.

[0121] In some embodiments, the text candidate box generation module 310 includes: a feature pyramid network 311 for extracting multi-scale feature maps of an image, and a region generation network 312 for generating multiple text candidate boxes for each feature map.

[0122] The text candidate box feature generating module 320 is configured to generate a second text candidate box feature for a recognizable text instance of each text candidate box.

[0123] In some embodiments, the text candidate box feature generation module 320 includes: units 321-323.

[0124] The first text candidate box feature generating unit 321 is configured to generate a first text candidate box feature for each text candidate box based on the appearance semantics and geometric position of each text candidate box. The first text candidate box feature generating unit 321 is, for example, a candidate box region pooling.

[0125] The second relationship feature generating unit 322 is configured to generate a second relationship feature of identifiable text instances between each text candidate box and other text candidate boxes according to the first text candidate box features of each text candidate box by using the comparative learning result.

[0126] The second text candidate box feature generating unit 323 is configured to fuse the first text candidate box feature and the second relationship feature of each text candidate box to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

[0127] The text detection module 330 is configured to obtain text detection results, including detected text target boxes and text, through classification and regression of text candidate boxes and text area segmentation within the text candidate boxes according to the second text candidate box features of each text candidate box.

[0128] In some embodiments, the text detection module 330 includes: a classifier 331 for performing binary classification of foreground and background on each text candidate box, a regressor 332 for correcting the position and size of each text candidate box, and a text segmentation unit 333 for performing text area segmentation within the text candidate box.

[0129] Based on the detection model obtained by the aforementioned training method, the positioning and combination of text boxes are organically integrated to generate second text candidate box features of recognizable text instances for each text candidate box, and text detection is performed based on these features, thereby improving the text detection performance of the detection model.

[0130] The trained scene text detection model can be used to perform scene text detection.

[0131] Figure 4 A schematic diagram showing a scene text detection method according to some embodiments of the present disclosure.

[0132] like Figure 4 As shown, the scene text detection method includes: at steps 410-430.

[0133] In step 410 , a plurality of text candidate boxes are generated based on an image of a natural scene.

[0134] In the detection using the model, the image of the natural scene may be, for example, an image to be detected of the natural scene, and may be a different image from the image sample used in the model training.

[0135] The text candidate box generation module 310 is used to generate multiple text candidate boxes based on the image of the natural scene, including steps 411-412.

[0136] In step 411 , the feature pyramid network 311 is used to extract a multi-scale feature map of the image.

[0137] In step 412, for each feature map, multiple text candidate boxes are generated through the region generation network 312, specifically including: laying s anchor boxes of fixed size and aspect ratio at each position of the feature map of size H×W, and each feature map corresponds to H×W×s anchor boxes; using a fully convolutional neural network to perform binary classification of foreground and background on each anchor box, and correcting the position and size of each anchor box; retaining a preset number N of anchor boxes with the highest classification scores as output as text candidate boxes.

[0138] At step 420 , a second text candidate box feature of the recognizable text instance of each text candidate box is generated.

[0139] The text candidate box feature generation module 320 is used to generate second text candidate box features of the identifiable text instance of each text candidate box, for example, including steps 421 to 423 .

[0140] In step 421 , the first text candidate box feature generating unit 321 is used to generate a first text candidate box feature for each text candidate box based on the appearance semantics and geometric position of each text candidate box.

[0141] Generating the first text candidate box feature of each text candidate box includes: using candidate box region pooling to extract the first text candidate box feature of each text candidate box, including appearance semantic features and geometric position features.

[0142] In step 422, the second relation feature generating unit 322 uses the comparative learning result to generate second relation features for identifying text instances between each text candidate box and other text candidate boxes according to the first text candidate box features of each text candidate box.

[0143] In step 423, the second text candidate box feature generating unit 323 is used to fuse the first text candidate box feature and the second relationship feature of each text candidate box to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

[0144] In step 430, based on the second text candidate frame features of each text candidate frame, text detection results are obtained through classification and regression of the text candidate frames and segmentation of the text regions within the text candidate frames, including detected text target frames and text.

[0145] The text detection module 330 is used to perform text detection to obtain a text detection result, for example, including steps 431 to 433.

[0146] In step 431, using the classifier 331 to classify the text candidate boxes includes: using the classifier 331 to classify each text candidate box into foreground (ie, text) and background, and inputting the second text candidate box feature of the text candidate box into the classifier 331 to obtain the category of the text candidate box.

[0147] In step 432, regressing the text candidate box using the regressor 332 includes: using the regressor 332 to correct the position and size of each text candidate box, predicting the offset of the center point and aspect ratio of the text candidate box and the text target box, and inputting the second text candidate box feature of the text candidate box into the regressor 332 to obtain the corrected position and size of the text candidate box.

[0148] In step 433, using the text segmentation unit 333 to perform text area segmentation in the text candidate box includes: using the text segmentation unit 333, selecting a preset number (k, such as k=100) of text candidate boxes with the highest classification scores of the classifier as inputs for candidate box area pooling, and obtaining k feature maps of a fixed size (such as 14×14); inputting each feature map into a four-layer fully convolutional neural network to obtain a text separation area of ​​each feature map; mapping the text separation area of ​​the feature map to the text separation area of ​​the text candidate box according to the transformation relationship between the feature map and the text candidate box; and obtaining the text separation area and the corresponding text in the image as a text detection result according to the text separation area of ​​the text candidate box and the position of the text candidate box in the image.

[0149] As mentioned above, the detection model has better text detection performance. Using the detection model for scene text detection can achieve better text detection results.

[0150] Figure 5 A schematic diagram of the structure of a training device for a scene text detection model according to some embodiments of the present disclosure.

[0151] like Figure 5 As shown, the training device 500 for the scene text detection model of this embodiment includes: a memory 510 and a processor 520 coupled to the memory 510, and the processor 520 is configured to execute the training method for the scene text detection model in any of the aforementioned embodiments based on instructions stored in the memory 510.

[0152] The memory 510 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0153] The training device 500 of the scene text detection model may also include an input / output interface 530, a network interface 540, a storage interface 550, etc. These interfaces 530, 540, 550 and the memory 510 and the processor 520 may be connected, for example, via a bus 560. The input / output interface 530 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 540 provides a connection interface for various networked devices. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0154] Figure 6 The present invention is a schematic diagram of the structure of a scene text detection device according to some embodiments of the present invention.

[0155] like Figure 6 As shown, the scene text detection device 600 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the scene text detection method in any of the above embodiments based on the instructions stored in the memory 610. Scene text detection

[0156] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0157] The scene text detection device 600 may also include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650 and the memory 610 and the processor 620 may be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0158] According to business needs, the scene text detection model training device 500 and the scene text detection device 600 can be two independent devices or an integrated device.

[0159] The embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program implements the steps of training a scene text detection model or a scene text detection method.

[0160] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more non-transient computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0161] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A training method for a scene text detection model, characterized in that: include: Generate multiple text candidate boxes based on images of natural scenes; By using the simple relational features between the text candidate frames, by identifying whether the text candidate frames belong to the same text instance, the relationship between the text candidate frames is comparatively learned, wherein the method for determining the simple relational features between the text candidate frames comprises: generating a first text candidate frame feature of each text candidate frame based on the appearance semantics and geometric position of each text candidate frame; generating a first relational feature between each text candidate frame and other text candidate frames as a simple relational feature according to the first text candidate frame features of each text candidate frame; Generate text candidate box features of identifiable text instances of each text candidate box according to the comparative learning results; According to the text candidate box features of the identifiable text instances of each text candidate box, the text detection results are obtained through classification and regression of the text candidate boxes and segmentation of the text regions within the text candidate boxes, including the detected text target boxes and texts; According to the difference between the detected text target box and text and the annotated text target box and text, the total loss is calculated, and the parameters of the scene text detection model are updated according to the total loss.

2. The method according to claim 1, characterized in that The comparative learning of the relationship between the text candidate boxes includes: Based on the supervision of the contrastive learning loss function at the text instance level, the first relation features of each text candidate box are contrastively learned to enhance the relevance of the text candidate boxes belonging to the same text instance, while weakening the relevance of the text candidate boxes belonging to different text instances, so as to generate a second relation feature of the identifiable text instance between each text candidate box and other text candidate boxes.

3. The method according to claim 2, characterized in that The generating of the text candidate box features of the recognizable text instance of each text candidate box includes: The first text candidate box feature and the second relationship feature of each text candidate box are fused to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

4. The method according to claim 1, characterized in that: The calculation of total loss includes: The total loss is calculated based on the set total loss function, combining the gap between the detected text target box and text and the annotated text target box and text. Among them, the total loss function includes the generation loss term, classification loss term and regression loss term of the text candidate box, the text area segmentation loss term within the text candidate box, and the contrastive learning loss term.

5. The method according to claim 2, characterized in that: The contrastive learning loss function includes: contrastive learning loss, association representation of positive sample pairs and association representation of negative sample pairs, wherein the contrastive learning loss is negatively correlated with the association representation of positive sample pairs, and the contrastive learning loss is positively correlated with the association representation of negative sample pairs. Among them, for each annotated text target frame, a first text candidate frame that meets the proximity condition and a second text candidate frame that does not meet the proximity condition relative to the annotated text target frame are determined, the first relationship feature of the annotated text target frame and the first relationship feature of each first text candidate frame constitute a positive sample pair, and the first relationship feature of the annotated text target frame and the first relationship feature of each second text candidate frame constitute a negative sample pair.

6. The method according to claim 5, characterized in that The proximity conditions are: The ratio of the intersection to the union between the text candidate box and the text target box is within a preset low threshold range, and the ratio of the intersection between the text candidate box and the text target box to the text candidate box is within a preset high threshold range.

7. The method according to claim 1, characterized in that The step of generating a first relationship feature between each text candidate box and other text candidate boxes according to the first text candidate box feature of each text candidate box includes: For each text candidate box, the first relationship feature between each text candidate box and the other text candidate boxes is calculated by weighted summation using the similarity between the first text candidate box feature of each text candidate box and the first text candidate box features of other text candidate boxes as a weight.

8. The method according to claim 7, characterized in that The similarity is obtained by performing a dot multiplication of the first text candidate frame feature of each text candidate frame and the first text candidate frame features of other text candidate frames and performing normalization.

9. The method according to claim 1, characterized in that: Generating the first text candidate box feature of each text candidate box includes: Using candidate box region pooling, the first text candidate box features of each text candidate box are extracted, including appearance semantic features and geometric position features.

10. The method according to claim 1, characterized in that The step of generating a plurality of text candidate frames based on the image of the natural scene comprises: Extract multi-scale feature maps of images; For each feature map, multiple text candidate boxes are generated through the region generation network.

11. The method according to claim 10, characterized in that The feature pyramid network is used to extract multi-scale feature maps of the image.

12. The method according to claim 10, characterized in that For each feature map, multiple text candidate boxes are generated through the region generation network, including: Lay s anchor boxes of fixed size and aspect ratio at each position of the feature map of size H×W, and each feature map corresponds to H×W×s anchor boxes; Use a fully convolutional neural network to classify each anchor box into foreground and background, and correct the position and size of each anchor box; A preset number N of anchor boxes with the highest classification scores are retained and output as text candidate boxes.

13. The method according to claim 1, characterized in that The classification and regression of text candidate boxes include: Use the classifier to classify each text candidate box into foreground and background. The regressor is used to correct the position and size of each text candidate box.

14. The method according to claim 13, characterized in that The text area segmentation within the text candidate box includes: Select a preset number of k text candidate boxes with the highest classification scores of the classifier as the input of the candidate box area pooling to obtain k feature maps of fixed size; Input each feature map into the fully convolutional neural network to obtain the text separation region of each feature map; According to the transformation relationship between the feature map and the text candidate box, the text separation area of ​​the feature map is mapped to the text separation area of ​​the text candidate box; According to the text separation area of ​​the text candidate box and the position of the text candidate box in the image, the text separation area and the corresponding text in the image are obtained as the text detection result.

15. A scene text detection method, characterized in that: include: Generate multiple text candidate boxes based on images of natural scenes; Generating text candidate frame features of a recognizable text instance of each text candidate frame, including: generating a first text candidate frame feature of each text candidate frame based on the appearance semantics and geometric position of each text candidate frame; using the contrast learning result, generating a second relationship feature of a recognizable text instance between each text candidate frame and other text candidate frames according to the first text candidate frame features of each text candidate frame; fusing the first text candidate frame feature and the second relationship feature of each text candidate frame to obtain a second text candidate frame feature of the recognizable text instance of each text candidate frame; According to the text candidate box features of the recognizable text instances of each text candidate box, through the classification and regression of the text candidate boxes and the segmentation of the text regions within the text candidate boxes, the text detection results are obtained, including the detected text target boxes and text.

16. The method according to claim 15, characterized in that Generating the first text candidate box feature of each text candidate box includes: Using candidate box region pooling, the first text candidate box features of each text candidate box are extracted, including appearance semantic features and geometric position features.

17. The method according to claim 15, characterized in that The step of generating a plurality of text candidate frames based on the image of the natural scene comprises: Extract multi-scale feature maps of images; For each feature map, multiple text candidate boxes are generated through the region generation network.

18. The method according to claim 17, characterized in that The feature pyramid network is used to extract multi-scale feature maps of the image.

19. The method according to claim 17, characterized in that For each feature map, multiple text candidate boxes are generated through the region generation network, including: Lay s anchor boxes of fixed size and aspect ratio at each position of the feature map of size H×W, and each feature map corresponds to H×W×s anchor boxes; Use a fully convolutional neural network to classify each anchor box into foreground and background, and correct the position and size of each anchor box; A preset number N of anchor boxes with the highest classification scores are retained and output as text candidate boxes.

20. The method according to claim 15, characterized in that The classification and regression of text candidate boxes include: Use the classifier to classify each text candidate box into foreground and background. The regressor is used to correct the position and size of each text candidate box.

21. The method according to claim 20, characterized in that The text area segmentation within the text candidate box includes: Select a preset number of k text candidate boxes with the highest classification scores of the classifier as the input of the candidate box area pooling to obtain k feature maps of fixed size; Input each feature map into the fully convolutional neural network to obtain the text separation region of each feature map; According to the transformation relationship between the feature map and the text candidate box, the text separation area of ​​the feature map is mapped to the text separation area of ​​the text candidate box; According to the text separation area of ​​the text candidate box and the position of the text candidate box in the image, the text separation area and the corresponding text in the image are obtained as the text detection result.

22. A scene text detection model, characterized in that: include: A text candidate box generation module is configured to generate multiple text candidate boxes based on an image of a natural scene; A text candidate box feature generation module is configured to generate text candidate box features of a recognizable text instance of each text candidate box; The text detection module is configured to obtain text detection results, including detected text target boxes and text, by classifying and regressing the text candidate boxes and segmenting the text regions within the text candidate boxes according to the text candidate box features of the recognizable text instances of each text candidate box. The text candidate box feature generation module includes: A first text candidate box feature generating unit is configured to generate a first text candidate box feature of each text candidate box based on the appearance semantics and geometric position of each text candidate box; A second relationship feature generating unit is configured to generate a second relationship feature of a recognizable text instance between each text candidate box and other text candidate boxes according to the first text candidate box features of each text candidate box by using the comparative learning result; The second text candidate box feature generating unit is configured to fuse the first text candidate box feature and the second relationship feature of each text candidate box to obtain the second text candidate box feature of the identifiable text instance of each text candidate box.

23. The model according to claim 22, characterized in that The text candidate box generation module includes: Feature pyramid network, used to extract multi-scale feature maps of images; The region generation network is used to generate multiple text candidate boxes for each feature map.

24. The model according to claim 22, characterized in that The text detection module comprises: The classifier is used to classify each text candidate box into foreground and background. Regressor, used to correct the position and size of each text candidate box; The text segmentation unit is used to segment the text area within the text candidate box.

25. A training device for a scene text detection model, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the training method of the scene text detection model according to any one of claims 1-14 based on instructions stored in the memory.

26. A scene text detection device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the scene text detection method according to any one of claims 15 to 21 based on instructions stored in the memory.

27. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.

Images