Text detection methods, devices, electronic equipment and storage media

Abstract

This application discloses a text detection method, apparatus, electronic device, and storage medium. The method includes: extracting features from a target image to obtain first feature information, and extracting features from an auxiliary image to obtain second feature information; determining a target text detection region to be identified from the target image based on the first feature information and the second feature information; and performing text detection on the target text detection region to obtain a text detection result for the target image. In this way, it is possible to combine the feature information of the auxiliary image and the feature information of the target image to determine the target text detection region to be identified in the target image, thereby obtaining a text detection result determined by combining the auxiliary image.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a text detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In existing technologies, text detection is typically performed on images using trained models to identify text within them.

[0003] However, existing trained models can often only detect text of all trained text types in the dataset. That is, the output text detection results include all text types that the model can detect. When it is necessary to detect one or more specific text types in the image to be detected, it is difficult to directly obtain ideal text detection results. Summary of the Invention

[0004] In view of this, embodiments of this application aim to provide a text detection method, apparatus, electronic device, and storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a text detection method, the method comprising:

[0007] First feature information is obtained by extracting features from the target image, and second feature information is obtained by extracting features from the auxiliary image.

[0008] Based on the first feature information and the second feature information, the target text detection region to be identified is determined from the target image;

[0009] Text detection is performed on the target text detection region to obtain the text detection result of the target image.

[0010] Based on the above scheme, determining the target text detection region to be identified from the target image according to the first feature information and the second feature information includes:

[0011] Based on the first feature information, one or more first text detection regions of the target image are determined;

[0012] Based on the first feature information and the second feature information corresponding to multiple first text detection regions, the target text detection region is selected from the first text detection regions; wherein the text type contained in the target text detection region satisfies a preset similarity condition with the text type contained in the auxiliary image.

[0013] Based on the above scheme, the step of selecting the target text detection region from the first text detection regions according to the first feature information and the second feature information corresponding to the multiple first text detection regions includes:

[0014] A first matching map is generated based on the first feature information and the second feature information; wherein, the first matching map includes matching values;

[0015] When the matching value indicates that the first feature information and the second feature information are successfully matched, the target text detection region is selected from the first text detection region.

[0016] Based on the above scheme, generating the first matching map according to the first feature information and the second feature information includes:

[0017] The second feature information is weighted based on the first weight to generate the first matching graph; wherein the first weight is determined based on the first feature information.

[0018] Based on the above scheme, the step of determining the target text detection region to be identified from the target image according to the first feature information and the second feature information further includes:

[0019] When the matching value indicates that the first feature information and the second feature information fail to match, a second matching image is generated based on the first matching image; wherein, the second matching image includes pixel values ​​transformed based on the matching value;

[0020] Based on the second matching map, determine the second text detection region;

[0021] The target text detection region is determined based on the second text detection region.

[0022] Based on the above scheme, generating the second matching map from the first matching map includes:

[0023] When the nth matching value in the first matching image is greater than or equal to the first threshold, the nth pixel value in the second matching image is determined to be equal to the first value.

[0024] When the nth matching value in the first matching image is less than the first threshold, the nth pixel value in the second matching image is determined to be equal to the second value.

[0025] Based on the above scheme, determining the second text detection region according to the second matching map includes:

[0026] Merge any pixel value in the second matching graph that is equal to the first value into the same first connected region;

[0027] The second text detection region is determined based on the first connected region.

[0028] Secondly, embodiments of this application provide a text detection device, the device comprising:

[0029] The first obtaining module is used to extract features from the target image to obtain first feature information, and to extract features from the auxiliary image to obtain second feature information;

[0030] The determination module is used to determine the target text detection region to be identified from the target image based on the first feature information and the second feature information;

[0031] The second obtaining module is used to perform text detection on the target text detection region to obtain the text detection result of the target image.

[0032] Thirdly, embodiments of this application provide an electronic device, including:

[0033] Memory, which stores computer-readable instructions;

[0034] A processor, connected to the memory, is configured to implement the method provided in the first aspect of the technical solution by executing computer-executable instructions stored in the memory.

[0035] Fourthly, embodiments of this application provide a computer storage medium storing computer-executable instructions, which, when executed, can implement the method provided in the first aspect of the technical solution.

[0036] The text detection method, apparatus, electronic device, and storage medium provided in this application can extract features from a target image to obtain first feature information and extract features from an auxiliary image to obtain second feature information. Based on the first and second feature information, a target text detection region to be identified is determined from the target image. Thus, the target text detection region to be identified can be jointly determined by combining the feature information of the target image and the feature information of the auxiliary image. Text detection is performed on the target text detection region to obtain the text detection result of the target image. In this way, the target text detection region to be identified can be determined based on the feature information of the auxiliary image; that is, by adjusting the auxiliary image, the feature information of the auxiliary image can be adjusted, thereby adjusting the target text detection region to be identified, achieving the desired text detection result based on the auxiliary image. Attached Figure Description

[0037] Figure 1 A flowchart illustrating a text detection method provided in this application embodiment. Figure 1 ;

[0038] Figure 2 A flowchart illustrating a text detection method provided in this application embodiment. Figure 2 ;

[0039] Figure 3 A flowchart illustrating a text detection method provided in this application embodiment. Figure 3 ;

[0040] Figure 4 A flowchart illustrating a method for detecting arbitrary types of text in natural scenes based on few-sample attention, provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the output rectangular text detection box provided in the embodiments of this application;

[0042] Figure 6 This is a schematic diagram of the output quadrilateral text detection box provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the structure of a text detection device provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To gain a more detailed understanding of the features and technical content of this application, the implementation of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit this application.

[0046] like Figure 1 As shown in the figure, this application provides a text detection method, the method comprising:

[0047] Step S110: Perform feature extraction on the target image to obtain first feature information, and perform feature extraction on the auxiliary image to obtain second feature information;

[0048] Step S120: Determine the target text detection region to be identified from the target image based on the first feature information and the second feature information;

[0049] Step S130: Perform text detection on the target text detection region to obtain the text detection result of the target image.

[0050] The target image can be an image that requires text detection. The target image can contain multiple types of text, such as trademarks, document text, signs, graffiti, instant messaging text, and / or license plates.

[0051] The first feature information can be feature information extracted from the target image, such as text feature information and / or image feature information. In the embodiments of this application, the features of the target image can be determined through the first feature information, such as the text type contained in the target image, the color features and / or texture features of the target image, etc.

[0052] The second feature information can be feature information extracted from the auxiliary image, such as text feature information and / or image feature information. In the embodiments of this application, the features of the auxiliary image can be determined through the second feature information, such as the text type contained in the auxiliary image, the color features and / or texture features of the auxiliary image, etc.

[0053] The auxiliary image can be an image that assists in text detection. The auxiliary image can contain multiple types of text, such as trademarks, graffiti, and / or license plates.

[0054] In this embodiment, feature information can be obtained using machine learning models, such as neural network models. For example, the target image can be used as input to a fully convolutional network to extract first feature information, or an auxiliary image can be used as input to a fully convolutional network to extract second feature information. This allows for the improvement of the accuracy and speed of feature information extraction through neural network models.

[0055] The target text detection region can be a region determined based on the target image where text detection is required. The size and / or shape of the target text detection region can be determined based on its corresponding first and second feature information. For example, the region containing text in the target image is determined based on the feature information, and then the size and / or shape of the region containing the text is determined based on the size and / or shape of the region containing the text. In this embodiment, the corresponding target text detection region can be determined based on the text type contained in the dataset trained in the machine learning model, thereby realizing text detection of relevant text types in the target image.

[0056] In this embodiment of the application, the region where the target image contains text can be determined based on the first feature information and the second feature information. Thus, the region where the text is located can be determined as the target text detection region, thereby performing text detection on the text contained in the target image and finally obtaining the text detection result.

[0057] The text detection result can be text information. Specifically, it can include the content, shape, size, and / or orientation of the text. In this embodiment, the text information containing text in the target image can be directly determined through the text detection result, which is convenient and quick.

[0058] In this embodiment, by introducing an auxiliary image and combining its feature information with that of the target image, a target text detection region is selected in the target image for text detection. This achieves the goal of adjusting the text detection of the target image based on the auxiliary image, resulting in high detection efficiency.

[0059] In this embodiment of the application, determining the target text detection region to be identified from the target image based on the first feature information and the second feature information includes:

[0060] Based on the first feature information, one or more first text detection regions of the target image are determined;

[0061] Based on the first feature information and the second feature information corresponding to multiple first text detection regions, the target text detection region is selected from the first text detection regions; wherein the text type contained in the target text detection region satisfies a preset similarity condition with the text type contained in the auxiliary image.

[0062] The preset similarity condition can be that the text type contained in the target text detection region is the same as or similar to the text type contained in the auxiliary image. Different types of text may have certain differences in font, size, arrangement direction, sparsity and / or contrast, while the differences between the same or similar text types are relatively small. Therefore, in this embodiment, a target text detection region that meets the preset similarity condition can be selected in the first text detection region by using an auxiliary image. This allows the input auxiliary image to be adjusted, thereby changing the text type contained in the auxiliary image, and then selecting a target text detection region in the first text detection region that can detect the corresponding text type, thus achieving the detection of text of a specific text type in the target image. In this way, the detection of text of a specific text type in the target image can be automatically completed by inputting an auxiliary image, without the need for manual screening of the text detection results of the target image, thus improving the efficiency of text detection.

[0063] In this embodiment, by introducing an auxiliary image, the feature information of the auxiliary image is compared with the feature information of the target image to select detection regions of text type that are the same as or similar to the auxiliary image, thereby achieving targeted detection of specific text with high detection efficiency. In this embodiment, such as... Figure 2 As shown, selecting the target text detection region from the first text detection regions based on the first feature information and the second feature information corresponding to multiple first text detection regions includes:

[0064] Step S210: Generate a first matching map based on the first feature information and the second feature information; wherein, the first matching map includes matching values;

[0065] Step S220: When the matching value indicates that the first feature information and the second feature information are successfully matched, select the target text detection region from the first text detection region.

[0066] The first matching image can be an image generated based on first feature information and second feature information. For example, it can be generated by performing mathematical operations on the first feature information and the second feature information.

[0067] The matching value can be a numerical value, and its magnitude represents the degree of matching between the first feature information and the second feature information. Specifically, the larger the matching value, the higher the degree of matching between the first feature information and the second feature information corresponding to the first text detection region. That is, the matching value and the degree of matching are positively correlated. For example, the first feature information and the second feature information can be mapped to the same feature space, and then the distance between the vectors of the first feature information and the second feature information in that feature space can be calculated as the matching value. Of course, this is just an example, and the specific implementation is not limited to the above example.

[0068] In this embodiment, a first matching map is generated by processing first feature information and second feature information. The degree of matching between the first feature information and the second feature information is then determined based on the matching values ​​included in the first matching map. When the matching value indicates a successful match between the first feature information and the second feature information, it can be assumed that the text type contained within the first text detection region corresponding to the first feature information in the target image is the same as or similar to the text type contained in the auxiliary image. Therefore, the corresponding first text detection region is determined as the target text detection region. In this way, the target text detection region can be determined based on the text type contained in the auxiliary image, thereby enabling the detection of a specified type of text in the target image.

[0069] In this embodiment of the application, generating a first matching map based on the first feature information and the second feature information includes:

[0070] The second feature information is weighted based on the first weight to generate the first matching graph; wherein the first weight is determined based on the first feature information.

[0071] In this embodiment, the first weight of the weighted operation is determined by the first feature information. The weight value of the first weight can be adjusted according to the importance of the first feature information, thereby adjusting the influence of the first feature information on the weighted operation. This improves the accuracy of the generated first matching map.

[0072] In this embodiment, a first matching map is generated by weighting the second feature information based on a first weight, thereby determining the degree of matching between the target image and the auxiliary image. For example, the second feature information can be used as a convolution kernel, and a sliding convolution operation can be performed on the first feature information to generate the first matching map, thereby determining the degree of matching between the target image and the auxiliary image.

[0073] In the embodiments of this application, such as Figure 3 As shown, determining the target text detection region to be identified from the target image based on the first feature information and the second feature information further includes:

[0074] Step S310: When the matching value indicates that the first feature information and the second feature information fail to match, a second matching image is generated based on the first matching image; wherein, the second matching image includes pixel values ​​converted based on the matching value;

[0075] Step S320: Determine the second text detection region based on the second matching map;

[0076] Step S330: Determine the target text detection region based on the second text detection region.

[0077] In this embodiment, a second matching image and pixel values ​​included in the second matching image are obtained by processing the matching values ​​included in the first matching image. The pixel values ​​can be numerical values, used to indicate whether the first feature information and the second feature information match.

[0078] The second text detection region can be the region that needs to be detected based on the auxiliary image.

[0079] In this embodiment, a second matching map is generated by processing the first matching map, and a second text detection region is generated based on the second matching map. This allows for the determination of the second text detection region based on the degree of matching between the remaining first and second feature information when the matching value indicates a failure to match the first and second feature information corresponding to the first text detection region. The text detection result is then obtained by performing detection based on the second text detection region. Thus, when the target image contains text types not in the trained dataset, and the machine learning model cannot determine the first text detection region containing the corresponding text type based on the first feature information, the second matching map is generated based on the first matching map, and the second text detection region is determined based on the second matching map. This allows for text detection of any text type in the target image using the text types contained in the input auxiliary image, even if the text type to be detected is not in the trained dataset of the machine learning model. This enables text detection of any text type in the target image.

[0080] In this embodiment of the application, generating a second matching map based on the first matching map includes:

[0081] When the nth matching value in the first matching image is greater than or equal to the first threshold, the nth pixel value in the second matching image is determined to be equal to the first value.

[0082] When the nth matching value in the first matching image is less than the first threshold, the nth pixel value in the second matching image is determined to be equal to the second value.

[0083] In this embodiment of the application, the first value and the second value can be different values, and the specific size relationship is not limited.

[0084] In this embodiment, the second matching image is generated based on the matching values ​​of the first matching image, and the second matching image contains only pixel values ​​of two values. For example, the second matching image can be a binary image.

[0085] In this embodiment, by processing the matching value into pixel values, the matching relationship between the first feature information and the second feature information can be represented more intuitively. Specifically, when the pixel value is equal to the first value, it means that the first feature information and the second feature information at that pixel value have successfully matched; when the pixel value is equal to the second value, it means that the first feature information and the second feature information at that pixel value have failed to match.

[0086] In this embodiment of the application, determining the second text detection region based on the second matching map includes:

[0087] Merge any pixel value in the second matching graph that is equal to the first value into the same first connected region;

[0088] The second text detection region is determined based on the first connected region.

[0089] The first connected region can be a region obtained by merging only the regions where pixel values ​​equal to the first value are located.

[0090] In this embodiment of the application, by merging any pixel value equal to the first value into the first connected region, the region where the first feature information and the second feature information are successfully matched can be intuitively displayed, thereby facilitating the determination of the second text detection region.

[0091] In this embodiment of the application, determining the target text detection region based on the second text detection region includes: selecting a second connected region from the first connected region; wherein the shape of the second connected region satisfies a preset condition;

[0092] The target text detection region is determined based on the second connected region.

[0093] The preset condition can be that the shape of the second connected region is a preset shape. Specifically, the preset shape can be a rectangle or a quadrilateral. In the embodiments of this application, the second connected region is generally adopted as a four-neighbor region (the pixel values ​​in the four directions above, below, left, and right of the center pixel value are all equal to the first value), a diagonal neighbor region (the pixel values ​​in the four directions of the center pixel value ... four directions of the center pixel value

[0094] In this embodiment of the application, by selecting a second connected region that meets preset conditions as the target text detection region, the accuracy of the second connected region containing the text to be detected can be improved, thereby further improving the accuracy of text detection.

[0095] In this embodiment of the application, the step of selecting the target text detection region from the first text detection region when the matching value indicates that the first feature information and the second feature information match successfully includes:

[0096] Based on the first feature information, determine the confidence level of the first text detection region;

[0097] The first text detection region with a confidence level greater than the second threshold is determined as the target text detection region.

[0098] The confidence level is used to indicate the probability that the first text detection region contains the text to be detected, and its value can range from 0 to 1. The higher the confidence level, the higher the probability that the first text detection region contains the text to be detected.

[0099] The second threshold can be a value set according to requirements. The higher the second threshold, the higher the probability that the identified target text detection area contains the text to be detected, and the higher the accuracy of the text detection result. In the embodiments of this application, the second threshold can be 0.85 to 0.95, preferably 0.9.

[0100] In this embodiment of the application, the first feature information includes at least one of the following:

[0101] Image feature information, indicating the image features of the target image;

[0102] Text feature information indicates the text features of the target image.

[0103] The image feature information indicates the image features of the target image, including but not limited to color features, shape features, and / or texture features.

[0104] The text feature information indicates the text features of the target image, including but not limited to font, size, arrangement direction, sparsity and / or contrast.

[0105] In this embodiment of the application, the step of extracting features from the auxiliary image to obtain the second feature information includes:

[0106] When at least two auxiliary images exist, the target size of each auxiliary image is determined based on the size information of each auxiliary image;

[0107] Convert the image dimensions of each auxiliary image to the target size;

[0108] The image is obtained by stitching the images together to convert them into at least two auxiliary images of the target size.

[0109] Feature extraction is performed on the input image to obtain the second feature information.

[0110] The size information is used to indicate the size information of the auxiliary image, such as its dimensions. Specifically, the size information can be the length and / or width of the auxiliary image.

[0111] The target size is determined based on the size information of each of the auxiliary images. In this embodiment, to facilitate the integration and stitching of multiple auxiliary images, the target size can be set to the largest size among the auxiliary images. For example, this can be achieved by upsampling auxiliary images with a size smaller than the target size, or by downsampling auxiliary images with a size larger than the target size.

[0112] In this embodiment of the application, when it is necessary to detect text of two or more text types, an input image can be obtained by stitching together multiple auxiliary images including different text types. In this way, the input image can include feature information of multiple different text types, which facilitates the determination of text detection regions, and thus realizes text detection of multiple different text types in the target image.

[0113] Natural scene text images often contain many different types of text. For example, an image may contain trademarks, graffiti, license plates, and other types of text. Existing object detection methods often indiscriminately detect all text in an image or can only detect one fixed type of text (e.g., license plate text detection). In practical applications, the type of text to be detected is sometimes determined based on the user's needs. For example, in a survey of a commercial pedestrian street, only trademarks in the street view image may be identified. Moreover, different texts in natural scenes vary greatly in font, size, arrangement direction, sparsity, and contrast. Existing datasets are difficult to include all types of text. Most text detection methods in related technologies can only detect text types already present in the training set. For text types not present in the training set, the text detection results are often poor. For example, if the training set only contains ordinary street view text types such as trademarks, road signs, and license plates, a model trained on this dataset may perform poorly in detecting artistic fonts or handwritten text.

[0114] Therefore, based on this, this application proposes a method for detecting arbitrary text types in natural scenes based on few-sample attention. The Efficient and Accurate Scene Text (EAST) module is used as the basic text detection module, capable of detecting all trained text types in an image. By introducing a few-sample attention (FSA) module, text detection boxes of the same type as a few auxiliary images can be output independently. That is, the text detection type is specified through auxiliary images, and these auxiliary images can use untrained text types, achieving arbitrary text detection. The overall framework structure diagram is shown below. Figure 4 As shown. Figure 4 In this code, the score map represents the score feature map, the geometry map represents the geometric feature map, the RBOX geometry represents the rotated rectangle geometry, the QUAD geometry represents the quadrilateral geometry, the textboxes represent the text boxes, the text rotation angle represents the rotation angle, the text quadrangle dinates represent the quadrilateral coordinates, the unpool represents the upsampling layer, the conv stage represents the convolutional layer, the concat represents the channel dimension, the slipping convolution represents the sliding convolution, the global average pooling represents the global average pooling layer, and the similarity map represents the similarity prediction map.

[0115] The input image is one of the aforementioned target images; the rotated rectangular geometry and quadrilateral geometry are one of the aforementioned first text detection region, target text detection region, and second text detection region; the fractional feature map is one of the aforementioned confidence level; the similarity prediction map is one of the aforementioned first matching map; and the sliding convolution is one of the aforementioned weighted operations.

[0116] The specific steps for the EAST module are as follows:

[0117] Step 1: The backbone fully convolutional network extracts features from the input image.

[0118] The image is input into a backbone fully convolutional network to extract features. Here, a pre-trained VGG16 model is used as the backbone fully convolutional network, and the output features of four stages (conv stage 1 to conv stage 4) are extracted as backbone features, denoted as f1(x), f2(x), f3(x), and f4(x), respectively. The feature map sizes are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. The features of the input image are one type of the aforementioned first feature information.

[0119] Step 2: Consolidate and merge features along the channel dimension in the upsampling branch.

[0120] First, the f4(x) feature is upsampled through an unpooling layer to obtain an output that is magnified by a factor of 2. Then, the output is concatenated and merged with f3(x) along the channel dimension. The merged feature is then passed through 1*1 and 3*3 convolutional layers (the output feature is denoted as h1). The 1*1 convolutional layer reduces the number of channels to reduce computation, while the 3*3 convolutional layer is used to fuse the merged features. The above steps are repeated to merge the features from f4(x) to f1(x) in sequence, and finally, a 3*3 convolutional layer is passed to output the final merged feature map f(x).

[0121] Step 3: Output basic text detection results

[0122] The EAST module outputs a score map and a geometry map. The score map is obtained by passing f(x) through a 1*1 convolutional layer with 1 channel. The score map represents the confidence score of the detection box corresponding to each pixel. The geometry map has two representations, used to support text detection of two shapes: rotated rectangles (RBOX) and quadrilaterals (QUAD). (See diagram below.) Figure 5 and Figure 6As shown. Since a rotated rectangle (RBOX) can be obtained from the bounding box of a rectangle and the rotation angle, the RBOX geometry map requires passing f(x) through two 1x1 convolutional layers with output channels of 4 and 1. The 4 channels represent the distances of the pixel to the top, right, bottom, and left boundaries of the bounding rectangle, respectively, and the 1 channel represents the rotation angle of the rectangle. For a quadrilateral (QUAD), the coordinates of the four vertices are needed, so f(x) is passed through an 8-channel 1x1 convolutional layer. The 8 channels represent the offsets of the pixel from the coordinates (x, y) of the four vertices. Based on the score map, detection boxes with a confidence score greater than a threshold (set to 0.5 here) are selected. Duplicate detection boxes are merged using the NMS algorithm, thus obtaining the basic text detection results.

[0123] The specific steps for the FSA module are as follows:

[0124] Step 1: Auxiliary Image Preprocessing

[0125] The auxiliary images can be a set of a small number of images {y1, y2, ..., yn}, where n ≥ 1. First, obtain the largest length H and width W of the auxiliary images, and fill all the auxiliary images with the same size H*W (i.e., fill the edges of the images with 0 pixels). Then, stitch all the auxiliary images along the channel dimension to obtain the input image y of the FSA module. When n = 1, the auxiliary images can be directly used as input.

[0126] Step 2: Extract auxiliary image features using a fully convolutional network

[0127] The auxiliary image is input into a fully convolutional network to extract features, yielding f1(y), f2(y), f3(y), and f4(y). This fully convolutional network shares parameters with the backbone fully convolutional network in the EAST module. Then, f1(y) through f4(y) are passed through a global average pooling layer, with outputs denoted as g1(y), g2(y), g3(y), and g4(y). The dimensions of g1(y) through g4(y) are all 1*1, with 64, 128, 256, and 384 channels, respectively. The features of the auxiliary image are one type of the aforementioned second feature information.

[0128] Step 3: Sliding convolution on the input image features

[0129] g of the auxiliary image i (y) serves as the convolution kernel, in the feature f of the input image. i Perform a sliding convolution operation on (x), and denote the output as S. iAccording to the principle of convolution, the larger the response value of a feature pixel through the convolution kernel, the higher the similarity between the input image region corresponding to that pixel and the auxiliary image. Therefore, S i It can be understood as a similarity map between the input image and the auxiliary image.

[0130] Step 4: Merge similarity prediction graphs within the branches

[0131] The steps for merging similarity prediction maps are similar to those in EAST. First, S4 is upsampled through an unpooling layer to obtain an output that is magnified by a factor of 2. Then, this output is concatenated and merged with S3 along the channel dimension. Finally, it is fused through a 3x3 convolutional layer. These steps are repeated to merge S4 through S1 sequentially, resulting in the final similarity prediction map S∈R^. (1*(H / 4)*(W / 4)) S is the similarity prediction map, H is the maximum length of the auxiliary image, W is the maximum width of the auxiliary image, and R is the vector set.

[0132] Step 5: Obtain the detection results based on the similarity prediction map.

[0133] For auxiliary images that are already trained text types, detection boxes are selected based on the similarity prediction map S, using pixel pairs with similarity greater than a threshold t (set to 0.5 here). The NMS algorithm is then used to merge duplicate detection boxes, allowing for rapid acquisition of the detection results for the specified text type in the auxiliary image. The merged detection box represents one of the aforementioned target text detection regions.

[0134] For cases where the auxiliary image is an untrained text type, the similarity prediction map S is first upsampled and enlarged to the same size as the input image. Then, based on a set threshold t, the similarity prediction map is processed into a binary image, where all points in the similarity prediction map greater than the threshold t are set to 1, and those less than the threshold t are set to 0. Next, connected regions in the binary image are marked, and the location of these connected regions represents the predicted position of the specified text type in the input image. This predicted position is one of the aforementioned second text detection regions.

[0135] The loss function is as follows:

[0136] The loss function L is divided into EAST loss L EAST and FSA loss L FSA The two parts, λ, are the hyperparameters of the EAST loss and FSA loss, as shown in the following equation:

[0137] L = L EAST +λL FSA

[0138] EAST loss L EASTThe calculation formula is consistent with that in the EAST model, and includes two parts: the loss of the score map and the loss of the geometry map.

[0139] FSA loss L FSA The Dice loss function is used, and the calculation formula is as follows:

[0140]

[0141] Where i and j are both positive integers, G ij S represents the ground truth map of the text detection in the i-th row and j-th column. ij S is the similarity prediction image for the i-th row and j-th column. H is the maximum length of the auxiliary image, and W is the maximum width of the auxiliary image. G is a binary image that can be generated based on the original text detection labels of the input image. Pixels belonging to the text region are set to 1, and those not belonging to the text region are set to 0. Then, it is downsampled to the same size as S.

[0142] This application proposes a method for detecting arbitrary types of text in natural scenes based on few-shot attention, which has the following advantages:

[0143] (1) Similarity prediction: The features of the auxiliary image used for the specified text type are used as the convolution kernel, and a sliding convolution operation is performed on the features of the input image to obtain the similarity prediction map.

[0144] (2) Obtain the detection results of the specified text: Based on the pixels in the similarity prediction map that are greater than the threshold, select the corresponding detection box in the basic text detection results to quickly obtain the detection results of the specified text type.

[0145] (3) For cases where there are untrained text types in the input images, there is no need to retrain. Only a small number of untrained text type images need to be input. The detection results of the untrained text types can be obtained through similarity prediction maps and a series of post-processing procedures.

[0146] like Figure 7 As shown, a text detection device is provided in an embodiment of this application. The device includes:

[0147] The first obtaining module 110 is used to extract features from the target image to obtain first feature information, and to extract features from the auxiliary image to obtain second feature information;

[0148] The determining module 120 is used to determine the target text detection region to be identified from the target image based on the first feature information and the second feature information;

[0149] The second obtaining module 130 is used to perform text detection on the target text detection region to obtain the text detection result of the target image.

[0150] In some embodiments, the first obtaining module 110, the determining module 120, and the second obtaining module 130 may all be program modules. After being executed by the processor, the program module can realize the functions of the above-mentioned modules.

[0151] In other embodiments, the first obtaining module 110, the determining module 120, and the second obtaining module 130 may all be hardware-software hybrid modules; the hardware-software hybrid modules include, but are not limited to, various programmable arrays; the field-programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0152] In some embodiments, the first obtaining module 110, the determining module 120, and the second obtaining module 130 may all be pure hardware modules; the pure hardware modules include, but are not limited to, application-specific integrated circuits.

[0153] In some embodiments, the determining module 120 is configured to:

[0154] Based on the first feature information, one or more first text detection regions of the target image are determined;

[0155] Based on the first feature information and the second feature information corresponding to multiple first text detection regions, the target text detection region is selected from the first text detection regions; wherein the text type contained in the target text detection region satisfies a preset similarity condition with the text type contained in the auxiliary image.

[0156] In some embodiments, the determining module 120 is configured to:

[0157] A first matching map is generated based on the first feature information and the second feature information; wherein, the first matching map includes matching values;

[0158] When the matching value indicates that the first feature information and the second feature information are successfully matched, the target text detection region is selected from the first text detection region.

[0159] In some embodiments, the determining module 120 is configured to:

[0160] The second feature information is weighted based on the first weight to generate the first matching graph; wherein the first weight is determined based on the first feature information.

[0161] In some embodiments, the determining module 120 is configured to:

[0162] When the matching value indicates that the first feature information and the second feature information fail to match, a second matching image is generated based on the first matching image; wherein, the second matching image includes pixel values ​​transformed based on the matching value;

[0163] Based on the second matching map, determine the second text detection region;

[0164] The target text detection region is determined based on the second text detection region.

[0165] In some embodiments, the determining module 120 is configured to:

[0166] When the nth matching value in the first matching image is greater than or equal to the first threshold, the nth pixel value in the second matching image is determined to be equal to the first value.

[0167] When the nth matching value in the first matching image is less than the first threshold, the nth pixel value in the second matching image is determined to be equal to the second value.

[0168] In some embodiments, the determining module 120 is configured to:

[0169] Merge any pixel value in the second matching graph that is equal to the first value into the same first connected region;

[0170] The second text detection region is determined based on the first connected region.

[0171] In some embodiments, the determining module 120 is configured to:

[0172] A second connected region is selected from the first connected region; wherein the shape of the second connected region satisfies a preset condition;

[0173] The target text detection region is determined based on the second connected region.

[0174] In some embodiments, the determining module 120 is configured to:

[0175] Based on the first feature information, determine the confidence level of the first text detection region;

[0176] The first text detection region with a confidence level greater than the second threshold is determined as the target text detection region.

[0177] In some embodiments, the first feature information includes at least one of the following:

[0178] Image feature information, indicating the image features of the target image;

[0179] Text feature information indicates the text features of the target image.

[0180] In some embodiments, the first receiving module 110 is configured to:

[0181] When at least two auxiliary images exist, the target size of each auxiliary image is determined based on the size information of each auxiliary image;

[0182] Convert the image dimensions of each auxiliary image to the target size;

[0183] The image is obtained by stitching the images together to convert them into at least two auxiliary images of the target size.

[0184] Feature extraction is performed on the input image to obtain the second feature information.

[0185] like Figure 8 As shown in the figure, this application provides an electronic device, which includes:

[0186] Memory, used to store computer-readable instructions;

[0187] A processor, connected to the memory, is configured to implement the methods provided in any of the foregoing embodiments by executing computer-readable instructions, for example, an executable... Figure 1 , Figure 2 , Figure 3 and / or Figure 4 The method shown in the figure.

[0188] The memory can be of various types, such as random access memory, read-only memory, flash memory, etc. The memory can be used for information storage, for example, storing computer-executable instructions. These computer-executable instructions can be various program instructions, such as object program instructions and / or source program instructions.

[0189] The processor can be of various types, such as a central processing unit, microprocessor, digital signal processor, programmable array, application-specific integrated circuit, or image processor. The processor can be connected to the memory via a bus, which can be an integrated circuit bus, etc.

[0190] like Figure 8 As shown, the electronic device may also include a network interface that can be used to interact with peer devices via a network.

[0191] This application also provides a computer storage medium storing computer-executable instructions. When executed, these instructions can implement the methods provided in any of the foregoing embodiments. For example, they can execute... Figure 1 , Figure 2 , Figure 3 and / or Figure 4 The method shown in the figure.

[0192] The computer storage media provided in this embodiment include various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0194] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0195] In addition, each functional unit in the various embodiments of this application can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0196] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text detection method, characterized in that, The method includes: First feature information is obtained by extracting features from the target image, and second feature information is obtained by extracting features from the auxiliary image. Based on the first feature information and the second feature information, a target text detection region to be identified is determined from the target image; wherein the text type contained in the target text detection region satisfies a preset similarity condition with the text type contained in the auxiliary image; the step of determining the target text detection region to be identified from the target image based on the first feature information and the second feature information includes: determining one or more first text detection regions of the target image based on the first feature information; performing a weighted operation on the second feature information based on a first weight to generate a first matching map; the first weight is determined based on the first feature information; the first matching map includes a matching value; when the matching value indicates that the first feature information and the second feature information are successfully matched, the target text detection region is selected from the first text detection region; Text detection is performed on the target text detection region to obtain the text detection result of the target image.

2. The method according to claim 1, characterized in that, The step of determining the target text detection region to be identified from the target image based on the first feature information and the second feature information further includes: When the matching value indicates that the first feature information and the second feature information fail to match, a second matching image is generated based on the first matching image; wherein, the second matching image includes pixel values ​​transformed based on the matching value; Based on the second matching map, determine the second text detection region; The target text detection region is determined based on the second text detection region.

3. The method according to claim 2, characterized in that, The step of generating a second matching map based on the first matching map includes: When the nth matching value in the first matching image is greater than or equal to the first threshold, the nth pixel value in the second matching image is determined to be equal to the first value. When the nth matching value in the first matching image is less than the first threshold, the nth pixel value in the second matching image is determined to be equal to the second value.

4. The method according to claim 3, characterized in that, The step of determining the second text detection region based on the second matching map includes: Merge any pixel value in the second matching graph that is equal to the first value into the same first connected region; The second text detection region is determined based on the first connected region.

5. A text detection device, characterized in that, The device includes: The first obtaining module is used to extract features from the target image to obtain first feature information, and to extract features from the auxiliary image to obtain second feature information; A determining module is configured to determine a target text detection region to be identified from the target image based on the first feature information and the second feature information; wherein the text type contained in the target text detection region satisfies a preset similarity condition with the text type contained in the auxiliary image; the step of determining the target text detection region to be identified from the target image based on the first feature information and the second feature information includes: determining one or more first text detection regions of the target image based on the first feature information; performing a weighted operation on the second feature information based on a first weight to generate a first matching map; the first weight is determined based on the first feature information; the first matching map includes a matching value; when the matching value indicates that the first feature information and the second feature information are successfully matched, selecting the target text detection region from the first text detection region; The second obtaining module is used to perform text detection on the target text detection region to obtain the text detection result of the target image.

6. An electronic device, characterized in that, include: Memory, which stores computer-readable instructions; A processor, connected to the memory, is configured to implement the method of any one of claims 1 to 4 by executing the computer-readable instructions.

7. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions; when the computer-executable instructions are executed by a processor, they can implement the method described in any one of claims 1 to 4.

Images