A training method, device, and storage medium for a scene text detection model

Abstract

The present application discloses a training method, device, and storage medium for a scene text detection model. The training method includes: obtaining a sample image; obtaining a target mask and a center mask of the sample image; fusing the mask of the sample image with the sample image to obtain a sample feature map, where the sample feature map includes a ground-truth target feature map, a predicted target feature map, a ground-truth center feature map, and a predicted center feature map; inputting the predicted target feature map and the ground-truth target feature map into a first discriminator to obtain a first discrimination result, and inputting the predicted center feature map and the ground-truth center feature map into a second discriminator to obtain a second discrimination result; calculating the loss of the generator based on the first discrimination result and the second discrimination result to update and adjust the parameters of the generator, and using the trained generator as the scene text detection model. Through the above method, the present application can improve the accuracy of scene text detection.

Description

Technical Field

[0001] The present application relates to the technical field of text detection, and particularly to a training method, device, and storage medium for a scene text detection model. Background Art

[0002] Scene text detection technology refers to the detection and recognition of text content in common life scenes such as shopping malls, roadsides, or home environments. Traditional scene text detection methods mainly rely on manually extracting and processing features based on connected components and sliding windows. However, due to the arbitrariness and complexity of real scenes, the randomness of text content, fonts, directions, and languages, as well as the influence of various factors such as the brightness change of light and the contrast between text and background in real scenes, the applicability and accuracy of traditional scene text detection methods for text detection in real scenes are very limited. Summary of the Invention

[0003] The present application provides a training method, device, and storage medium for a scene text detection model, which can improve the accuracy of scene text detection.

[0004] To solve the above technical problems, the technical solution adopted by the present application is: providing a training method for a scene text detection model, the training method includes: obtaining a sample image; obtaining an image mask of the sample image, the image mask includes a target mask and a center mask, the target mask carries a target area identifier and a background area identifier, the center mask carries a target center area identifier and a background area identifier, the target mask includes a ground truth target mask and a predicted target mask, the center mask includes a ground truth center mask and a predicted center mask, the predicted target mask and the predicted center mask are obtained by using the generator of a generative adversarial network, the generative adversarial network includes a generator, a first discriminator, and a second discriminator; fusing the image mask with the sample image to obtain a sample feature map, the sample feature map includes a ground truth target feature map, a predicted target feature map, a ground truth center feature map, and a predicted center feature map; inputting the predicted target feature map and the ground truth target feature map into the first discriminator to obtain a first discrimination result, inputting the predicted center feature map and the ground truth center feature map into the second discriminator to obtain a second discrimination result; calculating the loss of the generator based on the first discrimination result and the second discrimination result to update and adjust the parameters of the generator until the training end condition is met, and taking the trained generator as the scene text detection model.

[0005] To solve the above technical problems, another technical solution adopted by this application is: to provide a method for scene text detection, the method comprising: obtaining an image to be processed; using a scene text detection model to perform detection processing on the image to be processed to obtain a detection result, the detection result including a text region image carrying a text region identifier and a background region image carrying a background region identifier; wherein, the scene text detection model is trained by using the training method of the scene text detection model in the above technical solution.

[0006] To solve the above technical problems, another technical solution adopted by this application is: to provide a training device for a scene text detection model, the training device for the scene text detection model including a memory and a processor connected to each other, wherein, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the training method of the scene text detection model in the above technical solution.

[0007] To solve the above technical problems, another technical solution adopted by this application is: to provide a scene text region detection device, the scene text region detection device including a scene text detection model, the scene text detection model being used to implement the scene text detection method in the above technical solution.

[0008] To solve the above technical problems, another technical solution adopted by this application is: to provide a computer-readable storage medium, which is used to store a computer program, and when the computer program is executed by a processor, it is used to implement the training method of the scene text detection model or the scene text detection method in the above technical solution.

[0009] Through the above solution, the beneficial effect of this application is: using a dual discriminator to train the generator, regarding the scene text detection problem as a segmentation problem, extracting the target mask and the center mask in the sample image, using the target mask to segment the text region and the background region, using the center mask to segment the text center region and the background region, so as to train the generator using both the text region feature and the text center region feature, thereby distinguishing different text regions through the text center region feature, avoiding the situation of misidentifying different text regions that are adhered to each other and the same text regions that are far apart in the sample image, solving the problem that sparse arranged text is easily isolated, and greatly improving the detection accuracy of the scene text detection model. Description of the Drawings

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Among them:

[0011] Figure 1 It is a schematic flowchart of an embodiment of the training method of the scene text detection model provided by this application;

[0012] Figure 2 It is a schematic flowchart of another embodiment of the training method of the scene text detection model provided by this application;

[0013] Figure 3 It is a schematic flowchart of an embodiment of obtaining the ground truth center mask and the predicted center mask provided by this application;

[0014] Figure 4(a) is a schematic diagram of the text region border in the detection result provided by this application;

[0015] Figure 4(b) is a schematic diagram of the text region border in the ground truth result;

[0016] Figure 5 It is a schematic structural diagram of an embodiment of the GAN network provided by this application;

[0017] Figure 6 It is a schematic network structure diagram of an embodiment of the generator provided by this application;

[0018] Figure 7 It is a schematic network structure diagram of the CBM module provided by this application;

[0019] Figure 8 It is a schematic network structure diagram of the fully convolutional module provided by this application;

[0020] Figure 9 It is a schematic network structure diagram of an embodiment of the discriminator provided by this application;

[0021] Figure 10 It is a schematic flowchart of an embodiment of the scene text detection method provided by this application;

[0022] Figure 11 It is a schematic structural diagram of an embodiment of the training device of the scene text detection model provided by this application;

[0023] Figure 12 It is a schematic structural diagram of an embodiment of the scene text region detection device provided by this application;

[0024] Figure 13 It is a schematic structural diagram of an embodiment of the computer-readable storage medium provided by this application. Detailed implementation manners

[0025] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be specifically noted that the following embodiments are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following embodiments are only partial embodiments of the present application rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0026] When the term "embodiment" is mentioned in the present application, it means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0027] It should be noted that the terms "first", "second", and "third" in the present application are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", and "third" may explicitly or implicitly include at least one of such features. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0028] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an embodiment of the training method of the scene text detection model provided by the present application. The method includes:

[0029] Step 11: Obtain a sample image.

[0030] The sample image can be obtained from the training sample library. The sample image can be a scene text image, and the scene text image is an image including text region information and background region information, so as to train the scene text detection model using the sample image. Specifically, the training sample library can be the CTW1500 public dataset, which mainly includes text images containing languages such as Chinese, Japanese, Korean or English sorted out by network search. Among them, the public dataset contains 1000 training set images and 500 test set images, which respectively contain various text images in different directions and sizes.

[0031] Step 12: Obtain the image mask of the sample image.

[0032] The image mask may include a target mask and a center mask. The target may be a text target. The mask is used to represent the region division in the sample image. The target mask carries a target region identifier and a background region identifier. The target region identifier and the background region identifier are respectively used to identify the target region image (i.e., the text region image) and the background region image (i.e., the region other than the text region image) in the sample image. The target mask may include a ground-truth target mask and a predicted target mask. The center mask may carry a target center region identifier and a background region identifier. The target center region identifier and the background region identifier are respectively used to identify the target center region image (i.e., the text center region image) and the background region image (i.e., the region other than the text center region image) in the sample image. The center mask may include a ground-truth center mask and a predicted center mask. By using the center mask, the center position of each text region in the sample image can be identified, so as to accurately distinguish different text regions, avoid the problem of different text regions sticking together, and at the same time can accurately divide the same text region to improve the accuracy of scene text detection. Among them, the target region identifier, the target center region identifier, and the background region identifier may be binary data "0" and "1" in the binary map mask. For example, if the target region identifier is "1", the corresponding background region identifier is "0". The boundary between the target region identifier and the background region identifier represents the text region border. The target center region identifier and the background region identifier are the same and will not be elaborated here.

[0033] The ground-truth target mask may be a manually set label mask. The ground-truth target mask can be obtained according to the ground-truth label set manually for the sample image. The predicted target mask and the predicted center mask can be obtained by using the generator in the Generative Adversarial Networks (GAN). The GAN adopted in this embodiment may include a generator, a first discriminator, and a second discriminator. Specifically, the sample image is input into the generator. The generator can extract features through a backbone network composed of a traditional classification network, and then use the generator to output an image mask with the same size as the input sample image through deconvolution for the extracted image features, and then perform segmentation processing (clip) on the generated image mask to obtain the predicted target mask and the predicted center mask.

[0034] Step 13: Fuse the image mask with the sample image to obtain a sample feature map.

[0035] The task of the discriminator is to enable the image mask generated by the generator to reflect the actual text positions in the sample image. Before using the discriminator for feature discrimination, the image mask can be fused with the original sample image to obtain a sample feature map, and then the feature map is input into the discriminator. The sample feature map can include a ground-truth target feature map, a predicted target feature map, a ground-truth center feature map, and a predicted center feature map. That is to say, the predicted target mask, the ground-truth target mask, the predicted center mask, and the ground-truth center mask are respectively fused with the sample image, and then the corresponding predicted target feature map, ground-truth target feature map, predicted center feature map, and ground-truth center feature map are obtained respectively. Specifically, the ground-truth target feature map can include the target region image features and background region image features under the ground-truth label, the predicted target feature map can include the target region image features and background region image features obtained by the generator, the ground-truth center feature map can include the target center region image features and background center region image features under the ground-truth label, and the predicted center feature map can include the target center region image features and background center region image features obtained by the generator.

[0036] Step 14: Input the predicted target feature map and the ground-truth target feature map into the first discriminator to obtain a first discrimination result, and input the predicted center feature map and the ground-truth center feature map into the second discriminator to obtain a second discrimination result.

[0037] Input the predicted target feature map and the ground-truth target feature map into the first discriminator, and use the first discriminator to compare the predicted target feature map and the ground-truth target feature map to obtain a first discrimination result. Input the predicted center feature map and the ground-truth center feature map into the second discriminator, and use the second discriminator to compare the predicted center feature map and the ground-truth center feature map to obtain a second discrimination result. Specifically, the discriminator can distinguish real data (i.e., the ground-truth target feature map and the ground-truth center feature map obtained according to the ground-truth label) and fake data generated by the generator (i.e., the predicted target feature map and the predicted center feature map predicted by the generator). The first discrimination result can include the probability value that the predicted target feature map is real data and the probability value that the ground-truth target feature map is real data. The second discrimination result can include the probability value that the predicted center feature map is real data and the probability value that the ground-truth center feature map is real data. Generally speaking, the numerical range of the probability value can be [0, 1]. When the probability value is close to 0, it indicates that the feature map is fake data. When the probability value is close to 1, it indicates that the feature map is ground-truth data.

[0038] Step 15: Calculate the loss of the generator based on the first discrimination result and the second discrimination result to update and adjust the parameters of the generator until the training end condition is met, and use the trained generator as the scene text detection model.

[0039] Calculate the loss of the generator based on the first discrimination result and the second discrimination result to update and adjust the parameters of the generator until the training end condition is met. Then, use the trained generator as the scene text detection model to detect the scene text in the image to be detected.

[0040] Specifically, the training end condition may include: loss convergence, that is, the difference between the previous loss and the current loss value is less than a set value; determining whether the current loss value is less than a preset loss, where the preset loss is a pre-set loss threshold. If the current loss value is less than the preset loss, it is determined that the preset stop condition is reached; the number of training times reaches a set value (for example: training 10,000 times); or the accuracy rate obtained when using the test set for testing reaches a set condition (such as: exceeding the preset accuracy rate), etc.

[0041] This embodiment can use a dual discriminator to train the generator, regard the scene text detection problem as a segmentation problem, extract the target mask and the center mask in the sample image, use the target mask to segment the text region and the background region, and use the center mask to segment the text center region and the background region, so as to train the generator simultaneously using the text region feature and the text center region feature. Thus, different text regions can be distinguished through the text center region feature, avoiding misidentifying different text regions that are adhered to each other and the same text regions that are far apart in the sample image, solving the problem that sparsely arranged text is easily isolated, and greatly improving the detection accuracy of the scene text detection model.

[0042] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another embodiment of the training method of the scene text detection model provided by this application. The method includes:

[0043] Step 21: Obtain a sample image.

[0044] Step 21 is the same as Step 11 in the above embodiment and will not be elaborated here.

[0045] Step 22: Obtain the image mask of the sample image.

[0046] The image mask includes a target mask and a center mask. The target mask carries the target area identifier and the background area identifier, and the center mask carries the target center area identifier and the background area identifier. The target mask includes a ground truth target mask and a predicted target mask, and the center mask includes a ground truth center mask and a predicted center mask. The predicted target mask and the predicted center mask are obtained by using the generator of the generative adversarial network. The generative adversarial network includes a generator, a first discriminator, and a second discriminator.

[0047] Specifically, the steps to obtain the ground-truth target mask may include: obtaining the ground-truth label, and then identifying the target region and the background region in the sample image based on the ground-truth label, so as to obtain the ground-truth target mask. Among them, the ground-truth label may include the target position information in the sample image, and the target position information may be the point coordinates of the target or the bounding box coordinates of the target region, etc. The ground-truth label may be a label file in the manually set txt text format or a label file in other data formats, which is not limited here.

[0048] Further, the steps to obtain the corresponding predicted center mask and ground-truth center mask may include: processing the predicted target mask and the ground-truth target mask to obtain the corresponding predicted center mask and ground-truth center mask. In a specific embodiment, as Figure 3 shown, the steps to obtain the ground-truth center mask and the predicted center mask may include the following steps 31-33:

[0049] Step 31: Split the ground-truth target mask and the predicted target mask respectively to obtain a first ground-truth target mask and a first predicted target mask containing the target region identifier, and a second ground-truth target mask and a second predicted target mask containing the background region identifier.

[0050] First, after the feature extraction of the sample image by the generator, an output result of size [B, 4, W, H] can be obtained. The output result may be the predicted target mask and the predicted center mask. Among them, B represents the number of sample images (batch size), W and H respectively represent the width and height of the sample image, and 4 represents the number of channels, that is, the output result of the generator is four-channel data. The predicted target mask corresponds to the first two-channel data, that is, the coordinate information corresponding to the binary data in the first two channels of the output result constitutes the predicted target mask. For example, the coordinate positions identified by the binary data "1" can form the target region, and the coordinate positions identified by the binary data "0" can form the background region. The predicted center mask corresponds to the last two-channel data, that is, the coordinate information corresponding to the binary data in the last two channels of the output result constitutes the predicted center mask. For example, the coordinate positions identified by the binary data "1" can form the target center region, and the coordinate positions identified by the binary data "0" can form the background region. Similarly, the ground-truth target mask and the ground-truth center mask are also four-channel data, which will not be elaborated here.

[0051] Then, in the process of processing the predicted target mask and the ground-truth target mask to obtain the ground-truth center mask and the predicted center mask, the ground-truth target mask and the predicted target mask can be split respectively to obtain a first ground-truth target mask and a first predicted target mask containing the target region identifier, and a second ground-truth target mask and a second predicted target mask containing the background region identifier.

[0052] Step 32: Reduce the sizes of the first true target mask and the first predicted target mask to obtain a third true target mask and a third predicted target mask.

[0053] Reducing the sizes of the first true target mask and the first predicted target mask that contain target region identifiers can simultaneously reduce the centers of each target in each target region, thereby obtaining the target center regions corresponding to each target region, and thus obtaining a third true target mask and a third predicted target mask that contain target center region identifiers; specifically, the sizes of the first true target mask and the first predicted target mask can be reduced by 30% to obtain the target center regions.

[0054] Step 33: Superimpose the third true target mask and the second true target mask to obtain a true center mask, and superimpose the third predicted target mask and the second predicted target mask to obtain a predicted center mask.

[0055] By only reducing the sizes of the target regions and not changing the sizes of the background regions, the originally connected target regions can be separated, and then superimposing the third true target mask and the second true target mask can obtain the true center mask, and superimposing the third predicted target mask and the second predicted target mask can obtain the predicted center mask.

[0056] In another specific embodiment, the predicted target mask and the true target mask may not be split. The target regions identified by the predicted target mask and the true target mask can be obtained first, and then the target regions are reduced to obtain the target center regions, so as to obtain a predicted center mask and a true center mask with target center region identifiers.

[0057] An image mask is obtained through the above steps, and then the image mask is fused with the sample image to obtain a sample feature map, and then the sample feature map is input into the discriminator. The sample feature map includes a true target feature map, a predicted target feature map, a true center feature map, and a predicted center feature map. The content of obtaining the sample feature map is described in the following steps 23 to 27:

[0058] Step 23: Split the image mask to obtain a first sample mask containing target identifiers and a second sample mask containing background region identifiers.

[0059] The target identifier may include a target area identifier or a target central area identifier, that is, the target mask and the central mask can be split respectively. The target mask is split into a first sample mask including the target area identifier and a second sample mask including the background area identifier, and the central mask is split into a first sample mask including the target central area identifier and a second sample mask including the background area identifier. Understandably, the target mask also includes a true target mask and a predicted target mask, and the central mask also includes a true central mask and a predicted central mask. Then, in the step of fusing the image mask with the sample image to obtain the sample feature map, the true target mask, the predicted target mask, the true central mask, and the predicted central mask are respectively subjected to the same fusion operation with the sample image to obtain their respective corresponding sample feature maps, that is, the true target feature map, the predicted target feature map, the true central feature map, and the predicted central feature map. For the convenience of description, in this embodiment, the "image mask" is used as an example for illustration, and the fusion operations performed on each mask are not described one by one.

[0060] Step 24: Split the sample image into channels to obtain a first-channel image, a second-channel image, and a third-channel image.

[0061] Taking the sample image in RGB format as an example, the size of the sample image can be [B, 3, W, H], that is, the sample image can contain three-channel data of R, G, and B. The sample image can be split into channels to obtain a first-channel image, a second-channel image, and a third-channel image respectively, that is, the R-channel image (red image), the G-channel image (green image), and the B-channel image (blue image).

[0062] Step 25: Respectively fuse the first sample mask with the first-channel image, the second-channel image, and the third-channel image to obtain a first feature image, a second feature image, and a third feature image, and stack the first feature image, the second feature image, and the third feature image based on the channel order to obtain a first sample feature image.

[0063] Respectively fusing the first sample mask containing the target identifier with the first-channel image, the second-channel image, and the third-channel image can obtain the distribution of the target area in different-channel images, obtain a first feature image, a second feature image, and a third feature image, and then stack the first feature image, the second feature image, and the third feature image in the channel order of R, G, and B to obtain the first sample feature image.

[0064] Step 26: Respectively fuse the second sample mask with the first-channel image, the second-channel image, and the third-channel image to obtain a fourth feature image, a fifth feature image, and a sixth feature image, and stack the fourth feature image, the fifth feature image, and the sixth feature image based on the channel order to obtain a second sample feature image.

[0065] Fusing the second sample mask containing the background region identifier with the first channel image, the second channel image, and the third channel image respectively can obtain the distribution of the background region in different channel images, resulting in the fourth feature image, the fifth feature image, and the sixth feature image. Then, in the channel order of R, G, and B, the fourth feature image, the fifth feature image, and the sixth feature image are superimposed to obtain the second sample feature image.

[0066] Step 27: Superimpose the first sample feature image and the second sample feature image to obtain the sample feature map.

[0067] By superimposing the first sample feature image and the second sample feature image, the sample feature map generated after the image mask and the sample image are fused can be obtained.

[0068] Step 28: Input the predicted target feature map and the ground truth target feature map into the first discriminator to obtain the first discrimination result, and input the predicted center feature map and the ground truth center feature map into the second discriminator to obtain the second discrimination result.

[0069] The first discriminator is used to compare the predicted target feature map and the ground truth target feature map. Inputting the predicted target feature map and the ground truth target feature map into the first discriminator can obtain the first discrimination result. Among them, the first discrimination result may include the predicted target probability value and the ground truth target probability value. The predicted target probability value is the probability that the predicted target feature map is true (i.e., real data), and the ground truth target probability value is the probability that the ground truth target feature map is true; the second discriminator is used to compare the predicted center feature map and the ground truth center feature map. Inputting the predicted center feature map and the ground truth center feature map into the second discriminator can obtain the second discrimination result. Among them, the second discrimination result may include the predicted center probability value and the ground truth center probability value. The predicted center probability value is the probability that the predicted center feature map is true, and the ground truth center probability value is the probability that the ground truth center feature map is true.

[0070] Step 29: Calculate the loss of the generator based on the first discrimination result and the second discrimination result to update and adjust the parameters of the generator until the training end condition is met, and use the trained generator as the scene text detection model.

[0071] In a specific implementation, the first discrimination loss and the second discrimination loss can be obtained first, and then the loss of the generator can be calculated by fusing the first discrimination loss and the second discrimination loss; specifically, the first discrimination loss may be the loss of the first discriminator for discriminating that the predicted target feature map is true. The first discrimination loss is the cross-entropy loss between the predicted target probability value and 1. The second discrimination loss is the loss of the second discriminator for discriminating that the predicted center feature map is true. The second discrimination loss is the cross-entropy loss between the predicted center probability value and 1.

[0072] Furthermore, the loss of the generator may also include a first prediction loss and a second prediction loss. The first prediction loss and the second prediction loss can be obtained, and then the first prediction loss, the second prediction loss, the first discriminant loss, and the second discriminant loss are fused to calculate the loss of the generator. Among them, the first prediction loss is the cross-entropy loss between the predicted target mask and the true target mask, and the second prediction loss is the cross-entropy loss between the predicted center mask and the true center mask.

[0073] The calculation formula for the loss of the generator can be shown as the following formula (1):

[0074] L G = L CE (D2(G(I C )), 1) + L CE (D1(G(I)), 1) + L CE (G(I), T) + L CE (G(I), T C ) Formula (1)

[0075] In the above formula (1), L CE represents the cross-entropy loss, L G represents the loss of the generator, G(I) represents the predicted target feature map, G(I C ) represents the predicted center feature map, D2(G(I C )) represents the predicted center probability value, D1(G(I)) represents the predicted target probability value, T represents the true target feature map, and T C represents the true center feature map.

[0076] It can be understood that in the above embodiments, it is default to use the trained first discriminator and second discriminator to train the generator. In a specific implementation, the first discriminator and the second discriminator can also be trained first, and then after obtaining the trained first discriminator and second discriminator, the generator is trained, so as to obtain a more adaptable first discriminator and second discriminator to improve the training effect of the generator. Specifically, the loss of the first discriminator can be calculated based on the first discrimination result, and the loss of the second discriminator can be calculated based on the second discrimination result to update and adjust the parameters of the first discriminator and the second discriminator until the training end condition is met, and the parameters of the first discriminator and the second discriminator are fixed.

[0077] First, the third discriminant loss, the fourth discriminant loss, the fifth discriminant loss, and the sixth discriminant loss can be obtained, and then the third discriminant loss and the fourth discriminant loss are fused to obtain the loss of the first discriminator, and the fifth discriminant loss and the sixth discriminant loss are fused to obtain the loss of the second discriminator; specifically, the third discriminant loss is the loss of the first discriminator judging that the predicted target feature map is false (false data), the third discriminant loss is the cross-entropy loss between the predicted target probability value and 0, the fourth discriminant loss is the loss of the first discriminator judging that the true target feature map is true, the fourth discriminant loss is the cross-entropy loss between the true target probability value and 1, the fifth discriminant loss is the loss of the second discriminator judging that the predicted center feature map is false, the fifth discriminant loss is the cross-entropy loss between the predicted center probability value and 0, the sixth discriminant loss is the loss of the second discriminator judging that the true center feature map is true, and the sixth discriminant loss is the cross-entropy loss between the true center probability value and 1.

[0078] The calculation formula for the loss of the first discriminator is shown in Equation (2) below, and the calculation formula for the loss of the second discriminator is shown in Equation (3) below:

[0079] L D1 =L CE (D1(G(I)),0)+L CE (D1(T),1) Equation (2)

[0080] L D2 =L CE (D2(G(I C )),0)+L CE (D2(T C ),1) Equation (3)

[0081] In the above Equation (2), L CE represents the cross-entropy loss, L D1 represents the loss of the first discriminator, L D2 represents the loss of the second discriminator, G(I) represents the predicted target feature map, G(I C ) represents the predicted center feature map, T represents the true target feature map, T C represents the true center feature map, D2(G(I C )) represents the predicted center probability value, D1(G(I)) represents the predicted target probability value, D2(T C ) represents the true center probability value, D1(T) represents the true target probability value.

[0082] In a specific implementation scenario, starting from the input of the first sample image to the generator, first, all parameters in the generator are fixed. Then, according to the above formulas (2) and (3), the loss of the discriminator is calculated. Next, backpropagation is performed on the loss of the discriminator to update the parameters of the discriminator, thereby training the discriminator until the training end condition is met. Then, the parameters of the discriminator are fixed, and then the generator is trained. According to the above formula (1), the loss of the generator is calculated, and backpropagation is performed on the loss of the generator to update the parameters of the generator, thereby training the generator. This process is repeated until the entire training cycle (epoch) is completed, that is, all sample images in the training sample set have participated in one training of the generator and the discriminator. This is repeated 600 epochs, and the parameters of the generator are fixed to complete the training of the generator.

[0083] Specifically, at the beginning of the first epoch, the Adam optimizer can be used to set the initial learning rates of both the generator and the discriminator to 0.0001. Among them, the larger the learning rate, the greater the slope of the parameter adjustment of the generator and the discriminator, that is, the smaller the adjustment accuracy. After every 100 epochs of training, the learning rates of the generator and the discriminator can be adjusted to 1 / 10 of the current learning rate to gradually reduce the learning rate and the slope of the parameter adjustment of the generator and the discriminator, thereby gradually improving the parameter adjustment accuracy as the training progresses, and thus improving the training effect. After 600 epochs of training are completed, the trained generator is used as the scene text detection model, so that the sample images in the training sample set can be directly input into the scene text detection model, and the detection target mask is output. Then, the detection target mask and the sample image are merged to obtain the final detection result. It can be understood that the initial learning rate, the learning rate reduction amplitude, and the training end condition can all be set according to the actual situation and are not limited here.

[0084] As shown in Figure 4(a), Figure 4(a) uses the text region border to indicate. In the actual detection result, the text region within the text region border is represented by the text region identifier (such as "1"), and the background region outside the text region border is represented by the background region identifier (such as "0"). Compared with the ground truth result shown in Figure 4(b), the text region recognition of the detection result obtained by using the scene text detection model is more accurate, and it can detect text regions with complex shapes.

[0085] This embodiment can regard the scene text detection problem as a segmentation problem, directly perform scene text detection by generating a mask through a GAN network with a dual discriminator, separate the text region and the background region in the sample image in the form of a mask to obtain an image mask, and then fuse the image mask with the sample image to obtain a sample feature map containing the features of the text region and the background region. Then, use the dual discriminator to discriminate the feature map containing the text region features and the feature map containing the text center region features respectively, and use the discrimination results of the dual discriminator to train the generator to obtain a scene text detection model, greatly improving the accuracy of scene text detection, solving the problem of difficult detection of text boxes with complex shapes, and also solving the problem that sparsely arranged text is easily isolated.

[0086] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of an embodiment of the GAN network provided by this application. The GAN network is used to implement the training method of the scene text detection model in the above embodiment.

[0087] In a specific implementation manner, the generator may include a backbone network (for example: VGG16 network) and a deconvolution layer. The backbone network is used to extract image features, and the deconvolution layer is used to restore the feature map to the original size of the sample image. As Figure 6 shown, the backbone network includes two 3*3 convolutional layers 1-2, a 2*2 max-pooling layer (Maxpool), four convolutional modules 1-4 (Conv-BatchNorm-Maxpool, CBM), and a fully convolutional module. Among them, the network structure of the CBM module is as Figure 7 shown. The CBM module includes three 3*3 convolutional layers 1-3, batch normalization layers (Batch Normalization, BN) 1-3, and a 2*2 Maxpool 1. The network structure of the fully convolutional module is as Figure 8 shown. The fully convolutional module includes three 3*3 convolutional layers 1-3 and two dropout layers 1-2.

[0088] Specifically, the size of the sample image input to the generator can be set to 640*1024, and a deconvolution layer with a size of 32*32 and a stride of 32 is set. Since each Maxpool and CBM module in the backbone network of the generator can reduce the size of the image by half (that is, both the length and width are reduced by half) during the feature extraction process, the size of the feature map after passing through the backbone network is reduced to 1 / 32 of the sample image. Then, the feature map is input to the deconvolution layer so that the deconvolution layer expands the size of the feature map by 32 times, thereby obtaining a feature map with the same size as the sample image. That is, through Figure 6The generator shown can output a predicted target mask and a predicted center mask with a size of 640*1024.

[0089] Further, the discriminator (including the first discriminator and the second discriminator) may include a fully convolutional network (FCN). The fully convolutional network may include multiple convolutional modules, such as Figure 9 As shown, the fully convolutional network may include three 3*3 convolutional modules 1-3, three 1*1 convolutional modules 1-3, two Maxpool 1-2, and one average pooling (Avgpool). The network structure of each convolutional module is as Figure 7 shown and will not be elaborated here. Among them, the stride of Maxpool 1-2 can be set to 2, and the kernel size of Maxpool 1-2 can be set to 2, so as to reduce the size of the feature map by 4 times. Then, the window size of Avgpool can be set to (W / 4, H / 4), which can expand the width and height of the feature map by 4 times again, so that the size of the feature map remains unchanged after being input into the discriminator.

[0090] In the network structure of the GAN network set in this embodiment, no target detection network structure is required, which can save costs, reduce resource occupation, and make detection more efficient.

[0091] Please refer to Figure 10 , Figure 10 which is a schematic flowchart of an embodiment of the scene text detection method provided by this application. The scene text detection method includes:

[0092] Step 101: Obtain the image to be processed.

[0093] Step 102: Use the scene text detection model to detect and process the image to be processed to obtain a detection result.

[0094] After obtaining the image to be processed, the scene text detection model can be used to detect and process the image to be processed to obtain a detection result. Specifically, the detection result includes a text region image carrying a text region identifier and a background region image carrying a background region identifier. Among them, the scene text detection model is trained by using the training method of the scene text detection model in the above embodiment.

[0095] This embodiment can use the scene text detection model to detect the scene text in the image to be processed, greatly improving the accuracy and efficiency of scene text detection.

[0096] Please refer to Figure 11 , Figure 11FIG. 0 is a schematic structural diagram of an embodiment of a training device for a scene text detection model provided by the present application. The training device 110 of the scene text detection model includes a memory 111 and a processor 112 connected to each other. The memory 111 is used to store a computer program, and when the computer program is executed by the processor 112, it is used to implement the training method of the scene text detection model in the above embodiment.

[0097] Please refer to Figure 12 , Figure 12 FIG. 7 is a schematic structural diagram of an embodiment of a scene text area detection device provided by the present application. The scene text area detection device 120 includes a scene text detection model 121, and the scene text detection model 121 is used to implement the scene text detection method in the above embodiment.

[0098] Please refer to Figure 13 , Figure 13 FIG. 14 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 130 is used to store a computer program 131, and when the computer program 131 is executed by a processor, it is used to implement the training method of the scene text detection model or the scene text detection method in the above embodiment.

[0099] The computer-readable storage medium 130 may be various media that can store program codes, such as a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc.

[0100] In several embodiments provided by the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0103] The above are only embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A training method for a scene text detection model, characterized in that, Including: Obtain a sample image; Obtain an image mask of the sample image, the image mask including a target mask and a center mask, the target mask carrying a target area identifier and a background area identifier, the center mask carrying a target center area identifier and a background area identifier, the target mask including a ground-truth target mask and a predicted target mask, the center mask including a ground-truth center mask and a predicted center mask, the predicted target mask and the predicted center mask being obtained by using a generator of a generative adversarial network, the generative adversarial network including the generator, a first discriminator, and a second discriminator; Fuse the image mask with the sample image to obtain a sample feature map, the sample feature map including a ground-truth target feature map, a predicted target feature map, a ground-truth center feature map, and a predicted center feature map; Input the predicted target feature map and the ground-truth target feature map into the first discriminator to obtain a first discrimination result, and input the predicted center feature map and the ground-truth center feature map into the second discriminator to obtain a second discrimination result; Calculate the loss of the generator based on the first discrimination result and the second discrimination result, so as to update and adjust the parameters of the generator until the training end condition is satisfied, and use the trained generator as a scene text detection model.

2. The training method of the scene text detection model according to claim 1, wherein The first discrimination result includes a predicted target probability value, the predicted target probability value being the probability that the predicted target feature map is true, the second discrimination result includes a predicted center probability value, the predicted center probability value being the probability that the predicted center feature map is true; The step of calculating the loss of the generator based on the first discrimination result and the second discrimination result includes: Obtain a first discrimination loss and a second discrimination loss, the first discrimination loss being the loss of the first discriminator discriminating that the predicted target feature map is true, the first discrimination loss being the cross-entropy loss between the predicted target probability value and 1, the second discrimination loss being the loss of the second discriminator discriminating that the predicted center feature map is true, the second discrimination loss being the cross-entropy loss between the predicted center probability value and 1; Fuse the first discrimination loss and the second discrimination loss to calculate the loss of the generator.

3. The training method of the scene text detection model according to claim 2, characterized in that, The method further includes: Obtain a first prediction loss and a second prediction loss, the first prediction loss being the cross-entropy loss between the predicted target mask and the ground-truth target mask, the second prediction loss being the cross-entropy loss between the predicted center mask and the ground-truth center mask; Fuse the first prediction loss, the second prediction loss, the first discrimination loss, and the second discrimination loss to calculate the loss of the generator.

4. The training method of the scene text detection model according to claim 2, wherein Before the step of calculating the loss of the generator based on the first discrimination result and the second discrimination result, including: Calculate the loss of the first discriminator based on the first discrimination result, and calculate the loss of the second discriminator based on the second discrimination result, so as to update and adjust the parameters of the first discriminator and the second discriminator until the training end condition is met, and fix the parameters of the first discriminator and the second discriminator.

5. The training method of the scene text detection model according to claim 4, characterized in that The first discrimination result further includes a true target probability value, where the true target probability value is the probability that the true target feature map is true, and the second discrimination result further includes a true center probability value, where the true center probability value is the probability that the true center feature map is true; The step of calculating the loss of the first discriminator based on the first discrimination result and calculating the loss of the second discriminator based on the second discrimination result includes: Fuse the third discrimination loss and the fourth discrimination loss to obtain the loss of the first discriminator, and fuse the fifth discrimination loss and the sixth discrimination loss to obtain the loss of the second discriminator; Among them, the third discrimination loss is the loss of the first discriminator determining that the predicted target feature map is false, the third discrimination loss is the cross-entropy loss between the predicted target probability value and 0, the fourth discrimination loss is the loss of the first discriminator determining that the true target feature map is true, the fourth discrimination loss is the cross-entropy loss between the true target probability value and 1, the fifth discrimination loss is the loss of the second discriminator determining that the predicted center feature map is false, the fifth discrimination loss is the cross-entropy loss between the predicted center probability value and 0, and the sixth discrimination loss is the loss of the second discriminator determining that the true center feature map is true, and the sixth discrimination loss is the cross-entropy loss between the true center probability value and 1.

6. The training method of the scene text detection model according to claim 1, wherein The step of fusing the image mask and the sample image to obtain a sample feature map includes: Split the image mask to obtain a first sample mask containing the target identifier and a second sample mask containing the background region identifier, where the target identifier includes the target region identifier or the target center region identifier; Split the sample image into channels to obtain a first channel image, a second channel image, and a third channel image; Respectively fuse the first sample mask with the first channel image, the second channel image, and the third channel image to obtain a first feature image, a second feature image, and a third feature image, and stack the first feature image, the second feature image, and the third feature image based on the channel order to obtain a first sample feature image; Respectively fuse the second sample mask with the first channel image, the second channel image, and the third channel image to obtain a fourth feature image, a fifth feature image, and a sixth feature image, and stack the fourth feature image, the fifth feature image, and the sixth feature image based on the channel order to obtain a second sample feature image; Stack the first sample feature image and the second sample feature image to obtain the sample feature map.

7. The training method of the scene text detection model according to claim 1, characterized in that The method further includes: Split the true target mask and the predicted target mask respectively to obtain a first true target mask and a first predicted target mask containing the target region identifier, and a second true target mask and a second predicted target mask containing the background region identifier; Reduce the sizes of the first true target mask and the first predicted target mask to obtain a third true target mask and a third predicted target mask; Overlay the third true target mask and the second true target mask to obtain the true center mask, and overlay the third predicted target mask and the second predicted target mask to obtain the predicted center mask.

8. The training method of the scene text detection model according to claim 1, characterized in that The method further includes: Obtain the target regions identified by the predicted target mask and the true target mask respectively, reduce the target regions to obtain target center regions, and obtain the predicted center mask and the true center mask with the target center region identifier.

9. The training method of the scene text detection model according to claim 1, wherein The method further includes: Obtain a true label, where the true label includes the target position information in the sample image; Based on the true label, identify the target region and the background region in the sample image to obtain the true target mask.

10. A scene text detection method, characterized in that, Includes: Obtain an image to be processed; Use a scene text detection model to perform detection processing on the image to be processed to obtain a detection result, where the detection result includes a text region image carrying a text region identifier and a background region image carrying a background region identifier; wherein, the scene text detection model is trained by using the training method of the scene text detection model according to any one of claims 1-9.

11. A training device for a scene text detection model, characterized in that Includes a memory and a processor connected to each other, where the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the training method of the scene text detection model according to any one of claims 1-9.

12. A scene text area detection device, characterized in that, Includes a scene text detection model, where the scene text detection model is used to implement the scene text detection method according to claim 10.

13. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it is used to implement the training method of the scene text detection model according to any one of claims 1-9 or the scene text detection method according to claim 10.

Images