Text overlap detection method, apparatus, medium, and electronic device

Abstract

The present disclosure relates to a text overlap detection method, device, medium and electronic equipment, belonging to the technical field of computer, which can improve the accuracy and recall precision of text overlap detection. A text overlap detection method comprises: performing character recognition on a to-be-detected object to obtain character recognition confidence of text lines in the to-be-detected object, and adding text lines with character recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; intercepting text line images of each text line of the to-be-detected object from the to-be-detected object, performing text classification on the text line images, and adding text lines with text classification result as overlapping text to a second candidate abnormal region set; performing target detection on overlapping text in the to-be-detected object, and adding text lines with target detection result as overlapping text to a third candidate abnormal region set; and determining the intersection of the first, second and third candidate abnormal region sets as a text overlap detection result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to a text overlap detection method, device, medium and electronic equipment. BACKGROUND

[0002] The abnormal phenomenon of text overlap in any application program (such as shown in the text overlap schematic diagram of FIG. 1) will seriously affect the user experience, and the text overlap will even cause the user to be unable to understand the page information when the text overlap is serious. Figure 1

[0003] Therefore, an intelligent detection scheme for text overlap is urgently needed. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0005] In a first aspect, the present disclosure provides a text overlap detection method, comprising: performing character recognition on a to-be-detected object to obtain a character recognition confidence of a Chinese text line of the to-be-detected object, and adding a text line with a character recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; intercepting a text line image of each text line of the to-be-detected object from the to-be-detected object, performing text classification on the text line image, and adding a text line with a text classification result of overlapping text to a second candidate abnormal region set; performing target detection on overlapping text in the to-be-detected object, and adding a text line with a target detection result of overlapping text to a third candidate abnormal region set; and determining an intersection of the first candidate abnormal region set, the second candidate abnormal region set and the third candidate abnormal region set as a text overlap detection result.

[0006] ​In a second aspect, the present disclosure provides a text overlap detection apparatus, comprising: a character recognition module configured to perform character recognition on a to-be-detected object to obtain a character recognition confidence of a text line of the to-be-detected object, and add a text line with a character recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; a text classification module configured to cut out a text line image of each text line of the to-be-detected object from the to-be-detected object, perform text classification on the text line image, and add a text line with a text classification result of overlapping text to a second candidate abnormal region set; a target detection module configured to perform target detection on overlapping text in the to-be-detected object, and add a text line with a target detection result of overlapping text to a third candidate abnormal region set; and a determination module configured to determine an intersection of the first candidate abnormal region set, the second candidate abnormal region set, and the third candidate abnormal region set as a text overlap detection result.

[0007] In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program, which, when executed by a processing apparatus, implements the steps of any of the methods of the first aspect of the present disclosure.

[0008] In a fourth aspect, the present disclosure provides an electronic device, comprising: a storage apparatus having stored thereon a computer program; and a processing apparatus configured to execute the computer program in the storage apparatus to implement the steps of any of the methods of the first aspect of the present disclosure.

[0009] By adopting the above technical solution, the first candidate abnormal region set of the to-be-detected object is obtained by using a character recognition method, the second candidate abnormal region set of the to-be-detected object is obtained by using a text classification method, the third candidate abnormal region set of the to-be-detected object is obtained by using a target detection method, and the intersection of the first candidate abnormal region set, the second candidate abnormal region set, and the third candidate abnormal region set is used to determine the text overlap detection result. Since the probability of text overlap in a real environment is very low, the intersection of the above three candidate abnormal region sets is used as the final text overlap detection result, which greatly improves the accuracy of text overlap detection and improves the recall precision. In addition, the human cost of text overlap detection is greatly reduced.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent by describing in detail the following specific embodiments thereof with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic and elements and features are not necessarily to scale. In the drawings:

[0012] Figure 1 A schematic diagram of text overlap is shown.

[0013] Figure 2 A flowchart of a text overlap detection method according to an embodiment of the present disclosure is shown.

[0014] Figure 3 A schematic diagram of taking overlapping text lines and overlapping regions as detection targets in the object to be detected is shown.

[0015] Figure 4 A flowchart of text classification on text line images according to an embodiment of the present disclosure is shown.

[0016] Figure 5 A schematic diagram of the architecture of a transformer encoder is shown.

[0017] Figure 6 A schematic diagram of the architecture of a multi-layer perceptron is shown.

[0018] Figure 7 A schematic diagram of a text line image after padding is shown.

[0019] Figure 8 A schematic diagram of the architecture of a text classifier according to an embodiment of the present disclosure is shown.

[0020] Figure 9 A schematic diagram of the architecture of target detection according to an embodiment of the present disclosure is shown.

[0021] Figure 10 A schematic diagram of automatically generating training samples by extracting foreground characters and superimposing the foreground characters to other positions according to an embodiment of the present disclosure is shown.

[0022] Figure 11 A schematic block diagram of a text overlap detection apparatus according to an embodiment of the present disclosure is shown.

[0023] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail by referring to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather these embodiments are provided so as to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are merely for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that each step recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms have analogous meanings.

[0027] It should be noted that the terms "first", "second", and the like in the present disclosure are merely used to distinguish different devices, modules or units, and do not imply the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context.

[0029] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0030] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0031] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0032] As an optional but not limited implementation manner, in response to receiving the active request of the user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, and the prompt information may, for example, be presented in the form of text in the pop-up window. In addition, the pop-up window may, for example, carry a selection control for the user to select "agree" or "disagree" to provide the personal information to the electronic device.

[0033] It can be understood that the above notification and user authorization obtaining process is only illustrative and does not limit the implementation of the present disclosure, and other ways that meet relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0034] At the same time, it can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0035] Figure 2 is a flowchart of a text overlap detection method according to an embodiment of the present disclosure. As shown in Figure 2 , the text overlap detection method includes the following steps S21-S27.

[0036] In step S21, text recognition is performed on the to-be-detected object to obtain a text recognition confidence of a Chinese text line of the to-be-detected object.

[0037] The to-be-detected object refers to an object that needs to be subjected to text overlap detection. The to-be-detected object can be, for example, a page screenshot of an APP, a screenshot of a game interface, etc.

[0038] The text recognition can be implemented by using optical character recognition (OCR) or any other type of text recognition algorithm.

[0039] In this step, the text recognition confidence of each text line of the to-be-detected object is obtained through text recognition. For example, assuming that the to-be-detected object has three text lines, namely text line 1, text line 2 and text line 3, the text recognition is performed on the three text lines respectively to obtain the text recognition confidence 1 of text line 1, the text recognition confidence 2 of text line 2 and the text recognition confidence 3 of text line 3.

[0040] In step S22, a text line with a text recognition confidence lower than a preset recognition confidence threshold is added to a first candidate abnormal region set.

[0041] The preset recognition confidence threshold can be set according to experience or obtained through self-learning. For example, the text recognition confidence of a text line with text overlap is subjected to self-learning, and accordingly a suitable preset recognition confidence threshold is set. For example, the preset recognition confidence threshold can be set to 0.8 or other suitable numerical values.

[0042] In this step, if the text recognition confidence of a certain text line is lower than the preset recognition confidence threshold, the text line will be added to the first candidate abnormal region set. For example, if the aforementioned text recognition confidence 1 is lower than the preset recognition confidence threshold, and the text recognition confidence 2 and the text recognition confidence 3 are both greater than the preset recognition confidence threshold, the text line 1 will be added to the first candidate abnormal region set.

[0043] In step S23, a text line image of each text line of the to-be-detected object is intercepted from the to-be-detected object, and the text line image is subjected to text classification.

[0044] In some embodiments, intercepting the text line image of each text line of the to-be-detected object from the to-be-detected object can include: first, obtaining the coordinate information of each text line in the to-be-detected object, for example, the coordinate information of each text line can be obtained by using a text recognition tool (such as OCR recognition); and then, according to the coordinate information of each text line, the text line image of each text line is intercepted from the to-be-detected object, that is, one text line is intercepted as one text line image. Assuming that the to-be-detected object includes a total of 3 text lines, a total of 3 text line images will be intercepted.

[0045] In some embodiments, the text classification of the text line image can include: using various text classifiers to respectively perform text classification on each text line image. For example, a visual task-based classification algorithm is used to respectively perform text classification on each text line image.

[0046] In step S24, the text line whose text classification result is overlapping text is added to the second candidate abnormal region set.

[0047] After obtaining the text classification result of each text line image in step S23, the text line whose text classification result is overlapping text can be added to the second candidate abnormal region set in step S24.

[0048] For example, assuming that the text classification result of the text line 1 is overlapping text, and the text classification results of the text line 2 and the text line 3 are both normal text, the text line 1 will be added to the second candidate abnormal region set.

[0049] In step S25, the overlapping text in the to-be-detected object is subjected to target detection.

[0050] Various target detection algorithms can be used to perform target detection on the overlapping text in the to-be-detected object, such as the Darknet target detection algorithm.

[0051] In some embodiments, the target detection on the overlapped text in the to-be-detected object can include: performing the target detection by taking the overlapped text line and the overlapped region as the detection target in the to-be-detected object. That is, there are two objects for the target detection, one is the overlapped text line, and the other is the overlapped region.

[0052] In step S26, the text line with the target detection result of the overlapped text is added to the third candidate abnormal region set.

[0053] For example, if the target detection algorithm detects that the text line 1 is the overlapped text, and the text line 2 and the text line 3 are normal texts, the text line 1 will be added to the third candidate abnormal region set in this step.

[0054] In some embodiments, in the case of taking the overlapped text line and the overlapped region as the detection target in the to-be-detected object, the adding of the text line with the target detection result of the overlapped text to the third candidate abnormal region set in step S26 can include: if the target detection result indicates that the coordinates of a certain overlapped text line overlap with the coordinates of a certain overlapped region, the overlapped text line is added to the third candidate abnormal region set. In this way, the accuracy of the target detection can be improved, and the false positives of the target detection can be reduced. For example, for the to-be-detected object shown in FIG. 1, if the overlapped text line 1 and the overlapped region 2 are detected, and there is a coordinate overlap between the overlapped text line 1 and the overlapped region 2, the overlapped text line 1 can be added to the third candidate abnormal region set in this case. Figure 3

[0055] In step S27, the intersection of the first candidate abnormal region set, the second candidate abnormal region set and the third candidate abnormal region set is determined as the text overlap detection result.

[0056] By adopting the technical solution, the first candidate abnormal region set of the to-be-detected object is obtained by using the character recognition mode, the second candidate abnormal region set of the to-be-detected object is obtained by using the text classification mode, the third candidate abnormal region set of the to-be-detected object is obtained by using the target detection mode, and the intersection of the first candidate abnormal region set, the second candidate abnormal region set and the third candidate abnormal region set is used to determine the text overlap detection result. Since the probability of text overlap in the real environment is very low, the intersection of the above three candidate abnormal region sets is taken as the final text overlap detection result, which greatly improves the accuracy of the text overlap detection and improves the recall precision. In addition, the human cost of the text overlap detection is greatly reduced.

[0057] ​In some embodiments, the text classification of the text line image can include: cutting the text line image into a plurality of image blocks; and performing text classification on each image block by using a text classifier using a category vector, a transformer structure, and a binary multi-layer perceptron, wherein the category vector is used to integrate the overall image features of the text line image.

[0058] The text line image is cut because most text lines generally only partially overlap, such as Figure 1 The "1990 / USA / sci-fi" text line in the example is only partially overlapped in the "0 / USA" region, and the other regions are not overlapped. By cutting the text line image, the text line with text overlap and the normal text line can be better distinguished.

[0059] The binary multi-layer perceptron refers to a multi-layer perceptron using a binary classification model, and the classification result of the binary classification model includes overlapping text and normal text.

[0060] By using the above technical solution, the text classification of each image block and the category vector can be performed, and then the overall image features of the text line image (e.g., whether the text is overlapped or not) can be known according to the text classification result of the category vector, and the features of each image block (e.g., whether the text is overlapped or not) can be known according to the text classification result of each image block.

[0061] Figure 4 is a flowchart of text classification of a text line image according to an embodiment of the present disclosure. As shown in Figure 4 The text classification process includes steps S41 to S46.

[0062] In step S41, the text line image is cut into a plurality of image blocks to obtain a first vector.

[0063] In some embodiments, the size of the image blocks into which the text line image is cut is determined according to the size of the convolution kernel of the neural network used. For example, if the size of the convolution kernel used is 16*16, the size of the image blocks obtained by cutting should be 16*16.

[0064] After cutting, the first vector obtained is k*n*m*j. Wherein k represents the number of image blocks obtained by cutting, n*m represents the size of each image block, and j represents the number of channels. For example, if it is a color text line image, the number of channels j is 3.

[0065] The text line image is cut because most text lines generally only partially overlap, such as Figure 1In the text behavior example of "1990 / US / sci-fi" in the text, only the "0 / US" region has text overlap, and through the cutting of the text line image, the text line with text overlap phenomenon and the normal text line can be better distinguished.

[0066] In step S42, a linear transformation is performed on the first vector to obtain a second vector.

[0067] The linear transformation (i.e., a fully connected layer) refers to extracting a flat pixel vector in each image block and inputting each image block into a linear projection layer. In the linear transformation, the compression dimension D is taken, for example, D can be 512 or other values. This step can also be referred to as patch embedding.

[0068] The second vector obtained after step S42 is k*n*m*D.

[0069] In step S43, a learnable class vector is added to the second vector to obtain a third vector, wherein the class vector is used to integrate the whole image features of the text line image.

[0070] The third vector obtained after adding the class vector cls_token is (k+1)*n*m*D.

[0071] In step S44, a position encoding is added to the third vector to obtain a fourth vector, wherein the position encoding is used to represent the relative position relationship of each image block.

[0072] Since the order information of the input series will be lost in the subsequent encoding process, the position encoding is added here so that the relative position relationship of each image block can still be known after encoding. The fourth vector obtained after adding the position encoding is (k+1)*(n*m+1)*D.

[0073] In step S45, the transformer structure is used to encode the fourth vector.

[0074] Figure 5 The architecture of the transformer encoder is shown. As shown in Figure 5 After the embedded image block (i.e., the fourth vector) is input into the transformer encoder, it is first processed by layer normalization, then passed through a multi-head attention module for feature enhancement, the output result of the multi-head attention module is connected in residual with the embedded image block, then processed by layer normalization, and then processed by a multi-layer perception to extract features. The output result is again connected in residual with the embedded image block to obtain the final encoder output.

[0075] By encoding, the feature difference between the text overlap and the non-text overlap (i.e., normal text) is more obvious.

[0076] In step S46, the encoded fourth vector is classified by using a binary multi-layer perceptron to obtain a category vector and a text classification result of each image block.

[0077] The architecture of the binary multi-layer perceptron is various, Figure 6 An architecture diagram of the binary multi-layer perceptron is shown. As shown, Figure 6 First, the input from the encoder is linearized, then activated by using an activation function (such as GeLU), then the channel number is reduced, then linearized again, and then the channel number is reduced again to obtain the final text classification result.

[0078] By using the above text classification technical solution, the text classification of each image block and the category vector can be performed, and then the whole text line image feature (such as whether the text is overlapped or not) can be known according to the text classification result of the category vector, and the feature of each image block (such as whether the text is overlapped or not) can be known according to the text classification result of each image block.

[0079] In some embodiments, it is possible that the size of a certain text line image does not meet the size requirement of the convolution kernel of the neural network, for example, the size of the text line image is not a multiple of the size of the convolution kernel, in this case, the size of the text line image can be adjusted while keeping the aspect ratio of the text line image before cutting the text line image into multiple image blocks, so that the size of the adjusted text line image is an integer multiple of the size of the convolution kernel, and then the pixels (such as 0) are filled in the adjusted text line image, so that the size-adjusted text line image is obtained, and the padding here means expansion based on the original text line image. In addition, the mask of the padding position can also be recorded, that is, the padding mask is recorded, so that it can be known where the padding is performed. Figure 7 A filled text line image diagram is shown.

[0080] In the case of padding the text line image, Figure 4 The encoding of the fourth vector using the transformer structure in step S45 in the above method can include: encoding the fourth vector using the transformer structure, and not performing attention mechanism on the padding area in the encoding process. By not performing attention mechanism on the padding area, the feature difference between the text overlap and the non-text overlap (i.e., normal text) is more obvious.

[0081] In some embodiments, adding text lines whose text classification results are overlapping text to the second candidate anomaly region set in step S24 includes: if the text classification result of the category vector corresponding to the text line image is overlapping text, the text classification result of more than N consecutive image blocks in the text line image is overlapping text, and the text classification confidence of the category vector and the text classification result of more than N consecutive image blocks is greater than a preset classification confidence threshold, then the text line is added to the second candidate anomaly region set.

[0082] N is a positive integer greater than or equal to 3.

[0083] The preset classification confidence threshold can be set based on experience or obtained through self-learning. For example, the classification confidence of text lines with text overlap can be learned through self-learning, and an appropriate preset classification confidence threshold can be set accordingly. For example, the preset classification confidence threshold can be set to 0.9 or other suitable values.

[0084] For example, for a certain text line image, if the text classification result of the category vector corresponding to the text line image is overlapping text, and image blocks 1, 2 and 3 in the text line image are consecutive image blocks and the text classification result of these three image blocks is also overlapping text, and the text classification confidence of the category vector and the text classification results of image blocks 1, 2 and 3 are all greater than the preset classification confidence threshold, then the text line will be added to the second candidate abnormal region set.

[0085] By adopting the above technical solution, the corresponding text line is considered to belong to overlapping text only when the text classification result of the category vector corresponding to the text line image is overlapping text, the text classification result of more than N consecutive image blocks in the text line image is overlapping text, and the text classification confidence of both the category vector and the text classification result of more than N consecutive image blocks is greater than the preset classification confidence threshold. This improves the accuracy of text overlap detection and the recall precision.

[0086] Figure 8 A schematic diagram of the architecture of a text classifier according to an embodiment of this disclosure is shown. Figure 8 As shown, firstly, the text line image is filled while maintaining its aspect ratio; then, the text line image is segmented to obtain multiple image blocks. Figure 8 (The illustration uses 16 image patches as an example). Then, a linear transformation is performed on the image patches in the linear projection layer; then, positional encoding and category vectors are added, where... Figure 8 The "*" in the code represents the category vector; then it is encoded in the Transformer encoder; and then classified in the MLP to obtain the text classification result.

[0087] By adopting Figure 8 The architecture shown enables text classification for each line of text in the image.

[0088] Figure 9 A schematic diagram of an architecture for object detection according to an embodiment of this disclosure is shown. This architecture employs the Darknet53 architecture. Figure 9 As shown, after processing by Darknet53, three outputs are obtained, with dimensions (batch_size, 52, 52, 21), (batch_size, 26, 26, 21), and (batch_size, 13, 13, 21), where 21 = 3*(2+4+1), 3 represents the number of anchor boxes, 2 represents the number of target categories within the anchor boxes (in this disclosure, target categories include overlapping text lines and overlapping regions), 4 represents the coordinate offset values ​​(i.e., tx, ty, tw, th), and 1 represents whether it is an overlapping target. In this architecture, there are a total of 9 anchor boxes, evenly distributed across the aforementioned 3-dimensional feature layers. This achieves multi-scale detection, enabling the detection of overlapping targets of both large and small sizes. Additionally, it should be noted that... Figure 9 In the text, the dimensions of the anchor frames "52×52", "26×26" and "13×13" and the number of anchor frames 3 are merely examples, and this disclosure does not limit them.

[0089] In some embodiments, the text overlap detection method according to this disclosure further includes: training a classifier that performs text classification and an object detector that performs object detection.

[0090] For deep learning network training, the number of training samples is crucial. However, since real-world text overlap data is scarce, the text overlap detection method according to embodiments of this disclosure further includes a step of automatically generating training samples to avoid expending significant manpower to view and annotate text overlap data.

[0091] Training samples can be automatically generated using at least one of the following methods:

[0092] (1) Write text on a normal text line, where the font, color and string of the text are random;

[0093] (2) Extract the foreground characters from the text line images and overlay the extracted foreground characters onto other text line images, such as... Figure 10 As shown in label 4, the text at label 4 was originally normal text. By extracting foreground characters from other locations in the image and superimposing the extracted foreground characters onto label 4, overlapping text was formed.

[0094] In both of the above methods, the coordinates of the text line can be obtained through a text recognition algorithm (such as OCR), which can then determine where to write text or add foreground characters based on the coordinates of the text line.

[0095] Additionally, the coordinates of automatically generated text lines with overlapping text can be recorded for use when training text classifiers and object detectors.

[0096] After obtaining enough training samples, these training samples can be used to train the classifier that performs text classification and the object detector that performs object detection, so as to improve the classification accuracy of the classifier and the object detection accuracy of the object detector.

[0097] Figure 11 This is a schematic block diagram of a text overlap detection device according to an embodiment of the present disclosure. Figure 11 As shown, the text overlap detection device includes: a text recognition module 121, used to perform text recognition on the object to be detected, obtain the text recognition confidence of the text lines in the object to be detected, and add the text lines with text recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; a text classification module 122, used to extract text line images of each text line of the object to be detected from the object to be detected, perform text classification on the text line images, and add the text lines whose text classification result is overlapping text to a second candidate abnormal region set; a target detection module 123, used to perform target detection on the overlapping text in the object to be detected, and add the text lines whose target detection result is overlapping text to a third candidate abnormal region set; and a determination module 124, used to determine the intersection of the first candidate abnormal region set, the second candidate abnormal region set, and the third candidate abnormal region set as the text overlap detection result.

[0098] By employing the above technical solution, a first set of candidate abnormal regions for the object to be detected is obtained using character recognition, a second set using text classification, and a third set using object detection. The intersection of these three sets is then used to determine the text overlap detection result. Since the probability of text overlap in real-world environments is extremely low, using the intersection of these three candidate abnormal region sets as the final text overlap detection result significantly improves the accuracy and recall precision of text overlap detection. Furthermore, it greatly reduces the human resource cost of text overlap detection.

[0099] In some embodiments, the text classification module 122 performs text classification on the text line image, including: cutting the text line image into a plurality of image blocks; and performing text classification on each of the image blocks using a text classifier that utilizes a class vector, a transformer structure, and a binary multi-layer perceptron, wherein the class vector is used to integrate overall image features of the text line image.

[0100] In some embodiments, the text classification module 122 performs text classification on each of the image blocks using a text classifier that utilizes a class vector, a transformer structure, and a binary multi-layer perceptron, including: performing linear transformation on a first vector composed of the plurality of image blocks to obtain a second vector; adding a learnable class vector to the second vector to obtain a third vector; adding position encoding to the third vector to obtain a fourth vector, wherein the position encoding is used to represent relative position relationships of the image blocks; encoding the fourth vector using the transformer structure; and classifying the encoded fourth vector using the binary multi-layer perceptron to obtain the class vector and text classification results of the image blocks.

[0101] In some embodiments, the text classification module 122 is further configured to, before cutting the text line image into a plurality of image blocks, adjust the size of the text line image while maintaining the aspect ratio of the text line image, and fill pixels in the adjusted text line image.

[0102] The text classification module 122 is further configured to encode the fourth vector using the transformer structure, and not perform attention mechanism on the filled region during the encoding process.

[0103] In some embodiments, the text classification module 122 adds a text line with text classification result of overlapping text to the second candidate abnormal region set, including: if the text classification result of the class vector corresponding to the text line image is overlapping text, the text classification result of the image blocks in the text line image is overlapping text for more than N consecutive image blocks, and the text classification confidence of the class vector and the text classification result of the more than N consecutive image blocks is greater than a preset classification confidence threshold, then the text line is added to the second candidate abnormal region set.

[0104] In some embodiments, the object detection module 123 performs object detection on overlapping text in the object to be detected, including: taking overlapping text lines and overlapping regions as detection targets in the object to be detected to perform the object detection.

[0105] In some embodiments, the target detection module 123 adds text lines whose target detection results are overlapping text to the third candidate abnormal region set, including: if the target detection result indicates that the coordinates of a certain overlapping text line overlap with the coordinates of a certain overlapping region, then the overlapping text line is added to the third candidate abnormal region set.

[0106] In some embodiments, the text overlap detection apparatus according to this disclosure further includes a training module for: automatically generating training samples by at least one of the following methods: writing text on a normal text line, wherein the font, color, and string of the written text are all random; extracting foreground characters from the text line image and superimposing the extracted foreground characters onto other text line images; and training a classifier that performs the text classification and an object detector that performs the object detection using the training samples.

[0107] The specific implementation of the operations performed by each module in the text overlap detection device according to the embodiments of this disclosure has been described in detail in the relevant methods, and will not be repeated here.

[0108] The following is for reference. Figure 12 The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0109] like Figure 12 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0110] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although Figure 12 The electronic device 600 is shown with various devices, but it is understood that all of the illustrated devices are not required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0111] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing devices 601, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0112] It is noted that the aforementioned computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a computer-readable program code transmitted by a computer-readable medium or a carrier wave in a baseband or as part of a carrier wave. Such a propagated computer-readable signal medium can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium can also be any computer-readable medium that is not a computer-readable storage medium and that can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0113] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0114] The aforementioned computer-readable medium can be included in the aforementioned electronic device; or can exist separately from the electronic device and can be accessed via the electronic device.

[0115] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: perform text recognition on a to-be-detected object to obtain a text recognition confidence of a text line of the to-be-detected object, and add a text line with a text recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; cut out a text line image of each text line of the to-be-detected object from the to-be-detected object, perform text classification on the text line image, and add a text line with a text classification result of overlapping text to a second candidate abnormal region set; perform target detection on overlapping text in the to-be-detected object, and add a text line with a target detection result of overlapping text to a third candidate abnormal region set; and determine an intersection of the first candidate abnormal region set, the second candidate abnormal region set, and the third candidate abnormal region set as a text overlapping detection result.

[0116] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages, including object oriented programming languages such as Java, Smalltalk, C++, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0117] The flow and block diagrams in the drawings show architectural, functional, and operational representations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0118] The modules described in the embodiments of the present disclosure can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself. For example, the first obtaining module can also be described as a module for obtaining at least two Internet protocol addresses.

[0119] The functions described above in the present document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0120] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] According to one or more embodiments of the present disclosure, example 1 provides a text overlap detection method, comprising: performing character recognition on a to-be-detected object to obtain a character recognition confidence of a Chinese text line of the to-be-detected object, and adding a text line with a character recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; intercepting a text line image of each text line of the to-be-detected object from the to-be-detected object, performing text classification on the text line image, and adding a text line with a text classification result of overlapping text to a second candidate abnormal region set; performing target detection on overlapping text in the to-be-detected object, and adding a text line with a target detection result of overlapping text to a third candidate abnormal region set; and determining an intersection of the first candidate abnormal region set, the second candidate abnormal region set, and the third candidate abnormal region set as a text overlap detection result.

[0122] According to one or more embodiments of the present disclosure, example 2 provides the method of example 1, wherein the text classification of the text line image comprises: cutting the text line image into a plurality of image blocks; and performing text classification on each of the image blocks using a text classifier that utilizes a class vector, a transformer structure, and a binary multi-layer perceptron, wherein the class vector is used to integrate overall image features of the text line image.

[0123] According to one or more embodiments of the present disclosure, example 3 provides the method of example 2, wherein the text classification of each of the image blocks using the text classifier that utilizes the class vector, the transformer structure, and the binary multi-layer perceptron comprises: performing linear transformation on a first vector composed of the plurality of image blocks to obtain a second vector; adding a learnable class vector to the second vector to obtain a third vector; adding position encoding to the third vector to obtain a fourth vector, wherein the position encoding is used to represent relative position relationships of the image blocks; encoding the fourth vector using the transformer structure; and classifying the encoded fourth vector using the binary multi-layer perceptron to obtain the class vector and text classification results of the image blocks.

[0124] According to one or more embodiments of the present disclosure, example 4 provides the method of example 3, wherein, before the cutting of the text line image into the plurality of image blocks, the method further comprises: adjusting a size of the text line image while maintaining an aspect ratio of the text line image, and filling pixels in the adjusted text line image.

[0125] The encoding of the fourth vector using the transformer structure comprises: encoding the fourth vector using the transformer structure, and not performing attention mechanism on the filled region during the encoding.

[0126] According to one or more embodiments of the present disclosure, example 5 provides the method of example 3 or 4, wherein the adding of the text line with the text classification result of overlapping text to the second candidate abnormal region set comprises: if the text classification result of the class vector corresponding to the text line image is overlapping text, the text classification result of the image blocks in the text line image is overlapping text for more than N consecutive image blocks, and the text classification confidence of the class vector and the text classification result of the more than N consecutive image blocks is greater than a preset classification confidence threshold, then the text line is added to the second candidate abnormal region set.

[0127] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 1, wherein the target detection of overlapping text in the object to be detected includes: taking overlapping text lines and overlapping regions as detection targets in the object to be detected, and performing the target detection.

[0128] According to one or more embodiments of this disclosure, Example 7 provides the method of Example 6, wherein adding text lines whose target detection results are overlapping text to a third candidate anomaly region set includes: if the target detection result indicates that the coordinates of a certain overlapping text line overlap with the coordinates of a certain overlapping region, then adding the overlapping text line to the third candidate anomaly region set.

[0129] According to one or more embodiments of this disclosure, Example 8 provides the method of Example 1, wherein the method further includes: automatically generating training samples by at least one of the following methods: writing text on a normal text line, wherein the font, color, and string of the written text are random; extracting foreground characters from the text line image and overlaying the extracted foreground characters onto other text line images; and training a classifier performing the text classification and an object detector performing the object detection using the training samples.

[0130] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0131] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0132] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, in which various modules perform operations, the specific manner in which the operations are performed by the various modules has been described in detail in the embodiments relating to the method. Here, no detailed explanation will be given.

Claims

1. A method of text overlay detection, the method comprising: The method comprises the following steps: performing character recognition on a to-be-detected object to obtain a character recognition confidence of a text line of the to-be-detected object, and adding a text line with a character recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal region set; cutting a text line image of each text line of the to-be-detected object from the to-be-detected object, performing text classification on the text line image, and adding a text line with a text classification result of overlapping text to a second candidate abnormal region set; performing target detection on overlapping text in the to-be-detected object, and adding a text line with a target detection result of overlapping text to a third candidate abnormal region set; determining an intersection of the first candidate abnormal region set, the second candidate abnormal region set and the third candidate abnormal region set as a text overlapping detection result.

2. The method of claim 1, wherein, The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks:

3. The method of claim 2, wherein, adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks:

4. The method of claim 3, wherein, adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks:

5. The method according to any one of claims 2 to 4, characterized in that, adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks:

6. The method of claim 1, wherein, adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the text line image while keeping the aspect ratio of the text line image, and filling pixels in the adjusted text line image. The method further comprises the following steps before the text line image is cut into a plurality of image blocks: adjusting the size of the The target detection is performed on the overlapping text lines and the overlapping areas as detection targets in the object to be detected.

7. The method of claim 6, wherein, The target detection result of the text line of overlapping text is added to the third candidate abnormal area set. If the target detection result indicates that the coordinates of a certain overlapping text line overlap with the coordinates of a certain overlapping area, the overlapping text line is added to the third candidate abnormal area set.

8. The method of claim 1, wherein, The method further comprises: The training samples are automatically generated by at least one of the following ways: writing characters on normal text lines, wherein the font, color and string of the written characters are random; performing foreground character extraction on the text line images, and superimposing the extracted foreground characters on other text line images; The training samples are used to train a classifier for performing the text classification and a target detector for performing the target detection.

9. A text overlay detection apparatus characterized by comprising: It comprises: a character recognition module, configured to perform character recognition on an object to be detected to obtain a character recognition confidence of a text line of the object to be detected, and add a text line with a character recognition confidence lower than a preset recognition confidence threshold to a first candidate abnormal area set; a text classification module, configured to cut text line images of each text line of the object to be detected from the object to be detected, perform text classification on the text line images, and add a text line with a text classification result of overlapping text to a second candidate abnormal area set; a target detection module, configured to perform target detection on overlapping text in the object to be detected, and add a text line with a target detection result of overlapping text to a third candidate abnormal area set; a determination module, configured to determine an intersection of the first candidate abnormal area set, the second candidate abnormal area set and the third candidate abnormal area set as a text overlapping detection result.

10. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processing device to implement the steps of the method of any one of claims 1-8.

11. An electronic device, comprising: It comprises: a storage device having a computer program stored thereon; a processing device configured to execute the computer program in the storage device to implement the steps of the method of any one of claims 1-8.

Images